Minicourse - PDE Techniques for Image Inpainting Part Igkadmin/daten/minicourse_part1.pdf · Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Schonlieb¨ Institute
105
Minicourse - PDE Techniques for Image Inpainting Part I Carola-Bibiane Sch ¨ onlieb Institute for Numerical and Applied Mathematics University of G¨ ottingen G¨ ottingen - January, 7th 2010 Sch ¨ onlieb (NAM, G ¨ ottingen) PDEs for Image Inpainting Part I G¨ ottingen-7. January 2010 1 / 50
Minicourse - PDE Techniques for Image InpaintingPart I
Carola-Bibiane Schonlieb
Institute for Numerical and Applied MathematicsUniversity of Gottingen
Gottingen - January 7th 2010
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 1 50
Outline
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50
Outline
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50
Digital Image Processing
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50
Digital Image Processing Examples
Imaging tasks1
Image Denoising
dArr
CartoonTexture Decomposition
dArr
1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Outline
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50
Outline
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50
Digital Image Processing
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50
Digital Image Processing Examples
Imaging tasks1
Image Denoising
dArr
CartoonTexture Decomposition
dArr
1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Outline
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 2 50
Digital Image Processing
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50
Digital Image Processing Examples
Imaging tasks1
Image Denoising
dArr
CartoonTexture Decomposition
dArr
1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 3 50
Digital Image Processing Examples
Imaging tasks1
Image Denoising
dArr
CartoonTexture Decomposition
dArr
1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Imaging tasks1
Image Denoising
dArr
CartoonTexture Decomposition
dArr
1Examples from L Vese S Osher (2004)Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 4 50
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Imaging tasks (cont)
Image Segmentationa
aExample from C Guyader LVese (2008)
Image Inpaintinga
dArr
aExample from Bertalmio et al(1998)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 5 50
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Image inpainting - restore images of different kind
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 6 50
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Image inpainting - restore images of different kind(cont)
1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 7 50
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Image inpainting - in general the task is
a
asource Chan and Shen2005
Inpainting = ImageinterpolationReconstruct the idealimage u on the missingdomain D based on thedata of u availableoutside D
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 8 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications
Digital Restoration of FrescoesFife Senses Project on Mathematical Methods for Image Analysis and Processing inthe Visual Arts sponsored by the Viennese Technology Fund (WWTF)Collaboration between
Faculty of Mathematics University of Vienna (Peter A Markowich MassimoFornasier)
Restore the frescoes digitally in an automated wayrArr courtesy and template formuseums artists
Academy of Fine Arts Vienna Institute for Conservation and Restauration(Wolfgang Baatz)
Physcial Restoration of the FrescoesrArr Comparison with the digital result
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 9 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Preliminary ResultsGiven image Restored Image
Unfortunately this doesnt always work as straightforward as in this example
Large gaps
Lack of grayvalue contrast
Low color saturation and hue
we need more sophisticated algorithms to restore the frescoes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 10 50
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Results with Cahn-Hilliard inpainting
2
2Cahn-Hilliard inpainting after 200 timesteps with ε = 3 and additional 800timesteps with ε = 001
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 11 50
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Binary based total variation inpaintingBased on the so recovered binary structure the fresco is colorized
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 12 50
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Road Reconstruction3
Our data consists of satellite images of roads in Los Angeles of the
following kind
3Joint work with Andrea Bertozzi from UCLASchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 13 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain road
Results
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
Some applications (cont)
Road Recontruction with Bitwise Cahn-Hilliard
Goal Remove objects like trees that cover the road and recover thepicture of the plain roadResults
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 14 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples)
works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains
performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way
and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Examples
We seek for a method which
smoothly continues image contents into the missing domain
connects edgesobjects even across large gaps (cf roadexamples) works for all numbers and shapes of missing domains performs the restoration process in an automated way and all this with a reliable and efficient numerical scheme
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 15 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing What is a Digital Image and how do we Process it
Digital images
A digital image is obtained from an analogue image (representing thecontinuous world) by sampling and quantization
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 16 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing What is a Digital Image and how do we Process it
Digital images (cont)
it consists of pixels (grid element matrix positions) which areassigned with the mean grayscale or colour information within thiselement
grayscale imageu Ω = 1 2 m times 1 2 n rarr I = 0 1 255colour image u Ω rarr I3 where u(x y) = (r g b) = (red greenblue)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 17 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Multiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
Lattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fields
Level-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255
Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Image models and representations
Image function u Ω sub R2 rarr R Ω is open and bounded (eventuallywith Lipschitz boundary)
functions Sobolev images Hk BV imagesMultiscale representations eg Wavelets curvelets ridgeletsshearletsLattice and random field representations eg images as Markovrandom fieldsLevel-set representation ie representing the grayscale image asthe set of level lines for the values 0 1 255Mumford-Shah image model ie representing the image as acombination of a smooth parts and discontinuities (edges)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 18 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions
Let Ω be a two-dimensional Lipschitz domain ie open bounded withLipschitz boundary Then we define the set of sensors on Ω as
D(Ω) = φ isin Cinfin(Ω) supp(φ) sube Ω
An image u on Ω is then treated as a distribution ie a linearfunctional on D(Ω)
u φ rarr (u φ)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 19 50
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
with additional positivity constraint we have that the Riesz representationtheorem is valid ie u is a positive distribution then for any sensorφ isin D(Ω) there exists a Radon measure micro on Ω st
(u φ) =int
Ω
φ(x) dmicro
notion of derivatives ie distributional derivative v = partαpartβu is definedas a new distribution such that
(v φ) = (minus1)α+β(u partαpartβφ
) forallφ isin D(Ω)
sensing of distributions mimics the digital sensor devices in CCDcameras
very general class of functions
to capture intrinsic visual features in an image we have to impose moreinformation into the image model rArr Sobolev spaces bounded variation
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 20 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Distributions (cont)
Example 1 - Bright spot as delta distribution
Here u is a bright spot concentratedat the origin ie u(x) = δ(x) whereδ stands for the Dirac delta functionThen
(u φ) = φ(0) for any sensor φ isin D(Ω)
Example 2 - Step edge as Heaviside function
Here u describes a step edge from 0to 255 ie u(x) = u(x1 x2) = H(x1)where H(t) is the Heaviside 0 minus 255step function ie
H(t) =
255 t ge t1
0 t lt t1
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 21 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Lp functionsFor any p isin [0infin) the Lebesgue Lp function space is defined as
Lp(Ω) =
u int
Ω
|u(x)|p dx lt infin
For p = infin an Linfin image is understood as an essentially bounded functionProperties
Banach spaces with norms
up =[int
Ω
|u(x)|p dx
]1p
Lp images are naturally distributional images (Riesz respresentationtheorem for Lp + D(Ω) is dense in Lplowast for 1 le plowast lt infin)
but they carry more structures than gen-eral distributions cf layer-cake representa-tion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 22 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1
H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image information
Higher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - Hk functions
An image u isin L2(Ω) whose (distributional) gradient nablau is inL2 times L2 is said to be Sobolev H1H1(Ω) is a Hilbert space with inner product
(u v)H1 = (u v)L2 + (nablaunablav)L2timesL2
and corresponding norm
uH1 =radicu2L2 + nablau2L2timesL2
Measure image informationHigher-order Sobolev spaces Hk(Ω) defined in similar fashion(higher-order derivatives)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 23 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributions
For a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (TV)
For u isin L1loc(Ω)
V (u Ω) = supint
Ωunabla middot ϕ dx ϕ isin
[C1
c (Ω)]2
ϕinfin le 1
is the variation of u Furtheru isin BV (Ω) (the space of bounded variation functions)
hArrV (u Ω) lt infin
In such a case|Du|(Ω) = V (u Ω)
where |Du|(Ω) is the total variation of the finite Radon measure Duthe derivative of u in the sense of distributionsFor a function u isin C1(Ω) the total variation of u is equivalent to
|Du|(Ω) =intΩ|nablau| dx
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 24 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
PropertiesBV images allow edges (in contrast to W 11 ie H1 images)The total variation penalizes small irregularitiesoscillations whilerespecting instrinsic image features such as edges
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 25 50
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Deterministic image models - The total variation (cont)
What does it measure
Figure The total variation measures the size of the jump
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 26 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Mathematical Image Models
Level lines amp coarea formulaLet u = u(x) be a grayscale image on Ω For each real value λ we define theλ-level line of u to be
Γλ = x isin Ω u(x) = λ
Classical level line representation is the one-parameter family of all the levellines ie
Γλ λ isin R
Coarea formula (weak version) For a smooth image u we haveintΩ
|nablau| dx =int infin
minusinfinlength(Γλ) dλ
a similar relation holds for functions of bounded variation (length has to beredefined)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 27 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering
Let f be a grayscale image represented by a real-valued mappingf isin L1(R2) (zero-expansion of image from Ω to R2) Gaussiansmoothing of f by convolution
u = (Kσ lowast f)(x) =int
R2
Kσ(xminus y)f(y) dy
where Kσ denotes the 2D Gaussian of width σ gt 0 ie
Kσ(x) =1
2πσ2eminus|x
2|(2σ2)
PropertiesSince Kσ isin Cinfin(R2) we get u = Kσ lowast f isin Cinfin(R2) (even if f isonly absolutely integrable)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 28 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Gaussian filtering (cont)
Behaviour in the frequency domain
(F(Kσ lowast f))(w) = (FKσ)(w) middot (Ff)(w)
= eminus|w|2(2σ2) middot (Ff)(w)
Low-pass filter that attenuates high frequencies
Equivalence to linear diffusion filtering
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 29 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing
For a given f isin C(R2) we consider the linear diffusion prozess
ut = ∆u
u(x 0) = f(x)
with solution
u(x) =
f(x) t = 0(Kradic
2t lowast f)(x) t gt 0
PropertiesSolution is unique in the class of functions u with
|u(x t)| le M middot ea|x|2 (Ma gt 0)
The solution depends continuously on the initial image f wrt middot Linfin The solution fulfills the max-min principle
infR2
f le u(x t) le supR2
f on R2 times [0infin)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 30 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image smoothing (cont)
Gaussian smoothing structures of order σ requires to stop thediffusion process at time
T =12σ2
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 31 50
reka500heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Nonlinear diffusion filteringPerona-Malik model Replace linear diffusion process by a nonlinear onewhich reduces the diffusivity at edges
ut = div(g(|nablau|2)nablau
)
where g is called the diffusivity of the process eg
g(s2) =1
1 + s2δ2(δ gt 0)
Interplay of forward- and backward diffusion rArr image smoothing amp edgedetection in a single process
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 32 50
reka300nonlineardiffavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting
Construction of grayvalues inside the holes by rdquoaveragingldquo grayvaluesaround the holes
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 33 50
adameva250heatequavi
Media File (videoavi)
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λ
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Digital Image Processing Gaussian Filtering the Heat Equation and Nonlinear Diffusion
The heat equation for image inpainting (cont)
Let D sub Ω be a hole in the image domain We solve
ut = ∆u in D
u(x 0) = 0 in D
u|partD = f |partD
or we solve for the union of all holes D
ut = λ∆u + χΩD(uminus f) in Ωu(x 0) = 0 in Ω
where
χΩD(x) =
1 x isin Ω D
0 otherwise
and for 1 λSchonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 34 50
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting
Outline
1 Digital Image ProcessingExamplesWhat is a Digital Image and how do we Process itMathematical Image ModelsGaussian Filtering the Heat Equation and Nonlinear Diffusion
2 Image InpaintingState of the Art MethodsThe VariationalPDE ApproachSecond- Versus Higher-Order PDEs for Inpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 35 50
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Some history and classifications
The term inpainting was invented by art restoration workers [GEmile-Male 76 S Walden 85] and first appeared in the frameworkof digital restoration in the work of [Bertalmio et al 00] (PDEapproach)Before this works of engineers and computer scientists statisticaland algorithmic approaches in the context of image interpolationeg [A C Kokaram et al 95] image replacement eg [H Igehyand L Pereira 97] error concealment [K-H Jung et al 94] andimage coding eg [J R Casas 96] Some of these codingtechniques already used PDEs for this task eg [J R Casas 96]Mathematics community got involved in image restoration usingpartial differential equations and variational methods for this taskeg [M Nitzberg D Mumford and T Shiota 93 Masnou andMorel 98 V Caselles J-M Morel and C Sbert 98 Bertalmio etal 00]
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 36 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approaches
Let us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Some history and classifications (cont)
PDE- and variational methods are local inpainting methods ierestore structure (geometry) but are not applicable for texturerestorationsynthesisGlobal inpainting methods texture synthesis amp exemplar basedinpainting eg [Efros and Leung 99]Inpainting in transform spaces eg [Eldad Starck Sapiro]Simultaneous structuretexture inpainting eg [Aujol CaoGousseau Ladjal Masnou Perez Bugeau Bertalmio CasellesSapiro 08-09]
In the following we are especially interested in the inpainting ofstructures like edges and uniformly colored areas in the image usingPDEs and variational approachesLet us consider some examples
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 37 50
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Bertalmio et alrsquos approach 2000
Bertalmio et alrsquos model based on observations about the work ofmuseum artists who restorate old paintings
Principle prolongating the image intensity in the direction of thelevel lines (sets of image points with constant grayvalue) arrivingat the hole This results in solving a discrete approximation of thePDE
ut = nablaperpu middot nabla∆u
solved within the hole D extended by a small strip around itsboundaryTo avoid the crossing of level lines apply intermediate steps ofanisotropic diffusion
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 38 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Masnou amp Morelrsquos disocclusion 1998Masnou and Morel presented a variational technique for removing occlusionsof objects with the goal of image segmentation
Idea connect T-junctions at the occluding boundaries of objects withEuler elastica minimizing curves (A curve is said to be Eulers elastica ifit is the equilibrium curve of the Euler elastica energy
E(γ) =int
γ
(a + bκ2) ds
where ds denotes the arc length element κ(s) the scalar curvature anda b two positive constants)
Basic principle prolongate level lines by minimizing their length andcurvature
transformed into a functional approach by Chan amp Shen Eulers elasticainpainting
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 39 50
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting State of the Art Methods
Chan amp Shen TV inpainting 2001Chan amp Shenrsquos approach is based on the most famous model in imageprocessing the total variation (TV) model
Principle action of anisotropic diffusion inside the inpainting domain rArrpreserves edges and diffuses homogeneous regions and smalloscillations like noise
ut = λnabla middot(nablau
|nablau|
)+ χΩD(f minus u)
Disadvantage level lines are interpolated linearly
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 40 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω D
R(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transport
The fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
The variational approach
J (u) = R(u) +12λ
∥∥χΩD(uminus g)∥∥2
B1rarr min
uisinB2
where
χΩD(x) =
1 Ω D
0 D
is the characteristic function of Ω DR(u) fills in the image content into the missing domain D eg bydiffusion andor transportThe fidelity term only has impact on the minimizer u outside of theinpainting domain due to the characteristic function χΩD
Now for B1 = L2(Ω) we also have
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 41 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
PDE approach
the corresponding Euler-Lagrange equation
minusλnablaR(u) + χΩD(g minus u) = 0 in Ω
the corresponding steepest descent equation for u( t = 0) = g isthe given image
ut = minusλnablaR(u) + χΩD(g minus u) in Ω
in other situations we encounter equations that do not come fromvariational principles such as Cahn-Hilliard- and TV-Hminus1 inpaintingThen the image processing approach is directly given by an(evolutionary) PDE
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 42 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches
no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary
no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
VariationalPDE approach - some remarks
In the setting of such global inpainting approaches no regularity assumptions on the inpainting domain(s) arenecessary no explicit boundary conditions are imposed at the boundarypartD of the inpainting region
These properties make such inpainting approaches applicable to allkinds of damaged images
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 43 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life
rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon
more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models
seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting
Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)
TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting The VariationalPDE Approach
Inpainting review
R(u) =intΩ |nablau|2 dx Harmonic-inpainting
R(u) = |Du|(Ω) TV-inpainting (Chan and Shen 2001)R(u) =
inpainting (Caselles Masnou Morel Sbert 1998 Chan KangShen 2002) Why do we complicate our life rArr Will try to answerthis soon more inpainting models seeking for less complex (curvature term) models with thesame performance than Eulerrsquos elastica inpainting Inpainting for binary images with the Cahn-Hilliard equation(Bertozzi Esedoglu and Gillette 2006)TV-Hminus1 inpainting (M Burger L He C-BS 2008)
so why do we consider higher-order inpainting models
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 44 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λ
Penalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -
Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Second order approaches
Example TV-inpainting
|Du|(Ω) asympint
Ω|nablau| dx
Propagate sharp edges into the damaged domain +
minu
intΩ|nablau| dx lArrrArr min
Γλ
int infin
minusinfinlength(Γλ) dλ
where Γλ = x isin Ω u(x) = λPenalizes length of edges =rArr cannot connect contours acrossvery large distances -Can result in corners of the level lines across the inpaintingdomain -
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 45 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
Example Eulerrsquos elastica inpainting
minu
intΩ(a + b
(nabla middot
(nablau|nablau|
))2)|nablau| dx
lArrrArrminΓλ
intinfinminusinfin(a length(Γλ) + b curvature2(Γλ)) dλ
4Pictures taken from Chan Kang Shen 2002Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 46 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2
[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches (cont)
Example Cahn-Hilliard inpaintingThe inpainted version u of f isin L2(Ω) assumed with any (trivial)extension to the inpainting domain is constructed by following theevolution of
ut = ∆(minusε∆u +1εF prime(u)) + χΩD(f minus u) in Ω
where F (u) is a so called double-well potential eg F (u) = (u2 minus 1)2[Bertozzi et al 06] proved that in the limit λ0 rarrinfin a stationary solutionsolves
∆(ε∆uminus 1εF prime(u)) = 0 in D
u = f on partD
nablau = nablaf on partD
for f regular enough (f isin C2)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 47 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Higher order approaches
ChallengesHigher order equations are very new and little is known aboutthemOften they do not possess a maximum principle or comparisonprincipleFor the proof of well-posedness of higher order inpainting modelsvariational methods are often not applicableNeed of simple but effective modelsNeed of stable and fast numerical solvers
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 48 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
Next time
Cahn-Hilliard inpainting
ut = ∆(minusε∆u +1
εF prime(u)) +
1
λχΩD(f minus u) in Ω
(existence stationary solutions characterization of solutions boundaryconditions)
TV-Hminus1 inpainting (generalization of Cahn-Hilliard inpainting for grayvalueimages)
ut = ∆p +1
λχΩD(f minus u) p isin partTV (u)
(Stationary solution cooperation of transport and diffusion error analysis)
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 49 50
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion
Image Inpainting
State of the Art Methods
The VariationalPDE Approach
Second- Versus Higher-Order PDEs for Inpainting
Image Inpainting Second- Versus Higher-Order PDEs for Inpainting
End of Part I
Thank you for your kind attentionwrite to cbschonliebdamtpcamacuk
Schonlieb (NAM Gottingen) PDEs for Image Inpainting Part I Gottingen-7 January 2010 50 50
Outline
Main Talk
Digital Image Processing
Examples
What is a Digital Image and how do we Process it
Mathematical Image Models
Gaussian Filtering the Heat Equation and Nonlinear Diffusion