CS654: Digital Image Analysis

Lecture 22: Image Restoration - II

Recap of Lecture 21

• Image restoration vs. enhancement

• What is restoration

• Image restoration model

• Continuous, discrete formulation

• Point spread function

• Noise

Outline of Lecture 22

• 2D discrete domain modeling

• Restoration with only noise

• Restoration with degradation

• Blind deconvolution

• Motion Blur

• Inverse Filtering

Image restoration pipeline

𝑓 (𝑥 , 𝑦) 𝐻 (𝑥 , 𝑦 )∗ 𝜂(𝑥 , 𝑦 )+¿𝑔 (𝑥 , 𝑦 )=¿ 𝑔 (𝑥 , 𝑦 )→ �̂� (𝑥 , 𝑦)

�̂� (𝑥 , 𝑦)≅ 𝑓 (𝑥 , 𝑦 )Target

Images: Gonzalez & Woods, 3rd edition

2D Discrete Domain Representation

𝑔=𝐻𝑓 +𝜂

Vector of dimension

Vector of dimension

Matrix of dimension

𝑀𝑁

𝑀𝑁

0

…

𝑀

𝑀…0

�́�0 �́�1

�́�𝑀− 1 �́�0

…

… …

…

⋱

Block Circulant matrix

2D Discrete Domain Representation

�́� 𝑗=[ h( 𝑗 ,0) h ( 𝑗 ,𝑁−1) … h( 𝑗 ,1)h ( 𝑗 ,1) h ( 𝑗 ,0) … h( 𝑗 ,2)… … … …

h ( 𝑗 ,𝑁−1) h ( 𝑗 ,𝑁−2) … h ( 𝑗 ,0)]

Circulant matrix of dimension

Image restoration: 1D case

�⃗� (𝑘 )=[1

𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 𝑘]𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 2𝑘]

⋮

𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 (𝑀−1)𝑘] ] ScalarLet,

𝜆 (𝑘 )=h (0 )+h (𝑀−1 )𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 𝑘]+…+h(1)𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 (𝑀−1)𝑘]

VectorLet,

𝜆 (𝑘 ) �⃗� (𝑘)?What happens if we do

Image restoration

𝜆 (𝑘 ) �⃗�(𝑘)𝐻�⃗� (𝑘 )=¿

: Eigen vector of ; : Its corresponding Eigen value

𝜆 (𝑘 )=h (0 )+h (𝑀−1 )𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 𝑘]+…+h(1)𝑒𝑥𝑝[ 𝑗 2𝜋𝑀 (𝑀−1)𝑘]

𝑣 (𝑘 )=∑𝑛=0

𝑁−1

𝑢 (𝑛 )exp [ − 𝑗2𝜋𝑘𝑛𝑁 ]DFT : 1D DFT of the H matrix

𝐻=[ h (0 ) h (𝑀−1 ) h (𝑀 −2 ) … h (1 )h (1 ) h (0 ) h(𝑀 −1) … h(2)… … … … …

h(𝑀−1) h(𝑀−2) h(𝑀−3) … h (0)]

Restoration in presence of only noise

𝑔 (𝑥 , 𝑦 )= 𝑓 (𝑥 , 𝑦 )+𝜂(𝑥 , 𝑦 )Spatial domain:

𝐺 (𝑢 ,𝑣 )=𝐹 (𝑢 ,𝑣 )+𝑁 (𝑢 ,𝑣 )Frequency domain:

Spatial filtering is the choice when additive random noise is present

• Mean filter

• Median Filter (order statistics),max, min, mid-point

• Bandpass, band-reject filters

• Adaptive filters

Examples

In presence of degradation

• Degradation (spatial domain) = conv(PSF, image) + noise

• Degradation (Freq. domain) = H(PSF).H(image) + H(noise)where H=transformation function

• Image deconvolution

• Deconvolution filters

Degradation estimation

Estimation

Observation Experimentation Mathematical Modeling

Blind deconvolution

A technique that permits recovery of the target scene from distorted image(s) in the presence of a unknown point spread function (PSF)

𝒈 𝒔(𝒙 , 𝒚 )

Estimation by Observation

𝐺 (𝑢 ,𝑣 )=𝐻 (𝑢 ,𝑣 ) 𝐹 (𝑢 ,𝑣 )+𝑁 (𝑢 ,𝑣)

𝑔 (𝑥 , 𝑦 )=h (𝑥 , 𝑦 )∗ 𝑓 (𝑥 , 𝑦 )+𝜂 (𝑢 ,𝑣)Spatial domain:

Frequency domain:

�̂� (𝑥 , 𝑦)Processed sub-image:

𝐻 𝑠 (𝑢 ,𝑣 )=𝐺𝑠 (𝑢 ,𝑣)

𝐹 𝑠 (𝑢 ,𝑣)

How to choose

What is the dimension of ?

Estimation by Experimentation

Scene Acquired Image

Degradation function Impulse response

1. Impulse simulation

2. Degradation function estimation

Simulated impulse Impulse response

At F)(

𝐻 (𝑢 ,𝑣 )=𝐺(𝑢 ,𝑣)𝐹 (𝑢 ,𝑣)

=𝐺(𝑢 ,𝑣)

𝐴

Images: Gonzalez & Woods, 3rd edition

Estimation by Mathematical Modeling

𝐻 (𝑢 ,𝑣 )=𝑒𝑥𝑝 [−𝑘 (𝑢2+𝑣2 )56 ]

Physical characteristics of atmospheric turbulence

=nature of turbulence

𝒌=𝟎 .𝟎𝟎𝟐𝟓

𝒌=𝟎 .𝟎𝟎𝟏 𝒌=𝟎 .𝟎𝟎𝟎𝟐𝟓

Images: Gonzalez & Woods, 3rd edition

With motion

Camera

Estimation from Basic PrinciplesWithout motion

Camera

How to estimate the degradation if undergoes motion?

Uniform linear motion blur

=input image with linear motion

=Time varying component along direction

=Time varying component along direction

=Duration of the exposer

𝑔 (𝑥 , 𝑦 )=∫0

𝑇

𝑓 (𝑥−𝑥0 (𝑡 ) , 𝑦− 𝑦0 (𝑡 ) )𝑑𝑡

= Blurred image

Uniform linear motion blur

𝑔 (𝑥 , 𝑦 )=∫0

𝑇

𝑓 (𝑥−𝑥0 (𝑡 ) , 𝑦− 𝑦0 (𝑡 ) )𝑑𝑡

𝐹 (𝑢 ,𝑣 )=∫−∞

∞

∫−∞

∞

𝑓 (𝑥 , 𝑦 )𝑒𝑥𝑝 [− 𝑗2𝜋(𝑢𝑥+𝑣𝑦) ]𝑑𝑥𝑑𝑦2-D Fourier Transform:

𝐺 (𝑢 ,𝑣 )=∫−∞

∞

∫−∞

∞ [∫0

𝑇

𝑓 (𝑥−𝑥0 (𝑡 ) , 𝑦− 𝑦0 (𝑡 ) )𝑑𝑡]𝑒𝑥𝑝 [− 𝑗 2𝜋 (𝑢𝑥+𝑣𝑦) ]𝑑𝑥𝑑𝑦

𝐺 (𝑢 ,𝑣 )=∫0

𝑇 [∫−∞

∞

∫−∞

∞

𝑓 (𝑥−𝑥0 (𝑡 ) , 𝑦− 𝑦0 (𝑡 ) )𝑒𝑥𝑝 [− 𝑗2𝜋 (𝑢𝑥+𝑣𝑦 )]𝑑𝑥 𝑑𝑦 ]𝑑𝑡

𝑮 (𝒖 ,𝒗 )=∫𝟎

𝑻

𝑭 (𝒖 ,𝒗 )𝒆𝒙𝒑 [− 𝒋𝟐𝝅 (𝒖𝒙𝟎 (𝒕 )+𝒗 𝒚𝟎(𝒕 ))]𝒅𝒕

Uniform linear motion blur

𝐺 (𝑢 ,𝑣 )=∫0

𝑇

𝐹 (𝑢 ,𝑣 )𝑒𝑥𝑝 [− 𝑗2𝜋(𝑢𝑥0 (𝑡 )+𝑣 𝑦0 (𝑡)) ]𝑑𝑡

𝐺 (𝑢 ,𝑣 )=𝐹 (𝑢 ,𝑣 )∫0

𝑇

𝑒𝑥𝑝 [− 𝑗2𝜋(𝑢𝑥0 (𝑡 )+𝑣 𝑦0 (𝑡)) ]𝑑𝑡

is independent of

Let, ⇒𝐺 (𝑢 ,𝑣 )=𝐹 (𝑢 ,𝑣)𝐻 (𝑢 ,𝑣 )

Let, ,

𝐻 (𝑢 ,𝑣 )=∫0

𝑇

𝑒𝑥𝑝 [− 𝑗2𝜋(𝑢𝑎𝑡𝑇 )]𝑑𝑡

Example

Images: Gonzalez & Woods, 3rd edition

Input Image Motion Blurred Image (a=b=0.1, T=1)

Inverse Filtering

• Simplest approach for image restoration – direct inverse filtering

�̂� (𝑢 ,𝑣 )= 𝐺(𝑢 ,𝑣 )𝐻 (𝑢 ,𝑣)

𝐺 (𝑢 ,𝑣 )=𝐹 (𝑢 ,𝑣 )+𝑁 (𝑢 ,𝑣 )Frequency domain:

�̂� (𝑢 ,𝑣 )=𝐹 (𝑢 ,𝑣 )+𝑁 (𝑢 ,𝑣)𝐻 (𝑢 ,𝑣)

• is unknown

• If is small, is dominated by the ratio

ExampleFull filter CR = 40

CR = 85CR=70

Input image

Thank youNext Lecture: Image Restoration