chapter 05b image restoration (linear restoration)

University of Ioannina - Department of Computer Science

Image Restoration and Reconstruction(Linear Restoration Methods)

Christophoros Nikou

[email protected]

Digital Image Processing

2

C. Nikou – Digital Image Processing (E12)

Contents

In this lecture we will look at linear image restoration techniques

– Differentiation of matrices and vectors– Linear space invariant degradation– Restoration in absence of noise

• Inverse filter• Pseudo-inverse filter

– Restoration in presence of noise• Inverse filter• Wiener filter• Constrained least squares filter

3


Differentiation of Matrices and Vectors

Notation:

A is a MxN matrix with elements aij.

x is a Nx1 vector with elements xi.

f(x) is a scalar function of vector x.

g(x) is a Mx1 vector valued function of vector x.

4


Differentiation of Matrices and Vectors (cont...)

Scalar derivative of a matrix.


111

1

N

MNM

aa

t t

taa

t t

A

5



Vector derivative of a function (gradient).


f(x) is a scalar function of vector x.

1

T

N

f f ff

x x

xx

6



Vector derivative of a vector (Jacobian):


g(x) is a Mx1 vector valued function of vector x.

1 1

1

1

N

M M

N

g g

x x

g g

x x

g

x

7



Some useful derivatives.


b is a Nx1 vector with elements bi.

It is the derivative of the scalar valued function bTx with respect to vector x.

T

b x b

x

8






If A is symmetric:

T T

x Ax A A x

x

2T

x Ax Ax

x

9





b is a Nx1 vector with elements bi.


It may be proved using the previous properties.

22 T

Ax b A Ax bx

10


Linear, Position-Invariant Degradation

We now consider a degraded image to be modelled by:

where h(x, y) is the impulse response of the degradation function ( i.e. point spread function blurring the image).

The convolution implies that the degradation mechanism is linear and position invariant (it depends only on image values and not on location).

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

11


Linear, Position-Invariant Degradation (cont...)

In the Fourier domain:

where multiplication is element-wise.

In matrix-vector form:

where H is a doubly block circulant matrix and f,g, and η are vectors (lexicographic ordering).

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

g = Hf + η

12


Linear, Position-Invariant Degradation (cont...)

If the degradation function is unknown the problem of simultaneously recovering f(x,y) and h(x,y) is called blind deconvolution.

g = Hf + η

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

13


Linear Restoration

Using the imaging system

we want to estimate the true image from the degraded observation with known degradation H.

A linear method applies an operator (a matrix) P to the observation g to estimate the unobserved noise-free image f:

f̂ = Pg

g = Hf + η

14


Restoration in Absence of NoiseThe Inverse Filter

When there is no noise:

an obvious solution would be to use the inverse filter:

yielding

g = Hf

-1P = H

ˆ -1 -1f = Pg = H g = H Hf = f

15


Restoration in Absence of NoiseThe Inverse Filter (cont...)

For a NxN image, H is a N2xN2 matrix!

To tackle the problem we transform it to the Fourier domain.

H is doubly block circulant and therefore it may be diagonalized by the 2D DFT matrix W:

ˆ -1f = H g

-1H = W ΛW

16



where

Therefore:

-1H = W ΛW

{ (1,1),..., ( ,1), (1,2),... ( , )}diag H H N H H N NΛ

f̂ = Pg ˆ -1f = H g ˆ -1 -1f = (W ΛW) g

ˆ -1 -1f = W Λ Wg ˆ -1 -1Wf = WW Λ Wg

ˆ -1F = Λ G

17



Which is the vectorized form of the DFT of the image:

Take the inverse DFT and obtain f(m,n).

Problem: what happens if H(k,l) has zero values?

Cannot perform inverse filtering!

ˆ -1F = Λ G( , )ˆ ( , )( , )

G k lF k l

H k l =

18


Restoration in Absence of NoiseThe Pseudo-inverse Filter

A solution is to set:

which is a type of pseudo-inversion.

Notice that the signal cannot be restored at locations where H(k,l)=0.

( , ), ( , ) 0ˆ ( , )( , )

0 , ( , ) 0

G k lH k l

H k lF k l

H k l

=

19


Restoration in Absence of NoiseThe Pseudo-inverse Filter (cont...)

A pseudo-inverse filter also arises by the unconstrained least squares approach.

Find the image f, that, when it is blurred by H, it will provide an observation as close as possible to g, i.e. It minimizes the distance between Hf and g.

20


Restoration in Absence of NoiseThe Pseudo-inverse Filter (cont...)

This distance is expressed by the norm:

min ( )Jf

f 20

Hf - g

f

2 0T H Hf - g 2 2T T H Hf H g

1T T f H H H g

0J

f

2( )J f Hf - g

21


Restoration in Presence of NoiseThe Inverse Filter

Recall the imaging model with spatially invariant degradation and noise

g = Hf + η

( , ) ( , ) ( , ) ( , )g x y h x y f x y x y

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

22


Restoration in Presence of NoiseThe Inverse Filter (cont...)

Applying the inverse filter in the Fourier domain:

Even if we know H(k,l) we cannot recover F(k,l) due to the second term.

If H(k,l) has small values the second term dominates (it goes to infinity if H(k,l)=0!).

( , ) ( , ) ( , ) ( , )G k l H k l F k l N k l

( , )ˆ ( , ) ( , ) ( , )

N k lF k l F k l

H k l

23



One approach to get around the problem is to limit the ratio G(k,l) / H(k,l) to frequencies near the origin that have lower probability of being zero.

We know that H(0,0) is usually the highest value of the DFT.

Thus, by limiting the analysis to frequencies near the origin we reduce the probability of encountering zero values.

24



Blurring degradation

25


Restoration in Presence of NoiseThe Inverse Filter (cont...)Inverse filter with cut-off

26


Restoration in Presence of NoiseWiener Filter

So far we assumed nothing about the statistical properties of the image and noise.

We now consider image and noise as random variables and the objective is to find an estimate of the uncorrupted image f such that the mean square error between the estimate and the image is minimized:

where E[x] is the expected value of vector x.

2

ˆˆmin ( )E f

f f

27


Restoration in Presence of NoiseWiener Filter (cont...)

Recall also the definition of the correlation matrix between two vectors x and y:

We assume that the image and the noise are uncorrelated:

1 1 1 2 1

1 2

NT

N N N N

E x y E x y E x y

E

E x y E x y E x y

xyR xy

ηf fηR R 0

28



We are looking for the best estimate:

Let’s confine our estimate to be obtainable by a linear operator on the observation:

and the goal is to find the best matrix P.

2

ˆˆmin ( )E f

f f

ˆ f Pg

29



Denoting by the n-th row of P:

2 22ˆ ˆ ˆ( ) ( )J E E E f f f f f f Pg

2 2ˆ( ) T Tn n n n

n n

J E E f f p g f p g

Tnp

30



ˆ( )TT T

n n n nn

J E f f p g f p g

T T T T T Tn n n n n n n n

n

E f f p gf f g p p gg p

T T T T T Tn n n n n n n n

n

E E E E f f p gf f g p p gg p

2n n n

T Tn n n

n

f f gf ggR p R p R p

31



We can now minimize the sum with respect to each term:

2 0n n n

T Tn n n

n

f f gf ggR p R p R pp

2 2 0n n gf ggR R p 1

nn gg gfp R R

1

n

Tn

f g ggp R R

32



Assembling the rows together:

We have to compute the two matrices:

1 fg ggP R R

( )( )T TE E ggR gg Hf η Hf η

T T T T T TE Hff H Hfη ηf H η η

T T ff fη ηf ηηHR H HR R H R

33



Assumming noise is uncorrelated with image:

Also,

Finally the matrix we are looking for is

( ) ...T T TE E fg ffR fg f Hf η R H

T gg ff ηηR HR H R

11 T T fg gg ff ff ηηP R R R H HR H R

34



The estimated uncorrupted image is

which may be also expressed as

This result is known as the Wiener filter or the minimum mean square error (MMSE) filter.

ˆ f Pg 1ˆ T T ff ff ηηf R H HR H R g

11 1 1ˆ T T ηη ff ηηf H R H R H R g

35



Special cases:

No blur (H=I, g=f+η):

No noise (Rηη=0, g=Ηf):

This is the inverse filter.

No blur, no noise (H=I, Rηη=0):

Do nothing on the observation.

1ˆ ff ff ηηf R R R g

1ˆ f Η g

ˆ f g

36



The size of the matrix to be inverted poses difficulties and Wiener filter is implemented in the Fourier domain.

This occurs when H is doubly block circulant (represents convolution) and the image f and noise η are wide-sense stationary (w.s.s).

Definition of a w.s.s. signal:

1) E[f(m,n)]=μ, independent of m,n.

2) E[f(m,n)f(k,l)]=r(m-k,n-l), independent of location.

37



Reminder: the inverse DFT complex exponential matrix diagonalizes any circulant matrix:

The columns of W-1 are the eigenvectors of any circulant matrix H.

The corresponding eigenvalues are the DFT values of the signal producing the circulant matrix.

Remember also that

and that

1 HH W Λ W

T W W

1 1T W W

38



We will employ the following relations:

If H is real:

1 HH W Λ W

1 1TT H HH W Λ W WΛ W

* * *1 * * 1T T H HH H WΛ W W Λ W

1 * 1 *1( )N

N H HW Λ W W Λ W

39



The Wiener solution is now transformed to the Fourier domain:

Notice that the matrices are diagonal.

1ˆ T T ff ff ηηf R H HR H R g

11 1 * 1 1 1 * 1( )( ) ( )( )( ) ( ) ff H H ff H ηηW Λ W W Λ W W Λ W W Λ W W Λ W W Λ W g

11 * 1 *( ) ff H H ff H ηηW Λ Λ W W Λ Λ Λ Λ g

* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg

40



Sff (k,l)=DFT(Rff (m,n)) is the power spectrum of the image f (m,n).

Sηη(k,l)=DFT(Rηη (m,n)) is the power spectrum of the noise η(m,n).

* * 1ˆ ( ) ff H H ff H ηηWf Λ Λ Λ Λ Λ Λ Wg

*

2

( , ) ( , )( , ) ( , )

( , ) ( , ) ( , )

ff

ff

S k l H k lF k l G k l

S k l H k l S k l

41



If Sff (k,l) is not zero we may define the Signal to Noise Ratio in the frequency domain:

and the Wiener filter becomes:

( , )SNR( , )

( , )ffS k l

k lS k l

*

2 -1

( , )( , ) ( , )

( , ) SNR ( , )

H k lF k l G k l

H k l k l

42



A well known estimate of Sff (k,l) is the periodogram (the ML estimate of Rff when f ia assumed Gaussian):

In practice, as F(k,l) is unknown, we use

21( , ) ( , )ffS k l F k l

MN

21ˆ ( , ) ( , )ffS k l G k lMN

43



44



Inverse WienerDegraded image

Noise variance one order of magnitude less.

Noise variance ten orders of magnitude less.

45


Restoration in Presence of NoiseConstrained Least Squares Filter

When we do not have information on the power spectra the Wiener filter is not optimal.

Another idea is to introduce a smoothness term in our criterion.

We define smoothness by the quantity

where Q is a high pass filter operator, e.g. the Laplacian, (Q is a doubly block circulant matrix representing convolution by the Laplacian).

We look for smooth solutions minimizing

2Qf

2Qf

46


Restoration in Presence of NoiseConstrained Least Squares Filter (cont...)

We have the following constrained least squares (CLS) optimization problem:

Minimize subject to

yielding the Lagrange multiplier minimization of the function:

2 2( , )J f Hf g Qf

Smoothness termData fidelity term

2Qf Hf g

47



2 ( ) 2 0T T H Hf g Q Qf( , ) 0J

f

f

1T T T

f H H Q Q H g

Parameter λ controls the degree of smothness:

10, T T

f H H H g pseudo-inverse, ultra rough solution

, 0 f ultra smooth solution

48



In the Fourier domain, the constrained least squares filter becomes:

Keep always in mind to zero-pad the images properly.

*

2 2

( , )( , ) ( , )

( , ) ( , )

H k lF k l G k l

H k l Q k l

49



Low noise: Wiener and CLS generate equal results.High noise: CLS outperforms Wiener if λ is properly selected.It is easier to select the scalar value for λ than to approximate the SNR which is seldom constant.

50


Restoration Performance Measures

Mean square error (MSE):

1 1 2

0 0

1 ˆMSE ( , ) ( , )M N

m n

f m n f m nMN

Original image f and restored image f̂

51


Restoration Performance Measures (cont...)

The Signal to Noise Ratio (SNR) considers the difference between the two images as noise:

1 12

0 01 1 2

0 0

ˆ ( , )SNR

ˆ( , ) ( , )

M N

m nM N

m n

f m n

f m n f m n

chapter 05b image restoration (linear restoration)

Documents