colorado school of mines image and multidimensional signal...

Colorado School of Mines Department of Electrical Engineering and Computer Science

Colorado School of Mines

Image and Multidimensional Signal

Processing

Professor William Hoff

Department of Electrical Engineering and Computer

Science

http://inside.mines.edu/~whoff/

http://inside.mines.edu/~whoff/


Spatial Filtering

2

Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science

Spatial filtering

• Main idea

– Define a neighborhood of a pixel

– Define an operation on pixels in the neighborhood

– Output pixel value = result of the operation

• Topics

– Linear filtering

• Correlation, convolution

• Normalized cross correlation

– Important examples

• Smoothing filters (box average, Gaussian)

• Sharpening filters (Laplacian, Sobel)

– Non-linear filters

• Order-statistics filters (primarily median filter)

3


Linear Spatial Filtering

• Create a filter or mask w, size m x n

• Apply to image f, size M x N

• Compute the sum of products of mask coefficients with

corresponding pixels under mask

• Slide mask over image, apply at each point

• It is a linear operation

/2 /2

/2 /2

( , ) ( , ) ( , )

( , ) ( , )

m n

s m t n

g x y w s t f x s y t

w x y f x y

( , ) ( , ) ( , ) ( , )

( , ) ( , ) ( , ) ( , ) ( , ) ( , ) ( , )

w x y a f x y a w x y f x y

w x y f x y g x y w x y f x y w x y g x y

4

Also called “cross-

correlation”


2/

2/

2/

2/

),(),(),(m

ms

n

nt

tysxftswyxg

5


Example

• 3x3 averaging (“box”) filters

• Do manual calculation on a

corner

6

0 0 0 0 0 0 0 ..

0 0 0 0 0 0 0 ..

0 0 0 0 0 0 0 ..

0 0 0 9 9 9 9 ..

0 0 0 9 9 9 9 ..

0 0 0 9 9 9 9 ..

0 0 0 9 9 9 9 ..

: : : : : : : ..

( , ) ( , ) ( , )g x y w x y f x y ( , )f x y


Example

• 3x3 averaging filters

• Manual calculation on

corner

7


Matlab Examples

• fspecial

– useful to create filters

• imfilter

– does cross correlation

8

clear all

close all

% Synthetic image of a white square

I = zeros(200,200);

I(50:150, 50:150) = 1;

imshow(I,[]);

% Apply 3x3 box filter

w = [ 1 1 1; 1 1 1; 1 1 1]/9;

I2 = imfilter(I, w);

imtool(I2,[]);

% Apply 15x15 box filter

w = fspecial('average', [15 15]);


imtool(I3,[]);

% Show that averaging reduces noise

I = I + randn(200,200)*0.5;

imshow(I,[]);

w = fspecial('average', [5 5]);


figure, imshow(I4,[]);


Sharpening Spatial Filters – First Derivative

• First derivative

– In discrete form (can also do central difference)

– As a filter

• Sobel operators – combine smoothing with derivative

0

( , ) ( , ) ( , )lim

f x y f x y f x y

x

( , )( 1, ) ( , )

f x yf x y f x y

x

-1 +1

( 1) ( 1) / 2f

f x f xx

-.5 0 +.5

-1 0 1

-2 0 2

-1 0 1

-1 -2 -1

0 0 0

1 2 1

dx filter dy filter

9


Example

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

-1 0 1

-2 0 2

-1 0 1

( , ) ( , ) ( , )g x y w x y f x y

( , )w x y

( , )f x y

10


Example

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

0 0 1 1 1 1 0 0

-1 0 1

-2 0 2

-1 0 1

( , ) ( , ) ( , )g x y w x y f x y

( , )w x y

( , )f x y

11


Gradient

yf

xff

• Compute gradient components using first derivative operators

• Gradient magnitude indicates the location of edges in the image

• Gradient direction shows direction of edge

• We can use the Sobel operators for df/dx, df/dy

2/122

y

f

x

ff

1tanf y

f x

12


13

Matlab example

• Note - the result is not quite

correct

• We should first convert the

input image to type “double”

before applying “imfilter”

(why?)

clear all

close all

I = imread('moon.tif');

imshow(I,[]);

% Create Sobel filters

hy = [ % can also do hy = fspecial('sobel');

-1 -2 -1;

0 0 0;

+1 +2 +1];

hx = hy';

Ix = imfilter(I, hx);

Iy = imfilter(I, hy);

figure;

subplot(1,2,1), imshow(Ix,[]);

subplot(1,2,2), imshow(Iy,[]);


14

Matlab example (continued)

>> help atan2

atan2 Four quadrant inverse tangent.

atan2(Y,X) is the four quadrant arctangent of the real parts of the

elements of X and Y. -pi <= atan2(Y,X) <= pi.

See also atan.

Reference page in Help browser

doc atan2

% Compute gradient magnitude (note the use of ".")

Ig = (Ix.^2 + Iy.^2) .^ 0.5;

figure, imshow(Ig, []);

% Compute gradient direction

Iang = atan2(Iy,Ix);

figure, imshow(Iang, []);

% Only display angles where gradient magnitude is large

Mask = (Ig > 100);

figure, imshow(double(Mask) .* Iang, []);

colormap jet


Sharpening Spatial Filters – 2nd derivative

• Second derivative

– As a filter

• Two dimensional – the Laplacian

– As a filter

2

2

( , ) ( , )( 1, ) 2 ( , ) ( 1, )

f x y f x yf x y f x y f x y

x x x

+1 -2 +1

2

2

2

22

y

f

x

ff

0 +1 0

+1 -4 +1

0 +1 0

15

0 0 0

+1 -2 +1

0 0 0

0 +1 0

0 -2 0

0 +1 0

2

2

x

2

2

x

= +


High Boost Filtering

• We combine original image with results of Laplacian – result is

the original image with enhanced detail

– Also called unsharp masking

• In general

2( , ) ( , )g x y f x y f 0 -1 0

-1 5 -1

0 -1 0

16

),(),(),( yxfyxAfyxg


17


Vector representation of correlation

• Correlation is a sum of products of corresponding terms

• We can think of correlation as a dot product of vectors w and f

• If images w and f are similar, their vectors are aligned

1 1 2 2

1

mn mn

mnT

k k

k

c w f w f w f

w f

w f

/2 /2

/2 /2

( , ) ( , ) ( , )

( , ) ( , )

m n

s m t n

c x y w s t f x s y t

w x y f x y

18


Correlation can do matching

• Let w(x,y) be a template sub-image

• Try to find an instance of w in image f(x,y)

• The correlation score c(x,y) is high where w matches f

/2 /2

/2 /2

( , ) ( , ) ( , )

( , ) ( , )

m n

s m t n

c x y w s t f x s y t

w x y f x y

19


20


Template Matching (continued)

• Since f is not constant everywhere, we need to normalize

• The result is between 0..1 (for positive valued images)

• We can get better precision by subtracting off means

• This result is between -1 .. +1

T

c w f

w f

,

1/2

22

, ,

[ ( , ) ][ ( , ) ]

( , )

[ ( , ) ] [ ( , ) ]

s t

s t s t

w s t w f x s y t f

c x y

w s t w f x s y t f

• This is the

“normalized

correlation

coefficient”

21

22

3

2

2

2

1 where Nwwww w


Normalized Cross-correlation

• Normalized cross-correlation (values range from -1 .. +1)

• where

– w(s,t) is a subimage, size mxn

– f(x,y) is a large image, size MxN

– sums are taken over s=1..m, t=1..n

– is the local mean of f, in a mxn neighborhood around (x,y)

,

1/2

22

, ,

[ ( , ) ][ ( , ) ( , )]

( , )

[ ( , ) ] [ ( , ) ( , )]

s t

s t s t

w s t w f x s y t f x y

c x y


w(s,t) f(x,y)

Size mxn

Size MxN

c(x,y) Correlation

scores

,

1( , ) ( , )

s t

f x y f x s y tmn

( , )f x y

Size MxN 22

See pg. 870 in

the Gonzalez &

Woods textbook


Matlab Example

• Crop a small subimage from left image, find its match in the right image

I1 = imread('test000.jpg');

T = imcrop(I1, [66, 213, 86, 233]);

I2 = imread('test012.jpg');

C = normxcorr2(T, I2);

% The scores are in an image that is slightly bigger than the original

% image ... it is expanded by half the size of the template in all

% directions. So we will crop out the center portion.

Csub = imcrop(C, [(size(T,2)-1)/2+1 (size(T,1)-1)/2+1 size(I2,2)-1 size(I2,1)-1]);

cmax = max(Csub(:));

[y x] = find(Csub==cmax);

23


Doing cross correlation in Matlab without normxcorr2

• The easiest way is to implement the equation directly, using “for”

loops

• You scan through the image ... at every pixel (x,y):

– Extract the subimage fs(x,y) from the image f surrounding x,y

– Subtract off the local image mean value at (x,y)

– Compute the sum of the product of (fs-fmean) and (w-mean2(w); this is the

numerator

– The denominator can be similarly computed

– Dividing the two gives you the value of the normalized cross correlation at pixel

(x,y) in the ouput

• Note - there are faster ways to do this, using Matlab’s ability to

operate on whole matrices

24



(continued)

• Numerator

– where

• Then

• The first term is the correlation of (w-wmean) and f. The second term is zero because

• So in Matlab: N = imfilter(f,w-mean2(w));

,

[ ( , ) ][ ( , ) ( , )]s t

N w s t w f x s y t f x y

is the mean of all pixels in w; it can be computed once. It is a scalar.

( , ) is the local mean of pixels in f, in a mxn window centered at (x,y).

This can be represented as an image of size MxN.

w

f x y

,

, ,

[ ( , ) ][ ( , ) ( , )]

[ ( , ) ] ( , ) ( , ) [ ( , ) ]

s t

s t s t


w s t w f x s y t f x y w s t w

, , ,

[ ( , ) ] ( , ) 0s t s t s t

w s t w w s t w mnw mnw

25



(continued)

• Denominator

• In Matlab, you can compute fmean at every pixel by filtering with a mxn rectangle of 1’s (and dividing by mxn):

[m n] = size(w);

fmean = imfilter(f, ones(m,n))/(m*n);

• The first term is a scalar, and can be computed once. We rewrite the second term:

1/2

22

, ,

[ ( , ) ] [ ( , ) ( , )]s t s t

D w s t w f x s y t f x y

2

,

2 2

, , ,

2 2 2

,

2 2

,

[ ( , ) ( , )]

( , ) 2 ( , ) ( , ) ( , )

( , ) 2( ) ( , ) ( ) ( , )

( , ) ( ) ( , )

s t

s t s t s t

s t

s t

f x s y t f x y

f x s y t f x y f x s y t f x y

f x s y t mn f x y mn f x y

f x s y t mn f x y

This can be computed by filtering again with a mxn rectangle of 1’s; i.e., imfilter(f.^2, ones(m,n)) 26


Correlation and Convolution

• Correlation

• Convolution

– Note that

– So to implement convolution we can reflect w(x,y) and then do correlation

/2 /2

/2 /2

( , ) ( , ) ( , )

( , ) ( , )

m n

s m t n


w x y f x y

/2 /2

/2 /2

( , ) ( , ) ( , )

( , ) ( , )

m n

s m t n


w x y f x y

/2 /2 /2 /2

/2 /2 /2 /2

( , ) ( , ) ( , ) ( , )m n m n

s m t n s m t n

w s t f x s y t w s t f x s y t

27

• Convolution is very

similar to

correlation, except

that we flip one of

the functions

• Also a linear

operation


1D Examples

28

Figure 3.29


1D Examples

29


2D Examples

30


2D Examples

31


Properties

• Convolution is

– Commutative

f * g = g * f

– Associative

f * (g * h) = (f * g) * h

– Distributive

f * (g + h) = f * g + f * h

• Correlation is distributive but not commutative or associative

• Convolution of a filter h(x,y) with an impulse d(x,y) yields h(x,y)

32


Gaussian – an important smoothing filter

• 1D

• 2D

2

221

( )2

x

h x e

2 2

222

1( , )

2

x y

h x y e

-10 -8 -6 -4 -2 0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

10

20

30

40

0

10

20

30

40

0

0.002

0.004

0.006

0.008

0.01

Notes:

is the standard

deviation

Values should sum to 1

Mask size should be at

least ±3

33


Convolving with two Gaussians

• You can convolve a function with a Gaussian, then convolve again

with a Gaussian. The effect is the same as convolving with a single,

larger Gaussian

• Why? Since convolution is associative

• and if

• then

2 2

2 21 22 2

1 2

1 2

1 1,

2 2

x x

g e g e

2

2 21 22

1 22 2

1 2

1

2

x

g g e

1 2 1 2g g f g g f

i.e., it is a Gaussian

with standard

deviation

2 2

1 2new

34


Proof

• You can prove this from the definition of convolution

• And use the idea that

where the last term does not depend on t and the first one is just

another Gaussian, but centered around a different point.

( ) ( )f g f t g t x dt

2 22 2 2 22 /22 2 /2t x t xt t tx x xe e e e e

35


Interesting fact

• If you repeatedly convolve an image N times with any

filter, the effect (for large N) is that of convolving with a

Gaussian

• This comes from the Central Limit Theorem: “The sum

of N mutually independent random variables with zero

means and finite variances tends toward the normal

probability distribution”

• And the fact that the pdf of the sum of two independent

random variables is the convolution of their pdf’s


Separable kernels (filter masks)

• If a 2D correlation (or convolution) kernel can be separated into two

1D kernels, the operation is much faster

– If

– then

• Namely, do a 1D correlation with h2(y) along the individual columns

of the input image, then do a 1D correlation with h1(x) along the

rows of the result from the previous step

• If h(x,y) is nxn, and f(x,y) is NxN

– Cost of a 2D correlation?

– Cost of two 1D correlations?

)()(),( 21 yhxhyxh

1 2( , ) ( , ) ( ) ( ) ( , )h x y f x y h x h y f x y

37


Separable kernels (filter masks)

• If a 2D correlation (or convolution) kernel can be separated into two

1D kernels, the operation is much faster

– If

– then

• Namely, do a 1D correlation with h2(y) along the individual columns

of the input image, then do a 1D correlation with h1(x) along the

rows of the result from the previous step

• If h(x,y) is nxn, and f(x,y) is NxN

– Cost of a 2D correlation?

– Cost of two 1D correlations?

)()(),( 21 yhxhyxh

1 2( , ) ( , ) ( ) ( ) ( , )h x y f x y h x h y f x y Prove this

(HW2)

38


Nonlinear Filters

• Median filter - a non-linear smoothing filter

• Steps:

– Order (sort) pixel values within neighborhood

– Replace center with some value based on the ordering (i.e., median)

• Advantages

– Especially good for reducing impulse, or shot-and-pepper noise

– Doesn’t blur sharp edges in the image

1D noisy step edge median filtered, filter width=7 0 5 10 15 20 25 30 35 40 45 50

-2

0

2

4

6

8

10

12

0 5 10 15 20 25 30 35 40 45 500

2

4

6

8

10

12

Matlab

functions

medfilt1,

medfilt2,

ordfilt2

39


Example on ramp edge

1: 4,6 :10( )

20 5

x xf x

x

-5 0 5 10 15

0

5

10

15

20

25

40


Example on ramp edge

1: 4,6 :10( )

20 5

x xf x

x

-5 0 5 10 15

0

5

10

15

20

25

41


42


Examples - Matlab

• Read image “eight.tif”

– Add noise: imnoise

– Compare imfilter, medfilt2

43

clear all

close all

I = imread('eight.tif');

imshow(I,[]);

Inoisy = imnoise(I, 'salt & pepper', 0.05);

imtool(Inoisy,[]);

I1 = imfilter(Inoisy, fspecial('average'));

imshow(I1, []);

I2 = medfilt2(Inoisy);

imshow(I2, []);


Summary

• A spatial filter operates on a neighborhood of a pixel.

The output pixel value is the result of the operation.

• A “linear” spatial filter performs a sum of products of the

neighborhood values with the corresponding filter values.

• The median filter is an example of a “non-linear” filter.

• Filters can be used for smoothing and sharpening.

44


Questions

• Is the 2D Gaussian filter separable? Why or

why not?

• What is the difference between convolution and

correlation?

• Why might the median filter be preferable for

some situations?

45

colorado school of mines image and multidimensional signal...

Documents