lecture8 sampling f11 - new york universityyao/el5123/lecture8... · 2012. 11. 14. · • the...

53
Image Sampling and Image Sampling and Resizing Yao Wang Polytechnic Institute of NYU, Brooklyn, NY 11201 With contribution from Zhu Liu Partly based on A. K. Jain, Fundamentals of Digital Image Processing

Upload: others

Post on 16-Aug-2021

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Image Sampling andImage Sampling and Resizing

Yao WangPolytechnic Institute of NYU, Brooklyn, NY 11201

With contribution from Zhu LiuPartly based on A. K. Jain, Fundamentals of Digital Image Processing

Page 2: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Lecture Outline

• IntroductionN q ist sampling and interpolation• Nyquist sampling and interpolation theorem

• Common sampling and interpolation filters• Sampling rate conversion of discrete

images (image resizing)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 2

Page 3: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Illustration of Image Sampling and InterpolationInterpolation

dx=dy=2mmy

16 mm

dx=dy=1mm

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 3

How to choose dx, dy to reach a good trade off between accuracy and cost of storage, transmission, processing?

Page 4: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Uniform Sampling• f(x,y) represents the original continuous

image, fs(m,n) the sampled image, andimage, fs(m,n) the sampled image, and the reconstructed image.

• Uniform sampling),(ˆ yxf

Uniform sampling),,(),( ynxmfnmfs

– ∆x and ∆y are vertical and horizontal

.1,...,0;1,...,0 NnMmy

sampling intervals. fs,x=1/∆x, fs,y=1/ ∆y are vertical and horizontal sampling frequencies.

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 4

Page 5: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Image Sampling as Product with Impulse TrainProduct with Impulse Train

• Periodic impulse sequence

1 1M N

1

0

1

0),(),(

M

m

N

nyx nymxyxp

1 1

1

0

1

0

~

)()(

),(),(),(),(),(

M N

M

m

N

nyxyxs

f

nymxnmfyxpyxfyxf

0 0

),(),(m n

yxs nymxnmf

1m = 0

∆x1

…p(x,y)

M - 1

x y∆y

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 5

n= 0 1 N-12x y

Page 6: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Fourier Transform of Impulse Train• 1D

nfut

uPtnttp s

)(1)()()(

tfwhere

t

s

nnm

1

,

• 2D

nfvmfuyx

vuPynyxmxyxpnm

ysxsnm

),(1),(),(),(,

,,,

yf

xfwhere ysxs

1,1

,,

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 6

Page 7: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Frequency Domain Interpretation of SamplingSampling

• Sampling is equivalent to multiplication of the original signal with a sampling pulsethe original signal with a sampling pulse sequence.

s yxpyxfyxf ),(),(),(

I f d i

nm

ynyxmxyxpwhere,

),(),(

• In frequency domains vuPvuFvuF

11),(),(),(

nm

ysxss

ysxs

nmysxs nfvmfuF

yxvuF

ffwhere

nfvmfuyx

vuP,

,,,

,, ),(1),(

1,1

),(1),(

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 7

ysxs yf

xf ,, ,

Page 8: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Frequency Domain Interpretation of Sampling in 1D

Original signal

p p g

Sampling impulse train

The spectrum of the sampled signal includes the original

S l d i l

includes the original spectrum and its aliases (copies) shifted to k fs , k=+/- 1,2,3,…

Sampled signalfs > 2fm

s

When fs< 2fm , aliases overlap with the original spectrum >Sampled signal

fs < 2fm(Aliasing effect)

original spectrum -> aliasing artifact

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 8

Page 9: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Sampling of 1D Sinusoid Signals

Sampling aboveNyquist rateyqs=3m>s0

Reconstructedi i l=original

Sampling underNyquist rates=1.5m<s0

ReconstructedReconstructed!= original

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 9

Aliasing: The reconstructed sinusoid has a lower frequency than the original!

Page 10: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Frequency Domain Interpretation of Sampling in 2DSampling in 2D

• The sampled signal contains replicas of the original spectrum shifted by multiplesthe original spectrum shifted by multiples of sampling frequencies.

u uu u

fm x

fs,x

fs,x>2fm,x

vv

fm,x

fm,y fs,y

fs,y>2fm,y

Original spectrum F(u v) Sampled spectrum F (u v)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 10

Original spectrum F(u,v) Sampled spectrum Fs(u,v)

Page 11: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Illustration of Aliasing Phenomenon

u u

fm,x fs,xfs x<2fm x

v vfm,yfs,y

s,x fm,xfs,y<2fm,y

Original spectrum F(u,v) Sampled spectrum Fs(u,v)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 11

p p s( )

Page 12: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Nyquist Sampling Rate

• Nyquist sampling rateTo be able to preserve the original signal in– To be able to preserve the original signal in the sampled signals, these aliasing components should not overlap with thecomponents should not overlap with the original one. This requires that the sampling frequency fs,x, fs,y must be at least twice of the , ,yhighest frequency of the signal, known as Nyquist sampling rate.

• Interpolation– Remove all the aliasing components

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 12

Page 13: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Nyquist Sampling and Reconstruction TheoremTheorem

• A band-limited image with highest frequencies at fm,x, fm,ycan be reconstructed perfectly from its samples, provided that the sampling frequencies satisfy: fs,x >2fm,x, fs,y>2fm,y

• The reconstruction can be accomplished by the ideal• The reconstruction can be accomplished by the ideal low-pass filter with cutoff frequency at fc,x = fs,x/2, fc,y = fs,y/2, with magnitude ∆x∆y.

yfyf

xfxf

yxhotherwise

fv

fuyxvuH

ys

ys

xs

xsysxs

,

,

,

,,, sinsin

),(0

2||,

2||),(

• The interpolated image

)(sin)(sin),(),(ˆ ,, ymyfxmxf

nmfyxf ysxss

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 13

)()(),(),(

,, ymyfxmxfnmfyxf

ysxsm ns

Page 14: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Applying Nyquist Theorem• Two issues

– The signals are not bandlimited.The signals are not bandlimited.• A bandlimiting filter with cutoff frequency fc=fs/2 needs to be

applied before sampling. This is called prefilter or sampling filterfilter.

– The sinc filter is not realizable.• Shorter, finite length filters are usually used in practice for

b th filt d i t l ti filtboth prefilter and interpolation filter.

• A general paradigm

Prefilter Interpolation(postfilter)A D

B C

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 14

Sampling pulse fs

Page 15: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Non-ideal Sampling and InterpolationInterpolation

AliasedComponent

AliasedComponent

Pre-filteredSignal

-fs fs0

Non-idealInterpolationfilter

Aliasing Imaging

filter

Non ideal prefiltering causes AliasingNon ideal interpolation filter causes Imaging

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 15

p g g

Page 16: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Sampling a Sinusoidal Signal )1,2()1,2(

21),()24cos(),( vuvuvuFyxyxf

S l d t ∆ ∆ 1/3 f f 3Sampled at ∆x=∆y=1/3 fs,x=fs,y=3

v vOriginal Spectrum Sampled Spectrum

3(-2,1)

3(-2,1)

u

3

-3 3

(2,-1)

u

3

-3 3(2,-1) Ideal-3 (2, 1) -3

Original pulse Replicated pulse

interpolation filter

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 16

Replication center)22cos(),(ˆ yxyxf

Page 17: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Sampling in 2D: Sampling a 2D Sinusoidal PatternSampling a 2D Sinusoidal Pattern

f(x,y)=sin(2*π*(3x+y))Sampling: dx=0 01 dy=0 01 f(x,y)=sin(2*π*(3x+y))Sampling: dx=0.01,dy=0.01Satisfying Nyquist ratefx,max=3, fy,max=1fs,x=100>6, fs,y=100>2

Sampling: dx=0.2,dy=0.2(Displayed with pixel replication)Sampling at a rate lower than Nyquist rate

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 17

Page 18: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

A Simple Prefilter – Averaging Filter• Each sampling value is the mean value of the

original continuous function in a rectangularoriginal continuous function in a rectangular region of dimension ∆x and ∆y, i.e:

dxdyyxfnmf )(1)(

n

nmDyx

s dxdyyxfyx

nmf,),(

),(),(

where

m

The equivalent prefilter is

ynyynxmxxmD nm )2/1()2/1(,)2/1()2/1(,

yvyv

xuxuyxvuH

otherwise

yyxxyxyxh

sinsin),(

0

2/||,2/||1),(

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 18

otherwise 0

Page 19: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

How good is the averaging Filter?

• Look at its frequency response, how far is it from the ideal low pass filterit from the ideal low pass filter

Ideal low pass filter with cutoff at fs/2p fs

Averaging filter

f 2f 3f-f2f3f

Averaging filter

fs 2fs 3fsfs-2fs-3fs

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 19

Page 20: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Reconstruction of continuous image from Samplesfrom Samples

• The interpolation problem• Interpretation as weighted average of sample• Interpretation as weighted average of sample

values• Interpretation as filteringInterpretation as filtering• What filter should we use?

– Ideal interpolation filter: remove the replicatedIdeal interpolation filter: remove the replicated spectrum using ideal low-pass filter with cutoff frequency at fs/2 (the sinc function)

)()(sin

)()(sin

),(),(ˆ,

,

,

,

ynyfynyf

xmxfxmxf

nmfyxfys

ys

xs

xs

m ns

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 20

Page 21: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Interpretation as a weighted average of sample valuesof sample values

• The value of a function at arbitrary point (x, y) is estimated from a weighted sum of its sample values in the neighborhood of ([x/∆x],weighted sum of its sample values in the neighborhood of ([x/∆x], [y/∆y]):– Let h(x,y; m,n) specifies the weight assigned to sample m,n, when

determining the image value at x,y

nm

s nmfnmyxhyxf,

),(),;,(),(ˆ

H(x;8)

H(x;9)H(x;7)

x

H(x;10)

H(x;9)

H(x;6)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 21

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 n

H(x;5)

Page 22: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Desirable Properties of the Weight FunctionFunction

• The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial location of (m,n), i ei.e.

Should be a decreasing function of the distance

).,(),;,( ynyxmxhnmyxh

• Should be a decreasing function of the distance– Higher weight for nearby samples

• Should be an even function of the distanceL ft i hb d i ht i hb f di t h th– Left neighbor and right neighbor of same distance have the same weight

– h1(x)=h1(-x)• Generally Separable:• Generally Separable:

– h2(x,y)=h1(x) h1(y) • To retain the original sample values, should have

h(0 0)=1 h(mx ny)=0

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 22

– h(0,0)=1, h(mx,ny)=0

Page 23: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Interpretation as Filtering• The weighted

average operation is ),(),(),(ˆ

~

,nmfynyxmxhyxf s

nm

equivalent to filtering fs(x,y) with h(x,y)

• Usually h(x y) is ),(),(

),(),(),(

~

,

yxfyxh

ynyxmxnmfyxf

s

nmss

• Usually, h(x,y) is separable h(x,y) = hx(x)hy(y) ),(),(),(

),(),(~

ddynxmnmfyxh

ddfyxh

s

s

• To retain the original sample values, should have )()(

),(),(),(,

,

hf

ddynxmyxhnmfnm

s

nm

should have– h(0,0)=1,

h(mx,ny)=0Nyquist filter

),(),(),(ˆ

),(),(

~

,

yxfyxhyxf

ynyxmxhnmf

s

nms

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 23

– Nyquist filter

Page 24: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

A Simple Interpolation Filter: Sample-And-Hold (pixel replication)Sample And Hold (pixel replication)

• The interpolated value at a point is obtained from that of its nearest sampleobtained from that of its nearest sample

ynyynxmxxmnmfyxf s )2/1()2/1(,)2/1()2/1(),(),(ˆ

• Corresponding interpolation filter is

yyyxxx

yxh2/2/,2/2/1

),( n otherwise

y0

),(

m(m-1/2)∆x

m(m+1/2)∆x

(n 1/2)∆y

0th order interpolation filter

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 24

(n-1/2)∆y(n+1/2)∆y

Page 25: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Bilinear Filter

f0fa

f1(m,n) (m,n+1)

b

0 1aa

b

q1 q 2)1,1()1,()1()(

),1(),()1()(:1

2

1

Snmafnmfaqf

nmafnmfaqfStep

10)1( affafa

(m+1,n) (m+1,n+1)

pq1 q2

)()()1()(:2

21 qbfqfbpfStep

1D Linear interpolation

2D bilinear interpolation

Corresponding interpolation filter

yyyxxx

yy

xx

yxh ,||1||1),(

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 25

otherwise0

Page 26: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Which filter is better?

• Recall what happened in the frequency domain when we sample an imagedomain when we sample an image

• Ideal filter: ½ band ideal low pass filter• Quantitatively we can evaluate how far is

the filter from the ideal filter• But we should also look at visual artifacts

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 26

Page 27: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Frequency Domain Interpretation of Sampling in 2DSampling in 2D

• The sampled signal contains replicas of the original spectrum shifted by multiplesthe original spectrum shifted by multiples of sampling frequencies.

u uu u

fm x

fs,x

fs,x>2fm,x

vv

fm,x

fm,y fs,y

fs,y>2fm,y

Original spectrum F(u v) Sampled spectrum F (u v)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 27

Original spectrum F(u,v) Sampled spectrum Fs(u,v)

Page 28: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Ideal Interpolation Filter• The ideal interpolation filter should be a low-pass filter

with cutoff frequency at fc,x = fs,x/2, fc,y = fs,y/2, with y ymagnitude ∆x∆y

yfxfyxh

fv

fuyxvuH ysxs

ysxs,,

,, sinsin)(2

||,2

||)(

yfxf

yxhotherwise

yvuHysxs ,,

),(0

2||

2||),(

The sinc filter

• The interpolated image

The sinc filter

Weight function h(x,y;m,n)

)()(sin

)()(sin

),(),(ˆ,

,

,

,

ynyfynyf

xmxfxmxf

nmfyxfys

ys

xs

xs

m ns

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 28

Page 29: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Comparison of Different Interpolation FiltersFilters

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 29

Page 30: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Image Resizing• Image resizing:

– Enlarge or reduce the image size (number of pixels)– Equivalent to

• First reconstruct the continuous image from samples• Then Resample the image at a different sampling ratep g p g

– Can be done w/o reconstructing the continuous image explicitly

• Image down-sampling (resample at a lower rate)Image down-sampling (resample at a lower rate)– Spatial domain view– Frequency domain view: need for prefilter

• Image up-sampling (resample at a higher rate)– Spatial domain view– Different interpolation filters

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 30

Different interpolation filters• Nearest neighbor, Bilinear, Bicubic

Page 31: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Image Down-Sampling

• Example: reduce a 512x512 image to 256x256 = factor– reduce a 512x512 image to 256x256 = factor of 2 downsampling in both horizontal and vertical directionsvertical directions

– In general, we can down-sample by an arbitrary factor in the horizontal and vertical ydirections

• How should we obtain the smaller image ?How should we obtain the smaller image ?

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 31

Page 32: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Down Sampling by a Factor of Two

8x8 Image 4x4 Image

• Without Pre-filtering (simple approach)Without Pre filtering (simple approach)

Averaging Filter

)2,2(),( nmfnmfd

• Averaging Filter4/)]12,12()2,12()12,2()2,2([),( nmfnmfnmfnmfnmfd

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 32

Page 33: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Problem of Simple Approach• Aliasing if the effective sampling rate is below

the Nyquist sample rate = 2 * highest frequencythe Nyquist sample rate 2 highest frequency in the original continuous signal

• We need to prefilter the signal before down-p gsampling

• Ideally the prefilter should be a low-pass filter with a cut-off frequency half of the new sampling rate.– In digital frequency of the original sampled image, the

cutoff frequency is ¼.• In practice we may use simple averaging filter

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 33

• In practice, we may use simple averaging filter

Page 34: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Down Sampling by a Factor of K

)(f̂Hs ↓K

f(m,n) fd(m,n)),( nmf

Pre-filtering Down-sampling

),(ˆ),( KnKmfnmfd

For factor of K down sampling, the prefilter should be low pass filter with cutoff at fs/(2K), if fs is the original sampling frequency

I t f di it l f th t ff h ld b 1/(2K)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 34

In terms of digital frequency, the cutoff should be 1/(2K)

Page 35: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Example: Image Down-Sample

Without prefiltering

With prefiltering (no aliasing, but blurring!)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 35

Page 36: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Down-Sampling Using Matlab• Without prefiltering

– If f(,) is an MxN image, down-sampling by a factor of K can be done simply by

>> g=f(1:K:M,1:K:N)

• With prefilteringp g– First convolve the image with a desired filter

• Low pass filter with digital cutoff frequency 1/(2K)– In matlab, 1/2 is normalized to 1In matlab, 1/2 is normalized to 1

– Then subsample>> h=fir1(N, 1/K)

%design a lowpass filter with cutoff at 1/K%design a lowpass filter with cutoff at 1/K.>> fp=conv2(f,h)>> g=fp(1:K:M,1:K:N)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 36

Page 37: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Image Up-Sampling• Produce a larger image from a smaller one

– Eg. 512x512 -> 1024x1024M ll l b bit f t L– More generally we may up-sample by an arbitrary factor L

• Questions: – How should we generate a larger image?– Does the enlarged image carry more information?

• Connection with interpolation of a continuous image from discrete image– First interpolate to continuous image, then sampling at a higher sampling

rate, L*fs– Can be realized with the same interpolation filter, but only evaluate at

x=mx’ y=ny’ x’=x/L y’=y/Lx=mx , y=ny , x =x/L, y =y/L– Ideally using the sinc filter!

)(sin)(sin)()(ˆ ,, ymyfxmxf

ff ysxs

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 37

)()(),(),(

,

,

,

,

ymyfxmxfnmfyxf

ys

ys

xs

xs

m ns

Page 38: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Example: Factor of 2 Up-Sampling

(m,n+1)(m,n) (2m,2n) (2m,2n+1)

(2m+1 2n) (2m+1 2n+1)

(m+1,n+1)(m+1,n)

(2m+1,2n) (2m+1,2n+1)

Green samples are retained in the interpolated image;Orange samples are estimated from surrounding green samples.

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 38

Page 39: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Nearest Neighbor Interpolation(pixel replication)(pixel replication)

(m n+1)(m n)

a

b

(m,n+1)

(m’/M,n’/M)

(m,n)

a

(m+1,n+1)(m+1,n)

O[m’,n’] (the resized image) takes the value of the sample nearest to (m’/M,n’/M) in I[m,n] (the original image):

/Mn'=n/Mm'=m0 5)]+(n(int)0 5)+(mI[(int)=]n'O[m'

Also known as pixel replication: each original pixel is replaced by MxM pixels of

/M.nn/M,mm,0.5)]+(n(int)0.5),+(mI[(int) ]n,O[m

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 39

Also known as pixel replication: each original pixel is replaced by MxM pixels of the sample valueEquivalent to using the sample-and-hold interpolation filter.

Page 40: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Special Case: M=2

(m,n+1)(m,n) (2m,2n) (2m,2n+1)

(2m+1 2n) (2m+1 2n+1)

(m+1,n+1)(m+1,n)

(2m+1,2n) (2m+1,2n+1)

Nearest Neighbor:O[2m,2n]=I[m,n]O[2m,2n+1]= I[m,n]O[2m+1,2n]= I[m,n]

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 40

O[2m+1,2n+1]= I[m,n]

Page 41: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Bilinear Interpolation

(m,n) (m,n+1)

a

b

(m’/M,n’/M)

• O(m’,n’) takes a weighted average of 4 samples nearest to (m’/M,n’/M) in I(m,n).

(m+1,n) (m+1,n+1)

O(m ,n ) takes a weighted average of 4 samples nearest to (m /M,n /M) in I(m,n).

• Direct interpolation: each new sample takes 4 multiplications:O[m’,n’]=(1-a)*(1-b)*I[m,n]+a*(1-b)*I[m,n+1]+(1-a)*b*I[m+1,n]+a*b*I[m+1,n+1]

• Separable interpolation:i) interpolate along each row y: F[m,n’]=(1-a)*I[m,n]+a*I[m,n+1]ii) interpolate along each column x’: O[m’,n’]=(1-b)*F[m’,n]+b*F[m’+1,n]

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 41

Page 42: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Special Case: M=2

(m,n+1)(m,n) (2m,2n) (2m,2n+1)

(2m+1 2n) (2m+1 2n+1)

(m+1,n+1)(m+1,n)

(2m+1,2n) (2m+1,2n+1)

Bilinear Interpolation:O[2m,2n]=I[m,n]O[2m,2n+1]=(I[m,n]+I[m,n+1])/2O[2m+1,2n]=(I[m,n]+I[m+1,n])/2

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 42

O[2m+1,2n+1]=(I[m,n]+I[m,n+1]+I[m+1,n]+I[m+1,n+1])/4

Page 43: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Bicubic Interpolation

(m-1,n+1)(m-1,n)

b

(m,n) (m,n+1)(m,n-1) (m,n+2)

a

(m+1,n+1)(m+1,n)

(m’/M,n’/M)

(m+1,n-1) (m+1,n+2)

(m+2,n) (m+2,n+1)

O( ’ ’) i i t l t d f 16 l t t ( ’/M ’/M) i I( )• O(m’,n’) is interpolated from 16 samples nearest to (m’/M,n’/M) in I(m,n).• Direct interpolation: each new sample takes 16 multiplications• Separable interpolation:

i) interpolate along each row y: I[m,n]->F[m,n’] (from 4 samples)ii) i t l t l h l ’ F[ ’] O[ ’ ’] (f 4 l )

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 43

ii) interpolate along each column x’: F[m,n’]-> O[m’,n’] (from 4 samples)

Page 44: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Interpolation Formula

(m n) (m n+1)(m n 1) (m n+2)

(m-1,n+1)(m-1,n)(m-1,n-1) (m-1,n+2)

b

(m,n) (m,n+1)(m,n-1) (m,n+2)

F(m’,n-1) F(m’ n) F(m’ n+1) F(m’ n+2)a

(m+1,n+1)(m+1,n)

(m’/M,n’/M)

(m+1,n-1) (m+1,n+2)

(m+2,n) (m+2,n+1)

( , ) F(m ,n) F(m ,n+1) F(m ,n+2)

( , ) ( , )

(m+2,n-1) (m+2,n+2)

IbbIbbbIbbIbbF ]2[)1(]1[)1(][)21(]1[)1(]'[ 22322

mMmb

Mmmwhere

nmIbbnmIbbbnmIbbnmIbbnmF

','(int)

],,2[)1(],1[)1(],[)21(],1[)1(],'[ 22322

nnnmFaanmFaaanmFaanmFaanmO

''],2,'[)1(]1,'[)1(],'[)21(]1,'[)1(]','[ 22322

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 44

nMna

Mnnwhere ,(int)

Page 45: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Special Case: M=2

(2m,2n) (2m,2n+1)(m-1,n+1)(m-1,n)

(2m+1 2n) (2m+1 2n+1)

(m,n) (m,n+1)(m,n-1) (m,n+2)

(2m,2n+1) (2m+1,2n) (2m+1,2n+1)

(m+1,n+1)(m+1,n)(m+1,n-1) (m+1,n+2)

Bicubic interpolation in Horizontal direction

(m+2,n) (m+2,n+1)

Bicubic interpolation in Horizontal direction

F[2m,2n]=I[m,n]F[2m,2n+1]= -(1/8)I[m,n-1]+(5/8)I[m,n]+(5/8)I[m,n+1]-(1/8)I(m,n+2)

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 45

Same operation then repeats in vertical direction

Page 46: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Comparison of Interpolation MethodsInterpolation Methods

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 46

Resize_peak.m

Page 47: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Up-Sampled from w/o Prefiltering

NearestneighborOriginal

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 47

Bilinear Bicubic

Page 48: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Up-Sampled from with Prefiltering

NearestneighborOriginal

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 48

Bilinear Bicubic

Page 49: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Matlab for Image Resizing[img]=imread('fruit.jpg','jpg');

%downsampling without prefiltering

i 1 i i (i 0 5 ' ')img1=imresize(img,0.5,'nearest');

%upsampling with different filters:

img2rep=imresize(img1,2,'nearest');

i 2li i i (i 1 2 'bili ')img2lin=imresize(img1,2,'bilinear');

img2cubic=imresize(img1,2,'bicubic');

%down sampling with filtering%down sampling with filtering

img1=imresize(img,0.5,'bilinear',11);

%upsampling with different filters

img2rep=imresize(img1 2 'nearest');img2rep=imresize(img1,2, nearest );

img2lin=imresize(img1,2,'bilinear');

img2cubic=imresize(img1,2,'bicubic');

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 49

Page 50: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Filtering View: Up Sampling by a Factor of Kof K

)(~fHi↑K

f(m,n) fu(m,n)),( nmf

Zero-padding Post-filtering

Kfl lfKKf )//(

otherwise

KofmultiplearenmifKnKmfnmf

0,)/,/(

),(~

lk

u lnkmflkhnmf,

),(~),(),(

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 50

Ideally H should be a low pass filter with cutoff at 1/2K in digital frequency, or fs/2K in continuous frequency

Page 51: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Homework (1)1. Consider a 1D signal f(t) = sin(4πt). Illustrate the original and the sampled

signal f(n) obtained with a sampling interval ∆t = 1/3. Draw on the same figure the interpolated signal from the sampled one using the sample-and-hold and the linear interpolation filter respectively Explain theand-hold and the linear interpolation filter, respectively. Explain the observed phenomenon based on both the Nyquist sampling theorem as well as physical interpretation. What is the largest sampling interval that can be used to avoid aliasing?

2. Consider a function f(x, y) = cos2π(4x + 2y) sampled with a sampling2. Consider a function f(x, y) cos2π(4x 2y) sampled with a sampling period of ∆x = ∆y = ∆ = 1/6 or sampling frequency fs = 1/∆ = 6.

a) Assume that it is reconstructed with an ideal low-pass filter with cut-off frequency fcx = fcy = 1/2fs. Illustrate the spectra of the original, sampled, and reconstructed signals. Give the spatial domain function representation of the reconstructed signal Is the result as expected?reconstructed signal. Is the result as expected?

b) If the reconstruction filter has the following impulse response:

otherwise

yxyxh

02/,2/1

),(

Illustrate the spectra of the reconstructed signal in the range -fs ≤ u,v ≤ fs. Give a spatial domain function representation of the reconstructed signal if the reconstruction filter is band-limited to (-fs ≤ u,v ≤ fs). (i.e., this filter remains the

otherwise0

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 51

( s , s) ( ,same for the frequency range -fs ≤ u,v ≤ fs, and is set to 0 outsize this range.)

Page 52: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Homework(2)3. (Computer Assignment) Write your own program or programs

which can: a) Down sample an image by a factor of 2, with and without using the averaging filter; b) Up-sample the previously d l d i b f t f 2 i th i l li tidown-sampled images by a factor of 2, using the pixel replication and bilinear interpolation methods, respectively. You should have a total of 4 interpolated images, with different combination of down-sampling and interpolation methods. Your program could ith di tl di l th d i d ieither directly display on screen the processed images during

program execution, or save the processed images as computer files for display after program execution. Run your program with the image Barbara. Comment on the quality of the down/up

l d i bt i d ith diff t th dsampled images obtained with different methods.Note: you should not use the ”imresize” function in Matlab to do this assignment. But you are encouraged to compare results of your program with ”resize”.y p g

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 52

Page 53: lecture8 sampling F11 - New York Universityyao/EL5123/lecture8... · 2012. 11. 14. · • The weighting function h(x,y;m,n) should depend only on distance between (x,y) and the spatial

Reading

• R Gonzalez “Digital Image Processing ”• R. Gonzalez, Digital Image Processing, Section 2.4

• A K Jain “Fundamentals of Digital ImageA.K. Jain, Fundamentals of Digital Image Processing,” Section 4.1-4.4

Yao Wang, NYU-Poly EL5123: Sampling and Resizing 53