Sampling

Resampling

Warping

Morphing

Dr. Shai Avidan

Faculty of Engineering

Tel-Aviv University

Slide Credits (Partial List)

• Alyosha Efros

• Steve Seitz

• Rick Szeliski

• Bill Freeman

• Fredo Durand

Sampling/Resampling/Warping/Morphing/Retargeting

• Sampling

– Given a continuous signal g(x) and its sampled counterpart g_s(x), are the

samples of g_s(x) sufficient to exactly describe g(x)?

– If so, how can g(x) be reconstructed from g_s(x)?

• Resampling

– Image resampling is the process of transforming a sampled image from one

coordinate system to another

• Warping

– Is a geometric transformation that redefines the spatial relationship between

points in the image

• Morphing

– Is the process of blending two images

• Retargeting

– Is a content-aware resizing operation

Sampled representations• How to store and compute with continuous functions?

• Common scheme for representation: samples

– write down the function’s values at many points

[FvD

FH

fig.1

4.1

4b

/ W

olb

erg

]

Reconstruction

• Making samples back into a continuous function

– for output (need realizable method)

– for analysis or processing (need mathematical method)

– amounts to “guessing” what the function did in between

[FvD

FH

fig.1

4.1

4b

/ W

olb

erg

]

1D Example: Audio

low high

frequencies

Sampling in digital audio• Recording: sound to analog to samples to disc

• Playback: disc to samples to analog to sound again

– how can we be sure we are filling in the gaps correctly?

Sampling and Reconstruction

• Simple example: a sine wave

Undersampling

• What if we “missed” things between the samples?

• Simple example: undersampling a sine wave

– unsurprising result: information is lost

Undersampling

• What if we “missed” things between the samples?

• Simple example: undersampling a sine wave

– unsurprising result: information is lost

– surprising result: indistinguishable from lower frequency

Undersampling• What if we “missed” things between the samples?

• Simple example: undersampling a sine wave

– unsurprising result: information is lost

– surprising result: indistinguishable from lower frequency

– also was always indistinguishable from higher frequencies

– aliasing: signals “traveling in disguise” as other frequencies

Aliasing in video

Slide by Steve Seitz

Aliasing in images

What’s happening?Input signal:

x = 0:.05:5; imagesc(sin((2.^x).*x))

Plot as image:

Alias!

Not enough samples

Antialiasing

• What can we do about aliasing?

• Sample more often

– Join the Mega-Pixel craze of the photo industry

– But this can’t go on forever

• Make the signal less “wiggly”

– Get rid of some high frequencies

– Will loose information

– But it’s better than aliasing

Preventing aliasing• Introduce lowpass filters:

– remove high frequencies leaving only safe, low frequencies

– choose lowest frequency in reconstruction (disambiguate)

Sampling and Reconstruction

Image AcquisitionImaging

subsystem

*h(x,y)

Sampling

Subsytem

*s(x,y)Quantizer

f(x,y) g(x,y) gs(x,y) gd(x,y)

Digitized

Image

Scene Image Sampled

Image

g(x,y)=f(x,y)*h(x,y)The scene is mapped to the image via the

point spread function of the camera h(x,y)

gs(x,y)=g(x,y)s(x,y)The continuous image g(x,y) then enters

A sampling subsystem 2D comb function

s(x,y) given by: � � � �� ��

���

�

���

���m n

nymxyxs ,, �

Therefore, gs(x,y) is now a discrete-continuous

image with intensity values defined only over

integral indices of x and y

After sampling, the quantizer takes the continuous

Intensity values and quantizes them

(either uniform or non-uniform)

Sampling (I)

We start with continuous signal f(x). Due to the psf

We obtain a degraded, band-limited output g(x).

The frequency content of g(x) is given by its spectrum,

G(f), as determined by the Fourier transform:� � � �

� � � �fxicfxe

dxexgfG

fxi

fxi

���

�

2sin2cos2

2

�

�

��

��

Where x represents the spatial position and f denotes

Spatial frequency. Since g(x) does not have any

frequencies beyond fmax, we say that it is band-limited.

|G(f)|

-fmax fmax

Sampling (II)

The continuous output g(x) is then digitized by an ideal

Impulse sampler, the comb function, to get the sampled

signal gs(x). The ideal 1-D sampler is given by:

� � � ���

���

��n

snTxxs �

where \delta is the impulse function and T_s is the sampling

period. The running index n is used with \delta to define the

impulse train of the comb function. We now have:

� � � � � �xsxgxg s �

Sampling (III)Taking the Fourier transform of gs(x) yields � � � � � �

� � � �

� ��

��

���

�

���

��

��

��

���

�

n

ss

n

ss

s

nffGf

nffffG

fSfGfG

*

*

�

Where fs is the sampling frequency and * denotes convolution.

We make use of the following properties of the Fourier Transform

1. Multiplication in the spatial domain corresponds to convolution in the

frequency domain.

2. The Fourier transform on an impulse train is itself an impulse train.

3. The spectrum of a signal sampled with frequency fs (Ts=1/fs) yields the

original spectrum replicated in the frequency domain with period fs|Gs(f)|

-fmax fmax-fs fs

Sampling (IV)

The above result reveals that the sampling operation has left the

original input spectrum intact, merely replicating it periodically in

the frequency domain with a spacing of f_s.

This allows us to rewrite G_s(f) as a sum of two terms, the low

frequency (baseband) and high frequency components. The

baseband specturm is exactly G(f), and the high frequency

components, G_high(f), consist of the remaining replicated

versions of G(f).

G_s(f) = G(f) + G_high(f)

Exact signal reconstruction from sampled data requires us to

discard the replicated spectra G_high(f), leaving only G(f), the

spectrum of the signal we seek to recover.

Reconstruction Conditions

The only provision for exact reconstruction is that G(f) be

undistorted due to overlap with G_high(f). Two conditions

must hold for this to be true:

1. The signal must be bandlimited. This avoids spectra with

infinite extent that are impossible to replicate without

overlap. (In most imaging systems, the psf ensures this

condition is satisfied in practice).

2. The sampling frequency f_s must be greater than twice the

maximum frequency f_max, present in the signal. This

minimum sampling frequency, known as the Nyquist rate, is

the minimum distance between spectra copies, each with

bandwidth f_max.

f_s>f_nyquist=2*f_max

The Ideal Low-Pass Filter

Given that it is theoretically possible to perform reconstruction,

how may it be done? The act of reconstruction requires us to

completely suppress G_high(f). This is done by multiplying

G_s(g) with H(f) given as:

� ����

�

��

max

max

0

1

ff

fffH

|Gs(f)|

-fmax fmax-fs fs

H(f)

Sinc Function

In the spatial domain, the ideal low-pass filter is derived by

computing the inverse Fourier transform of H(f). This yields the

sinc function defined as:

Since multiplication in the frequency domain is identical to

convolution in the spatial domain, then we can convolve the

sample signal g_s(x) with sinc(x) to obtain the continuous g(x).

The problem is that sinc(x) requires infinite support which is

impossible in practice.

� � � �x

xx

��sin

sinc �

� � � � � � � � � � ��� dxgxgxxg ss ��� �

��sinc*sinc

Ringing effect

When using a truncated sinc function for reconstruction we get

ringing. These artifacts, known as the Gibbs phenomenon, are

the overshoots and undershoots caused by reconstructing a

signal with truncated frequency terms.

Approximations to the ideal low pass filter balance between

tampering frequencies below f_max and not suppressing fully

frequencies above it.

Nonideal Reconstruction

Since H(f), the ideal filter cannot be computed in practice, we

resort to nonideal filters.�This bring up the problem of how to

assess the quality of a filter

|G(f)|

-fmax fmax-fs fs

Hr(f)

H_r(f) attenuate some frequencies that should remain intact and pass and

pass some of the high frquency that should be suppressed.

This leads to the problem of aliasing:

aliasing: signals “traveling in disguise” as other frequencies high

frequencies disguised as low frequencies

Aliasingaliasing: signals “traveling in disguise” as other

frequencies

It can be solved in one of two ways:

1. Increasing sampling rate (to reach Nyquist rate)

2. Applying low pass filter (to bandlimit the signal)

Aliasing in video

Slide by Steve Seitz

Image Resampling

So, we have a discrete image and we want to warp it to another discrete

image. The two coordinate systems are related by a spatial warping

function. This permits the output image to be generated by the following

straightforward procedure. First, the inverse mapping function is applied to

the output sampling grid, projecting it onto the input. The result is a

resampling grid, specifying the locations at which the input is to be

resampled. Then, the input image is sampled at these points and the

values are assigned to their respective output pixels.

The problem is that resampling grid does not coincide with the input

sampling grid.

The solution is to reconstruct the input image to a continuous image and

then do the sampling.

Ideal Image Resampling

There are four basic elements to ideal image resampling: reconstruction,

warping, prefiltering and sampling.

The Bible

• http://www-2.cs.cmu.edu/~ph/texfund/texfund.pdf

Resampling2D texture space

2D screen space

warp due to perspective

Notations

• Input signal f(u)

• Forward mapping (texture-to-screen)

x=m(u)

• Output signal g(x)

f(u)

g(x)

m

Resampling

• What do we need to do?

f(u)

g(x)

m

Resampling

1. Reconstruct the continuous signal from the

discrete input signal

2. Warp the domain of the continuous signal

3. Prefilter the warped continuous signal

4. Sample this signal

f(u)

g(x)

m

Resampling

Resampling: progression• Discrete input texture f(u) for integer u

• Reconstructed input texture

fc(u) = f(u) * r(u) = � _{k \in Z} f(k) r(u-k)

• Warped texture gc(x)=fc(m-1(x))

• Band-limited output g’c(x) = gc(x) * h(x) = \Integral gc(t) h(x-t) dt

• Discrete output g(x) = g’c(x) s(x)

f(u)

g(x)

m

Resampling

discrete input

warp

warped inputreconstructed input

reconstruct

continuous output

prefilter

source space

resampling with

prefiltering

[Heckbert 89]

discrete output

sample

destination space

Put it together

• Discrete input texture f(u) for integer u

• Reconstructed input texture fc(u) = f(u) - r(u) = �_k f(k) r(u-k)

• Warped texture gc(x)=fc(m-1(x))

• Band-limited output g’c(x)=gc(x)*h(x) = S gc(t) h(x-t) dt

• Discrete output g(x) = g’c(x)- s(x)

• g(x) = gc’(x)

= s gc(t) h(x-t) dt

= s fc(m-1(t)) h(x-t) dt

= s h(x-t) å f(k) r(m-1(t)-k) dt= � _k f(k) � (x, k)

• Where �(x,k) = s h(x – t) r (m-1(t) –k) dt

• Ignoring normalization

• The image space resampling filter combines a warped reconstruction

filter and a low-pass filter

• The signal x, in the output image is warped to the input image through

m^{-1}. Since it falls on non-integer pixel value we use the

reconstruction filter r to interpolate the discrete input signal f, and to

make sure the result conforms to the Nyquist rate we bandlimit it with

the function h

Resampling – convolution view

� �� �

i

ii xhxmrxfxg )())(()()( 1

resampling filter

Resampling

reconstruction kernels

reconstructed input

position

color

irregular spacing

Resampling

Source Space

3. Filter

Destination Space2. W

arp

Destination Space

4.

Sam

ple

Destination Space

Resampling

low-pass filter convolution

resampling filters

sum of resampling filters

warped reconstruction

kernel

Source Space

3. Filter

Destination Space2. W

arp

Destination Space

4.

Sam

ple

Destination Space

• Ignoring normalization

• The image space resampling filter combines a warped reconstruction filter and a low-pass filter

• This is great, but how do we warp reconstruction filters?

Resampling – convolution view

� �� �

i

ii xhxmrxfxg )())(()()( 1

resampling filter

Resampling

• Use local affine approximation of warp

• Elliptical Gaussian kernels [Heckbert 89]

– Closed under affine mappings and convolution

� �� �

i

iii xhxmrcxg )())(~()(1

��i

ii xGc )(

Gaussian resampling kernel

(EWA resampling kernel)

Resampling filter

• Depends on local warp

• For perspective,

approximated by local affine at

center of kernel

• Not bad approximation

because filter small at

periphery

Resampling Filter

warped recon-

struction kernel

low-pass

filter

resampling

filterminification

magnification

EWA resampling

Image Quality Comparison

• Trilinear mipmapping

EWA trilinear mipmapping

Minification/Magnification

In the special case of magnification, we may ignore the prefilter altogether, treating it instead as an impulse function. This is due to the fact that no high frequencies are introduced into the output upon magnification. So we only have to do reconstruction (i.e., interpolation).

Conversely, minifcation introduces high frequencies and does not require any reconstruction (i.e., interpolation) filter and treat it simply as an impulse function. Therefore,

1) For magnification we only need to interpolate

2) For minification we need to low-pass filter and then sub-sample.

For magnification, the shape of the reconstruction filter does not change in

response to the mapping function. In minifcation the reconstruction filter is

replaced by a prefilter. Unlike reconstruction filters, though, the actual

shape must be scaled by an amount linearly related to the minifcation

factor. As the input in increasingly decimated, the prefilter must become

broader and shorter. It becomes broader in order to average more

neighboring pixel together, thereby further bandlimiting the input. Since

larger neighborhood are used to compute each output pixel, the

normalized weights applied to the input decrease to reflect the diminishing

impact of each input sample. As a result, the prefilter grows shorter.

InterpolationInterpolation is the process of determining the values of a function at positions lying between its samples. I achieves this process by fitting a continuous function through the discrete input samples.

For equally spaced data, interpolation can be expressed as:

Where h is the interpolation kernel weighted by coefficients c_k and applied to K data sample x_k

� � � ���

�

��1

0

K

k

kk xxhcxf

Interpolation KernelsNearest neighbor method is given by the following interpolating polynomial:

It can be achieved by convolving the image with a one-pixel width rectangle in the spatial domain. The interpolation kernel for the nearest neighbor algorithm is defined as

Due to the prominent side lobes and infinite extent, a sinc function makes a poor low-pass filter.

� � � �22

11 �� ���

�� kkkk

k

xxx

xxxfxf

� ����

�

���

x

xxh

5.00

5.001

Interpolation KernelsLinear interpolation method is given by the following interpolating polynomial:

In the spatial domain, linear interpolation is equivalent to convolving the sampled input with the following interpolation kernel (tent filter)

� � � � � �

� � � �01

01

00

10

01101001

polynomialinginterpolatfollowing thegivesThis

11 :solvingby determinedare, where,

ffxx

xxfxf

xxaaffaaaxaxf

����

!!"

#

��

��

��

��

����

� ����

�

����

x

xxxh

10

101

Image Morphing History

Morphing is turning one image into another through a seamless transition

Michael Jackson’s “Black or White”

Cross-fading