fessler_2d interpolation [2013]
DESCRIPTION
2D Interpolation - Student versionFessler, February 18, 2013, 1TRANSCRIPT
![Page 1: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/1.jpg)
Chapter IN
2D Interpolation
Contents (student version)Introduction IN2
Problem statement and background IN3Ideal sinc interpolation IN4
Polynomial interpolation IN5Zero-order or nearest neighbor (NN) interpolation IN51D linear interpolation IN82D linear or bilinear interpolation IN9Interpolator properties IN13Lagrange interpolation IN15Quadratic interpolation IN16Cubic interpolation IN18Non-polynomial interpolation Lanczos kernel IN22
Interpolation in shift-invariant subspaces IN23Basis kernels supported on [-22] IN27B-spline interpolation IN332D interpolation in shift-invariant spaces IN38
Applications IN39Image registration IN40Image rotation is a separable operation IN44Image zooming (up-sampling) using the FFT IN45
IN1
c⃝ J Fessler February 18 2013 1754 (student version) IN2
Motion estimation for video temporal interpolation IN49Translation motion estimation IN52Region-matching methods IN53Space-time constraint methods IN56Region selection IN59Advanced topics IN59Motion compensated temporal interpolation IN60Application of motion estimation methods to spatial interpolation IN62
Summary IN62
Introduction
This chapter focuses on practical 2D interpolation methods Image interpolation is very important and used widely The ideasgeneralize readily to 3D interpolation problems
We also consider motion estimation and video interpolation In the context of video ie temporal image sequences temporalinterpolation can also be important Even though video is ldquo2D+timerdquo which is a kind of ldquo3D problemrdquo we use techniques thatdiffer from 3D spatial interpolation problems We focus initially on spatial interpolation
We focus on interpolation given equally spaced samples because those arise most often in DSP and image processing problemsBut there are also many applications that require interpolation from unequally spaced data and there are methods for those problemstoo [1 2] The MATLAB command griddata is one of those methods In 1D one classic method is spline interpolation
Nonlinear interpolation methods eg ldquomorphologicalrdquo methods have been proposed [3] but we focus on linear methods
For an excellent historical overview of interpolation see [4]
c⃝ J Fessler February 18 2013 1754 (student version) IN3
Problem statement and background
The image interpolation problem is the following We are given a discrete-space image gd[nm] that we assume corresponds tosamples of some continuous space image ga(x y) Typically we assume the samples are ideal samples taken on a rectilinear gridie gd[nm] = ga(n∆Xm∆Y) The usual goal is to compute values of the CS image ga(x y) at arbitrary (x y) locations
m
n
Digital image gd[nm]
82
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
14
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
1
0
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
91
0
0
65
99
99
99
99
99
99
69
28
0
0
0
71
99
99
99
92
0
0
0
50
57
5
6
5
3
22
0
0
70
7
0
0
98
99
93
1
0
23
50
0
2
1
1
26
55
0
24
69
4
0
0
0
99
93
3
0
5
9
0
30
23
63
23
83
42
27
83
83
40
0
0
64
1
5
0
0
2
0
7
31
26
53
53
0
0
83
83
91
0
0
2
0
0
0
0
38
0
3
53
24
49
50
15
2
85
82
95
69
0
7
0
0
0
0
95
0
0
23
2
12
50
40
61
57
85
95
17
0
99
1
0
0
0
1
95
6
0
2
0
3
1
3
21
45
0
0
80
99
1
2
2
3
0
0
27
56
0
85
82
0
0
0
0
0
5
99
99
2
2
2
0
6
65
63
99
57
13
40
50
0
0
0
35
99
99
99
3
3
1
0
52
59
54
99
99
99
80
13
17
99
99
99
99
99
99
37
8
7
74
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
rarrHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
There are two distinct situations where this type of operation is importantbull DA conversion such as a video monitor Here we really want a CS image ga(x y)bull Resampling situations where we want to evaluate ga(x y) at a discrete set of (x y) coordinates that are different from the
original sampling coordinates(n∆Xm∆Y)Why For image zooming display rotating warping coding motion estimation JAVA demonstration of the importance of good interpolation for image rotationhttpbigwwwepflchdemojrotationstartphp
Generally speaking interpolation is an example of an ill-posed inverse problem We are given only a finite or countable collectionof values gd[nm] yet we hope to recover ga(x y) a continuous-space function of uncountable arguments (x y) isin R2 Thereis an uncountably infinite collection of functions ga(x y) that agree with gd[nm] at the sample points (Connecting the dots is notunique) The only way for us to choose among the many possibilities is to make assumptions about the original CS signal ga(x y)Different interpolation methods are based on different assumptions or models
c⃝ J Fessler February 18 2013 1754 (student version) IN4
Ideal sinc interpolation
If gd[nm] = ga(n∆Xm∆Y) and if we assume that ga(x y) is band-limited and the sampling rates are appropriate and thesampling has infinite extent then in principle we can recover ga(x y) from gd[nm] exactly by the sinc interpolation formula
ga(x y) =infinsum
n=minusinfin
infinsumm=minusinfin
gd[nm]h(xminus n∆X y minusm∆Y) (IN1)
whereh(x y) =
This theoretical result is impractical because the sinc function (aka the sine cardinal function) has unbounded support and thesummations require infinitely many samples Furthermore real world images need not be exactly band-limited
In practice we always have a finite amount of data so we replace (IN1) by a practical method such as a general linear interpolator
ga(x y) ≜Nminus1sumn=0
Mminus1summ=0
gd[nm]h(xminus n∆X y minusm∆Y) (IN2)
Rarely will ga(x y) exactly match the true ga(x y) except perhaps at the sample locationsWe simply hope that ga(x y) asymp ga(x y) at least for 0 le x le N∆X and 0 le y leM∆Y
Different linear interpolation methods correspond to different choices for the interpolation kernel h(x y)
Note that any interpolator of the form (IN1) or (IN2) is linear in the sense that it is a linear function of the samples gd[nm]
Usually we choose interpolation kernels that are separable with the same 1D kernel shape in both x and y ie
h(x y) =
So we focus hereafter primarily on choices for the 1D interpolation kernel h(x) often with unit spacing ∆ = 1 for simplicity
c⃝ J Fessler February 18 2013 1754 (student version) IN5
Polynomial interpolation
We first consider polynomial interpolators ie those that have kernels that are piecewise polynomials
MATLABrsquos interp1 and interp2 commands support several such options
Zero-order or nearest neighbor (NN) interpolation
The simplest form of interpolation has the following recipe use the value at the nearest sample locationThat is a procedural description of the interpolator but we also want to have a mathematical description (and efficient software)A mathematical expression for this approach in 1D is
ga(x) = gd[round(x∆X)] = gd[n]∣∣∣n=round(x∆X)
(IN3)
One can verify that the interpolator (IN3) is integer shift invariant in the following sense If f [n] = gd[nminus n0] and interpolatef [n] using the same nearest neighbor rule (IN3) to obtain fa(x) then fa(x) = ga(xminus n0∆X) So shifting gd[n] by n0 samples causes the resulting interpolated signal to be shifted by n0∆X
Because this 1D interpolator is integer shift invariant we can also write it in a form like (IN1)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆X) (IN4)
for some 1D interpolation kernel h(x) When we are given a ldquoreciperdquo like ldquouse the nearest neighborrdquo how can we determine thatkernel From (IN4) we see that it suffices to use a Kronecker impulse gd[n] = δ[n] because h(x) =
suminfinn=minusinfin δ[n]h(xminus n∆X)
So interpolating a Kronecker impulse yields the interpolation kernel for integer shift invariant interpolation methods For 1DNN interpolation
h(x) = δ[round(x∆X)] =
1 minus12 le x∆X lt 120 otherwise -
x
6h(x)
-1 -05 0 05 1
1(IN5)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 2: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/2.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN2
Motion estimation for video temporal interpolation IN49Translation motion estimation IN52Region-matching methods IN53Space-time constraint methods IN56Region selection IN59Advanced topics IN59Motion compensated temporal interpolation IN60Application of motion estimation methods to spatial interpolation IN62
Summary IN62
Introduction
This chapter focuses on practical 2D interpolation methods Image interpolation is very important and used widely The ideasgeneralize readily to 3D interpolation problems
We also consider motion estimation and video interpolation In the context of video ie temporal image sequences temporalinterpolation can also be important Even though video is ldquo2D+timerdquo which is a kind of ldquo3D problemrdquo we use techniques thatdiffer from 3D spatial interpolation problems We focus initially on spatial interpolation
We focus on interpolation given equally spaced samples because those arise most often in DSP and image processing problemsBut there are also many applications that require interpolation from unequally spaced data and there are methods for those problemstoo [1 2] The MATLAB command griddata is one of those methods In 1D one classic method is spline interpolation
Nonlinear interpolation methods eg ldquomorphologicalrdquo methods have been proposed [3] but we focus on linear methods
For an excellent historical overview of interpolation see [4]
c⃝ J Fessler February 18 2013 1754 (student version) IN3
Problem statement and background
The image interpolation problem is the following We are given a discrete-space image gd[nm] that we assume corresponds tosamples of some continuous space image ga(x y) Typically we assume the samples are ideal samples taken on a rectilinear gridie gd[nm] = ga(n∆Xm∆Y) The usual goal is to compute values of the CS image ga(x y) at arbitrary (x y) locations
m
n
Digital image gd[nm]
82
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
14
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
1
0
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
91
0
0
65
99
99
99
99
99
99
69
28
0
0
0
71
99
99
99
92
0
0
0
50
57
5
6
5
3
22
0
0
70
7
0
0
98
99
93
1
0
23
50
0
2
1
1
26
55
0
24
69
4
0
0
0
99
93
3
0
5
9
0
30
23
63
23
83
42
27
83
83
40
0
0
64
1
5
0
0
2
0
7
31
26
53
53
0
0
83
83
91
0
0
2
0
0
0
0
38
0
3
53
24
49
50
15
2
85
82
95
69
0
7
0
0
0
0
95
0
0
23
2
12
50
40
61
57
85
95
17
0
99
1
0
0
0
1
95
6
0
2
0
3
1
3
21
45
0
0
80
99
1
2
2
3
0
0
27
56
0
85
82
0
0
0
0
0
5
99
99
2
2
2
0
6
65
63
99
57
13
40
50
0
0
0
35
99
99
99
3
3
1
0
52
59
54
99
99
99
80
13
17
99
99
99
99
99
99
37
8
7
74
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
rarrHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
There are two distinct situations where this type of operation is importantbull DA conversion such as a video monitor Here we really want a CS image ga(x y)bull Resampling situations where we want to evaluate ga(x y) at a discrete set of (x y) coordinates that are different from the
original sampling coordinates(n∆Xm∆Y)Why For image zooming display rotating warping coding motion estimation JAVA demonstration of the importance of good interpolation for image rotationhttpbigwwwepflchdemojrotationstartphp
Generally speaking interpolation is an example of an ill-posed inverse problem We are given only a finite or countable collectionof values gd[nm] yet we hope to recover ga(x y) a continuous-space function of uncountable arguments (x y) isin R2 Thereis an uncountably infinite collection of functions ga(x y) that agree with gd[nm] at the sample points (Connecting the dots is notunique) The only way for us to choose among the many possibilities is to make assumptions about the original CS signal ga(x y)Different interpolation methods are based on different assumptions or models
c⃝ J Fessler February 18 2013 1754 (student version) IN4
Ideal sinc interpolation
If gd[nm] = ga(n∆Xm∆Y) and if we assume that ga(x y) is band-limited and the sampling rates are appropriate and thesampling has infinite extent then in principle we can recover ga(x y) from gd[nm] exactly by the sinc interpolation formula
ga(x y) =infinsum
n=minusinfin
infinsumm=minusinfin
gd[nm]h(xminus n∆X y minusm∆Y) (IN1)
whereh(x y) =
This theoretical result is impractical because the sinc function (aka the sine cardinal function) has unbounded support and thesummations require infinitely many samples Furthermore real world images need not be exactly band-limited
In practice we always have a finite amount of data so we replace (IN1) by a practical method such as a general linear interpolator
ga(x y) ≜Nminus1sumn=0
Mminus1summ=0
gd[nm]h(xminus n∆X y minusm∆Y) (IN2)
Rarely will ga(x y) exactly match the true ga(x y) except perhaps at the sample locationsWe simply hope that ga(x y) asymp ga(x y) at least for 0 le x le N∆X and 0 le y leM∆Y
Different linear interpolation methods correspond to different choices for the interpolation kernel h(x y)
Note that any interpolator of the form (IN1) or (IN2) is linear in the sense that it is a linear function of the samples gd[nm]
Usually we choose interpolation kernels that are separable with the same 1D kernel shape in both x and y ie
h(x y) =
So we focus hereafter primarily on choices for the 1D interpolation kernel h(x) often with unit spacing ∆ = 1 for simplicity
c⃝ J Fessler February 18 2013 1754 (student version) IN5
Polynomial interpolation
We first consider polynomial interpolators ie those that have kernels that are piecewise polynomials
MATLABrsquos interp1 and interp2 commands support several such options
Zero-order or nearest neighbor (NN) interpolation
The simplest form of interpolation has the following recipe use the value at the nearest sample locationThat is a procedural description of the interpolator but we also want to have a mathematical description (and efficient software)A mathematical expression for this approach in 1D is
ga(x) = gd[round(x∆X)] = gd[n]∣∣∣n=round(x∆X)
(IN3)
One can verify that the interpolator (IN3) is integer shift invariant in the following sense If f [n] = gd[nminus n0] and interpolatef [n] using the same nearest neighbor rule (IN3) to obtain fa(x) then fa(x) = ga(xminus n0∆X) So shifting gd[n] by n0 samples causes the resulting interpolated signal to be shifted by n0∆X
Because this 1D interpolator is integer shift invariant we can also write it in a form like (IN1)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆X) (IN4)
for some 1D interpolation kernel h(x) When we are given a ldquoreciperdquo like ldquouse the nearest neighborrdquo how can we determine thatkernel From (IN4) we see that it suffices to use a Kronecker impulse gd[n] = δ[n] because h(x) =
suminfinn=minusinfin δ[n]h(xminus n∆X)
So interpolating a Kronecker impulse yields the interpolation kernel for integer shift invariant interpolation methods For 1DNN interpolation
h(x) = δ[round(x∆X)] =
1 minus12 le x∆X lt 120 otherwise -
x
6h(x)
-1 -05 0 05 1
1(IN5)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 3: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/3.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN3
Problem statement and background
The image interpolation problem is the following We are given a discrete-space image gd[nm] that we assume corresponds tosamples of some continuous space image ga(x y) Typically we assume the samples are ideal samples taken on a rectilinear gridie gd[nm] = ga(n∆Xm∆Y) The usual goal is to compute values of the CS image ga(x y) at arbitrary (x y) locations
m
n
Digital image gd[nm]
82
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
14
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
1
0
0
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
91
0
0
65
99
99
99
99
99
99
69
28
0
0
0
71
99
99
99
92
0
0
0
50
57
5
6
5
3
22
0
0
70
7
0
0
98
99
93
1
0
23
50
0
2
1
1
26
55
0
24
69
4
0
0
0
99
93
3
0
5
9
0
30
23
63
23
83
42
27
83
83
40
0
0
64
1
5
0
0
2
0
7
31
26
53
53
0
0
83
83
91
0
0
2
0
0
0
0
38
0
3
53
24
49
50
15
2
85
82
95
69
0
7
0
0
0
0
95
0
0
23
2
12
50
40
61
57
85
95
17
0
99
1
0
0
0
1
95
6
0
2
0
3
1
3
21
45
0
0
80
99
1
2
2
3
0
0
27
56
0
85
82
0
0
0
0
0
5
99
99
2
2
2
0
6
65
63
99
57
13
40
50
0
0
0
35
99
99
99
3
3
1
0
52
59
54
99
99
99
80
13
17
99
99
99
99
99
99
37
8
7
74
99
99
99
99
99
99
99
99
99
99
99
99
99
99
99
rarrHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
There are two distinct situations where this type of operation is importantbull DA conversion such as a video monitor Here we really want a CS image ga(x y)bull Resampling situations where we want to evaluate ga(x y) at a discrete set of (x y) coordinates that are different from the
original sampling coordinates(n∆Xm∆Y)Why For image zooming display rotating warping coding motion estimation JAVA demonstration of the importance of good interpolation for image rotationhttpbigwwwepflchdemojrotationstartphp
Generally speaking interpolation is an example of an ill-posed inverse problem We are given only a finite or countable collectionof values gd[nm] yet we hope to recover ga(x y) a continuous-space function of uncountable arguments (x y) isin R2 Thereis an uncountably infinite collection of functions ga(x y) that agree with gd[nm] at the sample points (Connecting the dots is notunique) The only way for us to choose among the many possibilities is to make assumptions about the original CS signal ga(x y)Different interpolation methods are based on different assumptions or models
c⃝ J Fessler February 18 2013 1754 (student version) IN4
Ideal sinc interpolation
If gd[nm] = ga(n∆Xm∆Y) and if we assume that ga(x y) is band-limited and the sampling rates are appropriate and thesampling has infinite extent then in principle we can recover ga(x y) from gd[nm] exactly by the sinc interpolation formula
ga(x y) =infinsum
n=minusinfin
infinsumm=minusinfin
gd[nm]h(xminus n∆X y minusm∆Y) (IN1)
whereh(x y) =
This theoretical result is impractical because the sinc function (aka the sine cardinal function) has unbounded support and thesummations require infinitely many samples Furthermore real world images need not be exactly band-limited
In practice we always have a finite amount of data so we replace (IN1) by a practical method such as a general linear interpolator
ga(x y) ≜Nminus1sumn=0
Mminus1summ=0
gd[nm]h(xminus n∆X y minusm∆Y) (IN2)
Rarely will ga(x y) exactly match the true ga(x y) except perhaps at the sample locationsWe simply hope that ga(x y) asymp ga(x y) at least for 0 le x le N∆X and 0 le y leM∆Y
Different linear interpolation methods correspond to different choices for the interpolation kernel h(x y)
Note that any interpolator of the form (IN1) or (IN2) is linear in the sense that it is a linear function of the samples gd[nm]
Usually we choose interpolation kernels that are separable with the same 1D kernel shape in both x and y ie
h(x y) =
So we focus hereafter primarily on choices for the 1D interpolation kernel h(x) often with unit spacing ∆ = 1 for simplicity
c⃝ J Fessler February 18 2013 1754 (student version) IN5
Polynomial interpolation
We first consider polynomial interpolators ie those that have kernels that are piecewise polynomials
MATLABrsquos interp1 and interp2 commands support several such options
Zero-order or nearest neighbor (NN) interpolation
The simplest form of interpolation has the following recipe use the value at the nearest sample locationThat is a procedural description of the interpolator but we also want to have a mathematical description (and efficient software)A mathematical expression for this approach in 1D is
ga(x) = gd[round(x∆X)] = gd[n]∣∣∣n=round(x∆X)
(IN3)
One can verify that the interpolator (IN3) is integer shift invariant in the following sense If f [n] = gd[nminus n0] and interpolatef [n] using the same nearest neighbor rule (IN3) to obtain fa(x) then fa(x) = ga(xminus n0∆X) So shifting gd[n] by n0 samples causes the resulting interpolated signal to be shifted by n0∆X
Because this 1D interpolator is integer shift invariant we can also write it in a form like (IN1)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆X) (IN4)
for some 1D interpolation kernel h(x) When we are given a ldquoreciperdquo like ldquouse the nearest neighborrdquo how can we determine thatkernel From (IN4) we see that it suffices to use a Kronecker impulse gd[n] = δ[n] because h(x) =
suminfinn=minusinfin δ[n]h(xminus n∆X)
So interpolating a Kronecker impulse yields the interpolation kernel for integer shift invariant interpolation methods For 1DNN interpolation
h(x) = δ[round(x∆X)] =
1 minus12 le x∆X lt 120 otherwise -
x
6h(x)
-1 -05 0 05 1
1(IN5)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 4: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/4.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN4
Ideal sinc interpolation
If gd[nm] = ga(n∆Xm∆Y) and if we assume that ga(x y) is band-limited and the sampling rates are appropriate and thesampling has infinite extent then in principle we can recover ga(x y) from gd[nm] exactly by the sinc interpolation formula
ga(x y) =infinsum
n=minusinfin
infinsumm=minusinfin
gd[nm]h(xminus n∆X y minusm∆Y) (IN1)
whereh(x y) =
This theoretical result is impractical because the sinc function (aka the sine cardinal function) has unbounded support and thesummations require infinitely many samples Furthermore real world images need not be exactly band-limited
In practice we always have a finite amount of data so we replace (IN1) by a practical method such as a general linear interpolator
ga(x y) ≜Nminus1sumn=0
Mminus1summ=0
gd[nm]h(xminus n∆X y minusm∆Y) (IN2)
Rarely will ga(x y) exactly match the true ga(x y) except perhaps at the sample locationsWe simply hope that ga(x y) asymp ga(x y) at least for 0 le x le N∆X and 0 le y leM∆Y
Different linear interpolation methods correspond to different choices for the interpolation kernel h(x y)
Note that any interpolator of the form (IN1) or (IN2) is linear in the sense that it is a linear function of the samples gd[nm]
Usually we choose interpolation kernels that are separable with the same 1D kernel shape in both x and y ie
h(x y) =
So we focus hereafter primarily on choices for the 1D interpolation kernel h(x) often with unit spacing ∆ = 1 for simplicity
c⃝ J Fessler February 18 2013 1754 (student version) IN5
Polynomial interpolation
We first consider polynomial interpolators ie those that have kernels that are piecewise polynomials
MATLABrsquos interp1 and interp2 commands support several such options
Zero-order or nearest neighbor (NN) interpolation
The simplest form of interpolation has the following recipe use the value at the nearest sample locationThat is a procedural description of the interpolator but we also want to have a mathematical description (and efficient software)A mathematical expression for this approach in 1D is
ga(x) = gd[round(x∆X)] = gd[n]∣∣∣n=round(x∆X)
(IN3)
One can verify that the interpolator (IN3) is integer shift invariant in the following sense If f [n] = gd[nminus n0] and interpolatef [n] using the same nearest neighbor rule (IN3) to obtain fa(x) then fa(x) = ga(xminus n0∆X) So shifting gd[n] by n0 samples causes the resulting interpolated signal to be shifted by n0∆X
Because this 1D interpolator is integer shift invariant we can also write it in a form like (IN1)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆X) (IN4)
for some 1D interpolation kernel h(x) When we are given a ldquoreciperdquo like ldquouse the nearest neighborrdquo how can we determine thatkernel From (IN4) we see that it suffices to use a Kronecker impulse gd[n] = δ[n] because h(x) =
suminfinn=minusinfin δ[n]h(xminus n∆X)
So interpolating a Kronecker impulse yields the interpolation kernel for integer shift invariant interpolation methods For 1DNN interpolation
h(x) = δ[round(x∆X)] =
1 minus12 le x∆X lt 120 otherwise -
x
6h(x)
-1 -05 0 05 1
1(IN5)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 5: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/5.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN5
Polynomial interpolation
We first consider polynomial interpolators ie those that have kernels that are piecewise polynomials
MATLABrsquos interp1 and interp2 commands support several such options
Zero-order or nearest neighbor (NN) interpolation
The simplest form of interpolation has the following recipe use the value at the nearest sample locationThat is a procedural description of the interpolator but we also want to have a mathematical description (and efficient software)A mathematical expression for this approach in 1D is
ga(x) = gd[round(x∆X)] = gd[n]∣∣∣n=round(x∆X)
(IN3)
One can verify that the interpolator (IN3) is integer shift invariant in the following sense If f [n] = gd[nminus n0] and interpolatef [n] using the same nearest neighbor rule (IN3) to obtain fa(x) then fa(x) = ga(xminus n0∆X) So shifting gd[n] by n0 samples causes the resulting interpolated signal to be shifted by n0∆X
Because this 1D interpolator is integer shift invariant we can also write it in a form like (IN1)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆X) (IN4)
for some 1D interpolation kernel h(x) When we are given a ldquoreciperdquo like ldquouse the nearest neighborrdquo how can we determine thatkernel From (IN4) we see that it suffices to use a Kronecker impulse gd[n] = δ[n] because h(x) =
suminfinn=minusinfin δ[n]h(xminus n∆X)
So interpolating a Kronecker impulse yields the interpolation kernel for integer shift invariant interpolation methods For 1DNN interpolation
h(x) = δ[round(x∆X)] =
1 minus12 le x∆X lt 120 otherwise -
x
6h(x)
-1 -05 0 05 1
1(IN5)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 6: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/6.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN6
This formula assumes 05 is rounded up to 1 which is how the MATLAB and C round function works Unfortunately MATLABand C round function rounds away from 0 which is not quite integer shift invariant for x le 12 To avoid this replace round(x)with floor(x+05)
These technicalities about what to do when x∆X = n + 05 for n isin Z stem from the ambiguity of the original recipe There isnot a unique ldquonearest neighborrdquo at these midway points A more precise recipe is ldquouse the nearest neighbor except at points thatare midway between two samples use the sample to the rightrdquo This more precisely stated interpolator has the mathematical form(IN4) with interpolation kernel (IN5) We often write the interpolation kernel for NN interpolation as a rect function
h(x) = rect(x∆X) =
1 minus12 le x∆X lt 120 otherwise
(IN6)
Up until this point the definition of the rect function at x = plusmn12 was unimportant But for interpolation it is important and theappropriate definition is that in (IN5) for the ldquorounding uprdquo convention
Exercise Determine h(x) for the modified NN interpolator where at the midway points one uses the average of the two nearestneighbors
Although the general form (IN4) and interpolation kernel (IN5) are convenient for mathematical analysis the software implemen-tation of 1D NN interpolation looks far more like the ldquoreciperdquo version (IN3)
gd = [2 1 2 4 3] signal sample valuesN = length(gd)n = 1Nx = linspace(05N+0497(N+1)+1)dx = 1index = round(x dx) 1 N hopefullygi = gd(index)plot(1N gd rsquoorsquo x gi rsquorsquo)
0
1
4
5
0 1 2 3 4 5 6
gd[n] and ga(x)
x
Exercise How to modify code to handle values of x outside the range of the sample locations ie per (IN2)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 7: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/7.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN7
2D NN interpolation
In 2D using the NN interpolator (with the same ldquorounding uprdquo convention as in 1D) can be written as
ga(x y) = gd[round(x∆X) round(y∆Y)]
Within the region that is sampled this type of interpolation is equivalent to using (IN2) with the following kernel
h(x) = rect(x) =rArr h(x y) =
Again in a practical implementation with a finite number of samples one must handle (x y) locations that are outside of thesample locations appropriately The ldquonearest neighborrdquo can be much less ldquonearrdquo for such samples
End conditions
The expression (IN2) evaluates to zero for (x y) locations not within the support rectangle
[minusw∆X (N minus 1 + w)∆Xminusw∆Y (M minus 1 + wy)∆Y]
where w is the half-width of the kernel h(x)
Often we want to extrapolate the border values instead of using zero
MATLABrsquos interp1 and interp2 commands offer options to control the extrapolation method
Exercise Compare the MATLAB output for the following interp1 options
interp1([0 1] [10 20] [-1 0 1 2])interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo 0)
interp1([0 1] [10 20] [-1 0 1 2] rsquonearestrsquo rsquoextraprsquo)
How would we write the last option mathematically
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 8: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/8.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN8
1D linear interpolation
The linear interpolation method is the second simplest method As a recipe in 1D we interpolate by connecting the two nearestpoints by a line This interpolator literally ldquoconnects the dotsrdquo with line segments It assumes (implicitly) that ga(x) is piecewiselinear A mathematical formula for this recipe is
ga(x) = (nx + 1minus x∆X) gd[nx] + (x∆X minus nx) gd[nx + 1] nx ≜ lfloorx∆Xrfloor (IN7)
This interpolation procedure is also integer shift invariant so we can determine the corresponding interpolation kernel by inter-polating the Kronecker impulse gd[n] = δ[n] The result is a triangular function
h(x) = tri(x∆X) -x
6h(x)
minus∆X 0 ∆X
1
Thus a mathematical expression for linear interpolation is
ga(x) =sumn
gd[n] tri
(xminus n∆X
∆X
)
This expression is convenient for mathematical analysis but the software implementation usually looks more like (IN7)
Example
-n
6gd[n]
0 1 2 3 4 5 6
1234
-x
6ga(x)
0 ∆X 2∆X 3∆X 4∆X 5∆X 6∆X
1234
Here there is an additional meaning of the word linear because the tri function is a piecewise linear function of its (spatial)argument The more general meaning of linear is that the interpolated signal ga(x) is a linear function of the samples gd[n]
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 9: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/9.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN9
2D linear or bilinear interpolation
For bilinear interpolation of 2D samples roughly speaking one first doeslinear interpolation along x and then linear interpolation along y (or viceversa)More precisely the procedure is to find the four nearest sample locations in-terpolate the top and bottom pair along x and then interpolate the resultingtwo points along y This figure illustrates the process graphically for the case∆X = ∆Y = 1
-x
6y
0 1 2 3
1
2
Clearly this process is (integer) shift invariant so we can determine its interpolation kernel by interpolating the 2D Kroneckerimpulse δ2[nm] For a point (x y) that is within the unit square [0 1] times [0 1] one can verify that the interpolation kernel is(1minusx)(1minusy) Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the interpolationkernel is (1minus |x|)(1minus |y|) Otherwise the kernel is zero ie
h(x y) =
(1minus |x|)(1minus |y|) |x| lt 2 |y| lt 20 otherwise
In other words bilinear interpolation is equivalent to using (IN1) or (IN2) with the following interpolation kernel
h(x y) =
An alternative way to view bilinear interpolation is as follows For any point (x y) we find the four nearest sample points and fitto those four points a polynomial of the form
α0 + α1x+ α2y + α3xy (IN8)
Then we evaluate that polynomial at the desired (x y) location
Example Consider a point (x y) that is within the unit square [0 1] times [0 1] Then the four nearest samples are gd[0 0] gd[1 0]gd[0 1] gd[1 1] To fit the polynomial we set up 4 equations in 4 unknowns
1 0 0 01 1 0 01 0 1 01 1 1 1
α0
α1
α2
α3
=
gd[0 0]gd[1 0]gd[0 1]gd[1 1]
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 10: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/10.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN10
Solving for the coefficients and substituting into the polynomial and simplifying yields
gd[0 0](1minus x)(1minus y) + gd[1 0]x(1minus y) + gd[0 1](1minus x)y + gd[1 1]xy
Considering the symmetries of the problem for locations (x y) within the square [minus1 1]times [minus1 1] the gd[0 0] value will in generalbe multiplied by (1minus|x|)(1minus|y|) rect(x2) rect(y2) = tri(x) tri(y) (Because the method is invariant to integer shifts it sufficesto find the expression for the gd[0 0] term)
The following figure shows the (separable) bilinear interpolation kernel and its contours and profilesNote that the profile along the diagonal of the kernel is not a piecewise linear function
Bilinear kernel
minus1 0 1
minus1
minus05
0
05
1
0
1
minus10
1minus1
01
0
05
1
h(xy)
minus1 0 1minus1
minus05
0
05
1
x
y
Contours
minus1 0 10
05
1
x
h(x
0) a
nd h
(xx
)
Profiles
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 11: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/11.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN11
Example Here is an example of nearest neighbor and bilinear interpolation of a famous image
The ldquopyramidrdquo shape of the interpolation kernel is evident and somewhat undesirableHarmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
Bilinear interpolation
1
19
1 15
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 12: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/12.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN12
An alternative interpolation approach that is strictly piecewise-linear is to first use Delaunay triangulation of the sample loca-tions (this is applicable even to non-rectilinear samples) and then use a linear function within each triangle (because three pointsdetermine a plane) This is how MATLABrsquos griddata works with the rsquolinearrsquo option The following figures show the result ofinterpolating δ2[nminus n0mminusm0] for 9 choices of (n0m0) locations
The interp2 bilinear method is invariant to integer shifts which is called shift invariant in the context of interpolation methodsIn contrast this griddata approach is shift variant due to the triangulation used These griddata results are piecewiselinear functions whereas these interp2 results are piecewise polynomials of the form (IN8)
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
interp2 rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
griddata rsquolinearrsquo
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
minus1 0 1
minus101
The bottom line is that griddata is not a good method for interpolating equally spaced data
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 13: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/13.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN13
Interpolator properties
The preceding interpolators are used quite frequently
Why should we need anything elseOne reason is that neither the zero-order or bilinear interpolator are differentiable
Before proceeding it seems reasonable to assess desirable properties of interpolators
In general desirable properties of interpolation kernels include the following (See [5] for an excellent modern treatment)bull h(0) = 1 and h(n) = 0 for n = plusmn1plusmn2 ie
h(n) = δ[n] -x
6h(x)
-2 -1 0 1 2
1
This property ensures the interpolation property holds ie that the following ldquoself consistencyrdquo holds
ga(x)∣∣∣x=n∆
=infinsum
m=minusinfinga(m∆)h(xminusm∆)
∣∣∣∣∣x=n∆
= ga(n∆) = gd[n] (IN9)
bull Continuous and smooth (differentiable) (particularly when derivatives are of interest)
d
dxga(x) =
infinsumn=minusinfin
gd[n]d
dxh(xminus n∆)
bull Short spatial extent (support) to minimize computation If S sub R denotes the support of h(x) then (Picture)
ga(x) =infinsum
n=minusinfingd[n]h(xminus n∆) =
sumnisinZ xminusn∆isinS
gd[n]h(xminus n∆)
bull Frequency response approximately rect(ν)bull Symmetric (To ensure invariance to mirror images)bull Small side lobes to avoid ringing ldquoartifactsrdquobullsum
n h(xminus n) = 1 forallx isin R (Perfect DC interpolation also called partition of unity)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 14: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/14.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN14
bull More generally we might want to be able to interpolate (sampled) monomials xm perfectly If h(x) perfectly interpolatesmonomials up to xLminus1 then L is called the approximation order of the interpolator [5] The reason for calling it ldquoLrdquo insteadof ldquoLminus 1rdquo is that for an interpolator having approximation order L the approximation error decays as ∆L
∥g minus g∥L2= Ch∆
L∥∥∥g(L)
∥∥∥L2
as ∆rarr 0
for some constant Ch that depends on h(x) where ∥g∥2L2=int|g(x)|2 dx and g(L) denotes the Lth derivative of g(x)
Example The linear interpolation kernel tri(x) can recover affine functions ga(x) = a + bx from samples perfectly so itsapproximation order is L = 2
One must choose between these considerations they cannot all be achieved simultaneously
Why is circular symmetry missing from this list The only nontrivial separable function that is circularly symmetric is the2D gaussian which does not satisfy the interpolation property
Example The following figure illustrates why we may prefer interpolators with small side lobes
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
x
f(x)
fr(x
)
Sinc interpolation
minus6 minus4 minus2 0 2 4 6minus02
0
02
04
06
08
1
12
xf(
x) f
r(x)
Linear interpolation
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Sinc Interpolation Error
minus6 minus4 minus2 0 2 4 6minus05
0
05
x
f r(x)
minus f(
x)
Linear Interpolation Error
(Nonminusbandlimited signal)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 15: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/15.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN15
Lagrange interpolationThe classical Lagrange interpolation method works as followsbull For any given value of x choose the M + 1 nearest sample locations (a sliding block)bull Fit a M th-degree polynomial to those nearest M + 1 pointsbull Evaluate fitted polynomial at desired location(s) M = 2 example
-x
6
0 1 2 3 4
Remarksbull See httpmathworldwolframcomLagrangeInterpolatingPolynomialhtmlbull This method is used fairly frequentlybull Equivalent to linear interpolation in 1D for M = 1bull One can smooth noise by using more points than the polynomial degree and performing a least-squares fit
Does this method have the form (IN1) In other words in 1D can we write ga(x) =sum
n gd[n]h(
xminusn∆X
∆X
)
Yes the method is a linear function of the data values and is (integer) shift invariant
Knowing that the method is linear and (integer) shift invariant we can determine the corresponding kernel h(x) by ldquointerpolatingrdquoa Kronecker impulse signal gd[n] = δ[n] for the case ∆X = 0 because in this situation ga(x) = h(x)
Example Consider the case M = 2 We will determine h(x)bull If |x| le 12 then the M + 1 = 3 nearest sample points are minus1 0 1 and the corresponding 2nd-degree polynomial is 1minus x2bull If 12 le x le 32 then the M + 1 = 3 nearest sample points are 0 1 2 and for a Kronecker impulse the corresponding
2nd-degree polynomial (which has roots at 1 and 2) can be found to be (xminus 1)(xminus 2)2bull If 32 le x then the M + 1 = 3 nearest sample points all have value zerobull Combining using symmetry we conclude that the interpolation kernel for M = 2 is
h(x) =
1minus x2 |x| le 12(|x| minus 1)(|x| minus 2)2 12 le |x| le 320 otherwise
minus3 minus2 minus1 0 1 2 3
0
05
1
x
quadratic fits (M=2)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
Lagrange interpolation kernel (M=2)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 16: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/16.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN16
Quadratic interpolation
What is the approximation order of a Lagrange interpolator with M The Lagrange interpolator for M = 2 is piecewise quadratic and it has approximation order 3 meaning that it can interpolate(samples of) polynomials of degree 2 perfectly but it is discontinuous which seems quite undesirable
Other quadratic interpolation kernels have been proposed These seem to be used less than the cubic interpolators described nextApparently this is in part due to a misperception that quadratic interpolators are nonzero phase [6]
A quadratic kernel on [minus1 1] that is continuous differentiable nonnegative and has a continuous first derivative is
h(x) =
1minus 2x2 |x| le 122(|x| minus 1)2 12 lt |x| le 10 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
Putting the ldquobreak pointrdquo at x = 12 is a design choice One could put it anywhere between 0 and 1Exercise Can you think of any advantage to putting the break at x = 12
Exercise What is the approximation order of this kernel (See figures next page) Dodgson [6] discusses a different quadratic kernel
h(x) =
1minus 2|x|2 |x| le 12|x|2 minus 52|x|+ 32 12 lt |x| lt 320 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This kernel has support [minus32 32] which is consistent with the rect tri quad cubic series but it is not differentiable at plusmn32
Exercise What is the approximation order of this kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 17: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/17.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN17
Example The following figure illustrates empirically the approximation orders of the two preceding interpolation kernels
What tradeoff is illustrated here
0 10
0
10t
hminusde
gree
Differentiable Kernel
0 10
0
1
Dodgson
0 10minus05
0
05Dodgson Error 100
0 10
0
1
1stminus
degr
ee
0 10
0
1
0 10minus05
0
05
0 10
0
1
2ndminus
degr
ee
x0 10
0
1
0 10minus05
0
05
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 18: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/18.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN18
Cubic interpolation
In the literature there are quite a few flavors of cubic interpolators The typical choices of these kernels are differentiable whichis convenient for taking derivatives
Care must be taken when using cubic interpolation there are incorrect formulations in the literature that have led to false conclu-sions See [7] [8] for a nice analysis based on B-splines There is a family of cubic interpolators described in [9]
Older MATLAB versions used cubic interpolation based on a paper by R Keys [10] that describes the following interpolationkernel
h(x) =
1minus 52 |x|
2+ 3
2 |x|3 |x| le 1
2minus 4 |x|+ 52 |x|
2 minus 12 |x|
3 1 lt |x| lt 2
0 otherwise minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
One can still use this kernel by selecting the v5cubic option to interp1
In more recent versions selecting the cubic option to interp1 corresponds to ldquopiecewise cubic Hermite interpolationrdquo [11]which curiously is nonlinear
0
1
-2 -1 0 1 2 3
input
0
1
-2 -1 0 1 2 3
interp1 output
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
0
1
-2 -1 0 1 2 3
n
0
1
-2 -1 0 1 2 3
x
interp1 rsquocubicrsquosuperposition
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 19: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/19.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN19
As of version R2011a MATLABrsquos imresize command continues to use the Keyrsquos cubic kernel [10]
An alternative cubic interpolation kernel having smaller support is
h(x) =(1minus 3x2 + 2 |x|3
)rect
(x2
)
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
This is the unique symmetric function that is cubic on [01] and satisfies h(0) = 1 h(0) = 0 h(x)+h(1minus x) = 1
It also happens to be the output of MATLABrsquos default cubic interpolator for a Kronecker impulse input
Its spectrum is
H(ν) =
int 1
minus1
(1minus 3x2 + 2 |x|3
)eminusı2πνx dx = 2
int 1
0
(1minus 3x2 + 2x3
)cos(2πνx) dx
=
3minus 3πν sin(2πν)minus3 cos(2πν)
2π4ν4 ν = 0
1 ν = 0
(The symbolic toolbox in MATLAB is convenient for such integrals)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 20: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/20.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN20
Example The following figure shows that the ldquosmall cubicrdquo interpolation kernel has approximation order
whereas Keys has approximation order
0 100
10t
hminusde
gree
Small cubic
0 100
1
Keys cubic (v5cubic)
0 10minus01
01Keys error 100
0 100
1
1stminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
2ndminus
degr
ee
0 100
1
0 10minus01
01
0 100
1
3rdminus
degr
ee
x0 10
0
1
0 10minus01
01
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 21: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/21.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN21
The following figures illustrate that for interpolators based on piecewise polynomials generally as the polynomial degree increasesbull the interpolator support widens typically increasing computationbull the frequency response can better approximate the ldquoidealrdquo rect(ν)
minus3 minus2 minus1 0 1 2 3
0
1
x
h(x)
Interpolation kernels in 1D
nearestlinearv5cubic
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel v5cubic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
minus2 minus1 0 1 2
0
1
x
h(x)
Interpolation kernel Quadratic
minus05 0 05
0
1
ν
H(ν
)
Frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 22: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/22.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN22
Non-polynomial interpolation Lanczos kernel
MATLABrsquos imresize function includes two Lanczos interpolation kernels defined as follows
h2(x) ≜ sinc(x) sinc(x2) rect(x4
) h3(x) ≜ sinc(x) sinc(x3) rect
(x6
)
These are both special cases of the general Lanczos interpolation kernel of the following form
ha(x) ≜ sinc(x)︸ ︷︷ ︸ideal
kernel
sinc( x
2a
)rect
( x
2a
)︸ ︷︷ ︸
central lobestretched over
[minusa a]
minus3 minus2 minus1 0 1 2 3minus02
0
02
04
06
08
1
12
xh(
x)
Lanczos 2Lanczos 3
This kernel is twice differentiable for a isin N Some investigators have described it as a favorable compromise in terms of reductionof aliasing sharpness and minimal ringing Of course it is somewhat more expensive to compute than a polynomial
See Graphics Gems Andrew S Glasser (ed) Morgan Kaufman 1990 pp 156-158httpenwikipediaorgwikiLanczos_resampling
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 23: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/23.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN23
Interpolation in shift-invariant subspaces
The ldquoconsistencyrdquo requirement (IN9) makes sense if we use the signal samples ga(m∆) themselves as the coefficients in (IN9)but one can generalize this type of expression by allowing other values to serve as the coefficients The B-spline interpolatorsdiscussed next use this generalization
Consider the 1D case with ∆X = 1 for simplicityThus far we have used the following type of linear interpolation
ga(x) =sumn
gd[n]︸ ︷︷ ︸signal
samples
h(xminus n)︸ ︷︷ ︸interpolation
kernel
(IN10)
It is quite feasible to generalize this approach to consider arbitrary interpolation coefficients
ga(x) =sumn
c[n] b(xminus n) =sumk
c[k] b(xminus k) (IN11)
Clearly we will have to determine the coefficients c[n] from the given samples gd[n] somehowThe set of signals b(middot minus k) k isin Z is not orthogonal in generalWe now use the notation b(x) to emphasize that b(x) is part of a basis and no longer is necessarily the impulse response
The representation (IN11) is a subspace of all possible signals and such subspaces are called shift invariant subspaces(One must remember that they are invariant only to integer shifts in general)
Model S = g g(x) =sum
k c[k] b(xminus k∆) has property f isin S =rArr f(middot minus∆) isin S
With this more general representation it is not necessary for the basis kernel b(x) to satisfy the ldquosampling conditionsrdquo b(0) = 1and b(n) = 0 for n isin Zminus 0 This opens the door to using many possible kernels
Suppose we have chosen an basis kernel b(x) How shall we determine the coefficients c[n]A natural approach is to require that (IN11) exactly interpolate the given data samples ie we want
ga(n) = gd[n]
for all n isin N where N is the set of sample indices Typically N = 0 N minus 1 or N = Z
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 24: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/24.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN24
Consider first the case N = Z because it is easier to analyze The interpolation condition above is equivalent to requiring
gd[n] =infinsum
k=minusinfin
c[k] b(nminus k) = c[n] lowast b1[n]
where we define b1[n] ≜ b(n) = b(x)|x=n to denote the integer samples of the kernel b(x)
Using the convolution property of the DSFT the above expression becomes
Gd(ω) = C(ω)B1(ω) =rArr C(ω) = Gd(ω) B1(ω)
Define the inverse filter binv[n]DSFTlarrrarr Binv(ω) =
1B1(ω) so that b1[n] lowast binv[n] = δ[n] Then it follows that we can determine the
interpolation coefficients by prefiltering the signal samples
c[n] = binv[n] lowast gd[n] (IN12)
Here is a summary of this approach to interpolationbull Start with samples gd[n] n isin Zbull Choose an basis kernel b(x)bull Determine the samples b1[n] = b(n) of that kernelbull Determine the inverse filter binv[n] for that sampled kernel such that b1[n] lowast binv[n] = δ[n]
and equivalently Binv(ω) = 1B1(ω) Often this step is done with Z-transformsbull Convolve (ldquoprefilterrsquo) the samples gd[n] with the inverse filter binv[n] to get the interpolation coefficients c[n] = binv[n] lowast gd[n] bull Use those coefficients in the linear interpolation summation (IN11) for any and all values of x of interest
Summary ga(x)rarrideal
sampling rarr gd[n]rarrprefilterbinv[n]
rarr c[n]rarr ldquointerpolaterdquosumk c[k] b(xminus k)
rarr ga(x)
Example Suppose we choose a triangle kernel b(x) = tri(x) Then b1[n] = δ[n] so binv[n] = δ[n] and thus c[n] = gd[n] and theldquogeneralizedrdquo method reverts to conventional linear interpolation with impulse response h(x) = tri(x)
Example Suppose we choose a triangle kernel b(x) = tri(xw
)where 1 lt w lt 2
Then b1[n] = a δ[n+ 1]+ δ[n] +a δ[nminus 1] where a = 1minus 1w isin (0 12)
To determine binv[n] note that B1(z) = az + 1 + azminus1 so Binv(z) =1
B1(z)=
1
az + 1 + azminus1
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 25: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/25.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN25
We will see shortly that this is a stable filter that can be implemented as two passes of an IIR filter one pass from left to right(causal) and one pass from right to left (anti-causal) Applying these IIR filters to gd[n] yields the coefficients c[n]
The equivalent impulse response
Combining (IN11) and (IN12) we see that this kind of interpolator has the form
ga(x) =
infinsumn=minusinfin
c[n] b(xminus n) =
infinsumn=minusinfin
(binv[n] lowast gd[n]) b(xminus n) =
infinsumn=minusinfin
( infinsumk=minusinfin
binv[nminus k] gd[k]
)b(xminus n)
=
infinsumk=minusinfin
gd[k]
( infinsumn=minusinfin
binv[nminus k] b(xminus n)
)=
infinsumk=minusinfin
gd[k]
( infinsumm=minusinfin
binv[m] b(xminus k minusm)
)=
infinsumk=minusinfin
gd[k]h(xminus k)
where we let m = nminus k and the equivalent impulse response for this interpolation method is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) (IN13)
In other words (IN11) is equivalent mathematically to linear interpolation of the form (IN10) with the interpolation kernel (IN13)Although this shows there is ldquonothing newrdquo mathematically from a practical perspective there is an important difference becauseusually (IN13) has infinite support so it is impractical whereas (IN11) is practical if we choose b(x) to have finite support
The equivalent frequency response
Taking the (1D CS) Fourier transform of both sides yields the spectrum of the interpolation kernel
H(ν) =
infinsumm=minusinfin
binv[m]B(ν) eminusı2πνm = B(ν)
infinsumm=minusinfin
binv[m] eminusı2πνm = B(ν) Binv(ω)∣∣∣ω=2πν
=B(ν)
B1(2πν)(IN14)
where b(x)Flarrrarr B(ν) and binv[n]
DTFTlarrrarr Binv(ω) and b1[n]DTFTlarrrarr B1(ω)
Using small interpolators limits how well the numerator B(ν) in (IN14) can approximate the ideal rect(ν) The presence of thedenominator offers the opportunity to shape the frequency response
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 26: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/26.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN26
Example For the triangle basis kernel b(x) = tri(xw) discussed above the equivalent interpolation kernel has spectrum
H(ν) =w sinc2(wν)
a eı2πν + 1 + a eminusı2πν=
w sinc2(wν)
1 + 2wminus1w cos(2πν)
The following figure shows the frequency response of the equivalent interpolation kernel for some values of the width parameterw
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Frequency response of shift-invariant interpolation using tri(xw)
idealw = 1
w = 125w = 15w = 175
It is debatable whether any of the values of w gt 1 are better than the usual linear interpolation kernel where w = 1Perhaps tri(xw) is not a great choice for the basis kernel
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 27: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/27.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN27
Basis kernels supported on [-22]
For any kernel supported on [minus1 1] what is B1(ω) In other words we need kernels with larger support than [minus1 1] to provide the opportunity to shape the spectrumTherefore we now focus on basis kernels supported on [minus2 2] such as the Keys interpolator described previously This familyincludes those supported on [minus32 32] which also can be useful
minus3 minus2 minus1 0 1 2 3
0
05
1
x
h(x)
We want such kernels to be symmetric and continuous so we must have b(plusmn2) = 0Therefore the general form of the unit-spaced samples of such kernels is
b1[n] = a δ[n+ 1]+d δ[n] +a δ[nminus 1]
where a = b(1) d = b(0) We assume d gt 0 and a gt 0 hereafter (The case a = 0 is trivial) The frequency response of b1[n] is
B1(ω) = a eıω + d+ a eminusıω = d+ 2a cos(ω)
If dminus 2a gt 0 then B1(ω) gt 0forallω and the filter may be invertible Thus hereafter we assume d gt 2a gt 0 ie b(0) gt 2 b(1) gt 0This condition excludes b(x) = tri(x2) from consideration for example
Recall that the Z-transform of a signal is related to the DTFT of that signal by z = eıω
To determine binv[n] note that b1[n]Zlarrrarr B1(z) = az + d+ azminus1 so defining r =
d
2agt 1 we have
binv[n]Zlarrrarr Binv(z) =
1
B1(z)=
1
az + d+ azminus1=
za
z2 + 2rz + 1=
za
(z minus p)(z minus pminus1)=
1a
1minus pzminus1︸ ︷︷ ︸causal
zminus1
1minus pminus1zminus1︸ ︷︷ ︸anti-causal
(IN15)
where 2r = minuspminus pminus1 so p = minusr plusmnradicr2 minus 1
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 28: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/28.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN28
This form of Binv(z) is the product of two IIR filters one that is causal and one that is anti-causal This factorization will beuseful only if those filters are stable For the causal filter that means we need |p| lt 1 (pole within the unit circle) whereas for theanti-causal filter we need the pole at pminus1 to be outside the unit circle which again means we need |p| lt 1
Note from our earlier condition on a and d that r gt 1 Therefore we choose the following root
p = minusr +radicr2 minus 1
One can verify that this root satisfies minus1 lt p lt 0 so p is within the unit circle In terms of the basis kernel
p = minus b(0)
2 b(1)+
radic(b(0)
2 b(1)
)2
minus 1
By partial fraction expansion (PFE) or by using a table of Z-transforms one can verify the following Z-transform pair
p|n|Zlarrrarr (pminus pminus1)zminus1
1minus (p+ pminus1)zminus1 + zminus2=
(pminus pminus1)zminus1
(1minus pzminus1)(1minus pminus1zminus1) |p| lt |z| lt 1
|p|
Thus by (IN15) the impulse response of the inverse filter is
binv[n] =1
a(pminus pminus1)p|n| (IN16)
This is a relatively rare situation where an inverse filter (a deconvolution filter) has a simple explicit form for its impulse response
Using (IN15) the corresponding frequency response of this inverse filter is
binv[n]DTFTlarrrarr Binv(ω) =
1
d+ 2a cos(ω) (IN17)
This will be a ldquohigh boostrdquo filter
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 29: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/29.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN29
Example The following figure illustrates the case where b(0) = 68 b(1) = 18 This corresponds to the quadratic B-splinediscussed later for example
0
0125
075
-10 -5 0 5 10
b1[n]
0
141421
-10 -5 0 5 10
binv[n]
0
1
-10 -5 0 5 10binv[n] b1[n]
n
0
05
1
-p 0 p
B1(
ω)
0
1
2
-p 0 p
Binv(
ω)
0
1
2
-p 0 p
Binv(
ω) B1(
ω) = 1
ω
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 30: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/30.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN30
Efficient computation of interpolation coefficients
To compute the interpolation coefficients c[n] we must prefilter the signal samples gd[n] per (IN12) ie c[n] = binv[n] lowast gd[n]
However convolving gd[n] with the impulse response (IN16) directly would be quite inefficient because binv[n] has infinitesupport In fact performing this convolution directly would require effort comparable to having used the infinitely long ideal sincinterpolator in the first place
Based on the previous example it appears that one could truncate the inverse filter binv[n] but the following approach is even moreefficient
The practical approach is to recognize that Binv(z) (IN15) represents the product of two IIR filters So we can compute c[n] fromgd[n] using two IIR filter passes one pass from left to right (causal) and one pass from right to left (anti-causal) A block diagramfor this operation uses the following IIR filter cascade
gd[n]rarr causal1a
1minus pzminus1y[n]minusrarr anti-causal
zminus1
1minus pminus1zminus1rarr c[n]
We implement the recursions as follows
y[n] =1
agd[n] +p y[nminus 1] (causal)
c[n] = p (c[n+ 1]minus y[n]) (anti-causal) (IN18)
These two filters require only 3 multiplies and 2 additions per sampleIn contrast if we truncated the inverse filter to |n| le 3 how many multiplies and adds would we need per sample
So the IIR approach requires less computation
As a side note the recursions (IN18) are very closely related to the O(N) algorithm for inverting a tridiagonal matrixhttpenwikipediaorgwikiTridiagonal_matrix_algorithm
Note how we have used CS FT and DS FT and Z-transform to analyze fully this type of interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 31: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/31.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN31
End conditions aka boundary conditions
In practical situations we have a finite number of samples n = 0 N minus 1 so we must initialize the IIR filters (IN18) properly
The output of the ideal causal filter is
y[n] =1
a
infinsumk=0
pk gd[nminus k] =rArr y[0] =1
a
infinsumk=0
pk gd[minusk] asymp1
a
Ksumk=0
pk gd[minusk]
where we choose K such that pK asymp 0
For periodic end conditions we assume gd[minusn] = gd[N minus n] and we have y[0] asymp 1a gd[0]+
1a
sumKk=1 p
k gd[N minus k]
For mirror end conditions we assume gd[minusn] asymp gd[n] and we have y[0] asymp 1a
sumKk=0 p
k gd[k]
Having chosen one of these initial conditions for y[0] we can then use the recursion in (IN18) for y[n] for n = 1 N minus 1
The output of the ideal anti-causal filter is
c[n] = minusinfinsumk=0
pk+1 y[n+ k] =rArr c[N minus 1] = minusinfinsumk=0
pk+1 y[N minus 1 + k] asymp minusKsum
k=0
pk+1 y[N minus 1 + k]
where we choose K such that pK asymp 0
For periodic end conditions we assume y[N minus 1 + k] = y[n] +[k minus 1] so c[N minus 1] asymp minusp y[N minus 1]minussumK
k=1 pk+1 y[k minus 1]
For mirror end conditions we assume y[N minus 1 + k] asymp y[N minus 1minus k] so c[N minus 1] asymp minussumK
k=0 pk+1 y[N minus 1minus k]
Unser [7 Box 2] recommends the following ldquoin placerdquo initialization for mirror end conditions
c[N minus 1] = minus p
1minus p2(y[N minus 1]+p y[N minus 2])
Having chosen one of these boundary conditions for c[N minus 1] we can then use the recursion in (IN18) for c[n] for n = Nminus1 0
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 32: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/32.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN32
Equivalent impulse response
Combining (IN16) with (IN13) we see that the equivalent impulse response for an interpolation method with a basis kernelsupported on [minus2 2] is
h(x) =infinsum
m=minusinfinbinv[m] b(xminusm) =
1
a(pminus pminus1)
infinsumm=minusinfin
p|n| b(xminusm) (IN19)
Likewise the equivalent frequency response from (IN14) is
H(ν) =B(ν)
B1(2πν)= (IN20)
The key point here is that the equivalent impulse response h(x) has infinite support but our interpolation process requires just twosimple first-order recursive filters and a finite-length basis kernel b(x)
Example We return to the triangular basis kernel b(x) = tri(xw) for w = 15 Here a = 1minus 1w = 13 and r = 1(2a) = 32
and p = minusr +radicr2 minus 1 =
radic5minus32 asymp minus0382 Using (IN19) yields the following equivalent impulse response h(x)
minus8 minus6 minus4 minus2 0 2 4 6 8
minus02
0
02
04
06
08
1
x
h(x)
tri(x15)sinc
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 33: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/33.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN33
B-spline interpolation
See article by Unser [7] available from link on web page
A B-spline model for ga(x) uses the shift-invariant space model (IN11) with basis kernel b(x) = β(m)(x) where β(m)(middot) is aB-spline of degree m For equally spaced sample points (called knots in the spline literature) one can define B-splines as follows
β(m)(x) = rect(x) lowast middot middot middot lowast rect(x)︸ ︷︷ ︸m+ 1 times
Flarrrarr B(m)(ν) =
To develop a more explicit expression for β(m)(x) one can use the following trick for ν = 0
B(m)(ν) =sinm+1(πν)
(πν)m+1= sinm+1(πν)(2ı)m+1
[1
ı2πν+
1
2δ(ν)
]m+1
=[eıπν minus eminusıπν
]m+1Stepm+1(ν)
= Stepm+1(ν)
m+1sumk=0
(m+ 1
k
)eminusıπνk (minus1)k eıπν(m+1minusk)
= Stepm+1(ν)m+1sumk=0
(m+ 1
k
)(minus1)k eminusı2πν[kminusm+1
2 ]
wherestep(x)
Flarrrarr Step(ν) =1
ı2πν+
1
2δ(ν)
1
mxm step(x)
Flarrrarr Stepm+1(ν) m isin 0 1 2
Letting (t)+ ≜ t step(t) and taking the inverse FT yields the following ldquoexplicitrdquo expression for a B-spline of degree m
β(m)(x) =1
m
m+1sumk=0
(m+ 1
k
)(minus1)k
(xminus k +
m+ 1
2
)m
+
(IN21)
As an alternative to the above Fourier derivation one can prove (IN21) by induction by using the following easily verified equality
(t)n+ lowast step(t) =
1
n+ 1(t)
n+1+
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 34: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/34.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN34
The following recursive expressions can also be useful
β(m)(x) = rect(x) lowastβ(mminus1)(x) =1
m
(x+
m+ 1
2
)β(mminus1)
(x+
1
2
)+
1
m
(m+ 1
2minus x
)β(mminus1)
(xminus 1
2
)
The derivatives also have a useful recursion
d
dxβ(m)(x) =
d
dx
(rect(x) lowastβ(mminus1)(x)
)=
(d
dxrect(x)
)lowast β(mminus1)(x) = (δ(x+ 12)minus δ(xminus 12)) lowast β(mminus1)(x)
= β(mminus1)(x+ 12)minusβ(mminus1)(xminus 12)
The support of β(m)(x) is [minus(m+ 1)2 (m+ 1)2] so m+ 1 points are needed for interpolation
Fact As m increases β(m)(x) appears more like a ldquobell curverdquo Why Fact As m increases the equivalent impulse response h(m)(x) for β(m)(x) approaches sinc(x) [12]
Example For m = 0 we have β(0)(x) = (x+ 12)0+ minus (xminus 1 + 12)0+ = step(x+ 12)minus step(xminus 12) = Note that this difference of step functions matches (IN6)
So nearest-neighbor interpolation and B-spline interpolation of degree m = 0 are equivalent
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 35: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/35.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN35
Example The cubic B-spline can be written
β(3)(x) =1
6
[(x+ 2)3+ minus 4(x+ 1)3+ + 6(x)3+ minus 4(xminus 1)3+ + (xminus 2)3+
]=
23minus x2 + 12 |x|
3 |x| le 1
16 (2minus |x|)
3 1 lt |x| lt 20 otherwise
(IN22)
and is shown below This function does not satisfy the interpolation property But it still can be used as a basis kernel in (IN11)
minus3 minus2 minus1 0 1 2 3
0
16
46
t
β3 (t)
Cubic Bminusspline and its underlying components
Exercise Consider derivatives of β(3)(x) near x = 2
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 36: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/36.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN36
Example Determine the basis kernel and equivalent frequency response for the quadratic case (m = 2)
β(2)(x) =1
2
[(xminus 32)2+ minus 3(xminus 12)2+ + 3(x+ 12)2+ minus (x+ 32)2+
]=
34minus x2 |x| lt 1212 (|x| minus 32)2 12 lt |x| lt 320 otherwise
So the sampled quadratic B-spline isb(2)1 [n] = β(2)(n) =
By (IN20) the equivalent frequency response is thus
H(2)(ν) =B(2)(ν)
B(2)1 (2πν)
=
0
18
68
-3 -2 -1 0 1 2 3
β2(x)
x
B-spline of degree 2 and its underlying components
0
1
-3 -2 -1 0 1 2 3
H(
ν)
ν
Equivalent frequency response when using B-spline (m=2)
ideal rectB-spline m=2
sinc3(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 37: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/37.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN37
The equivalent impulse response h(x) has the form (IN19) with r =β(2)(0)
2β(2)(1)= 3 and p =
radic8minus 3 asymp minus0172
-02
0
1
-6 -4 -2 0 2 4 6
x
h(x) for β(2)sinc
h(x)
Exercise Consider the cubic case (m = 3) and find H(ν)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 38: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/38.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN38
2D interpolation in shift-invariant spaces
In 2D using a separable basis kernel the interpolator has the following form
ga(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b
(xminus n∆X
∆X
)b
(y minusm∆Y
∆Y
) (IN23)
We compute the coefficients c[nm] from the image sample values gd[nm] by pre-filtering with a separable discrete-space filter
c[nm] = gd[nm] lowastlowast (binv[n] binv[m])
The practical implementation applies the appropriate IIR filters left to right and then back right to left and then top to bottom andback bottom to top We filter each row independently so this step can be parallelized Likewise for filtering the columns
A practical issue on modern computer systems is the memory access order it may be beneficial to transpose the image betweenthe row and column filtering operations so that the image pixel values being filtered are ordered sequentially in memory therebyimproving cache utilization
After computing the coefficients c[nm] for any location (x y) of interest we evaluate (IN23) efficiently considering the support(minusT T ) of b(x) as follows
ga(x y) =sum
m minusTltyminusm∆YltT
( sumn minusTltxminusn∆XltT
c[nm] b
(xminus n∆X
∆X
))b
(y minusm∆Y
∆Y
)
The inner summation requires 2T operations for each the 2T values of m in the outer sum and then the outer sum needs another2T operations so in total 4T 2 +2T operations are needed where each ldquooperationrdquo is an evaluation of b(x) a multiply and an addThe O(T 2) load is a significant motivator for using small values of T like T = 2 for a cubic spline
In 3D the computations are O(T 3) and most medical imaging problems are 3D so interpolation (eg for image registration) canrequire quite significant computation
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 39: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/39.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN39
Example The following figure shows an sampled image gd[nm] and the interpolation coefficients c[nm]
gd[nm]
1
120
1 100
prefilter each row
1
120
1 100
c[nm]
1
120
1 100
As far as I can tell MATLAB does not provide B-spline based image interpolation even though it is considered an excellent methodThere are user-contributed tools available
Applications
We now describe a couple of applications of image interpolation
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 40: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/40.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN40
Image registration
There are many applications where we would like to transform the spatial coordinates of an image f(x y) so that (after beingtransformed) it better ldquomatchesrdquo another image g(x y) Specifically we want to find a spatial coordinate transformation (TX TY)such that
g(x y) = f(TX(x y) TY(x y))
Letting x = (x y) in 2D or x = (x y z) in 3D a more concise way of writing this is
g(x) = f(T (x))
where x isin Rd and T Rd rarr Rd denotes the spatial transformation
When T is a transformation that does not preserve distances between points then it is called a nonrigid transformation and theprocess is also called morphing or warping an image
Example warping many brain images into a standard coordinate system to facilitate automatic image analysis eg [13 14]
For surveys on the topic of image registration see [15ndash19] or refer to books on the topic [20] [21]
The main research topics in the field of image registration arebull choice of spatial transformationbull choice of similarity metricbull same modalitybull different modality
bull optimization methodbull constraints such as invertibility [22]bull handling multiple (more than two) images
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 41: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/41.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN41
Spatial transformations (aka warping)
Depending on the application one might choose from among the many possible families of spatial transformationsbull Shifttranslate only
g(x y) = f(xminus x0 y minus y0)
bull Rotatetranslate (rigid)g(x y) = fθ(xminus x0 y minus y0)
bull Affineg(x y) = f(a11x+ a12y minus x0 a21x+ a22y minus y0)
bull Tensor-product B-splines
TX(x y) = x+Kminus1sumk=0
Lminus1suml=0
αX[k l] b(xwX minus k) b(ywY minus l)
where the knot spacing wX is a tunable parameter and αX[k l] are called deformation parameters The y component TY isdefined similarlybull thin-plate splines (TPS) (describes smooth warpings using control points) [23] which in 2D has the form
TX(x y) = x0 + a11x+ a12y︸ ︷︷ ︸affine portion
+Ksum
k=1
αk b(radic
(xminus xk)2 + (y minus yk)2) b(r) = r2 log r
where the coefficients αk are chosen such that TX(xk yk) = x0 + a11xk + a12yk by solving a system of equationsbull radial basis functions [24] [25] [26] have the same form as TPS with other choices fr b(r)bull Other general nonrigid transformations see [14] for a survey
See MATLABrsquos cp2tform function for examples of defining spatial transforms from control points
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 42: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/42.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN42
Example The following figure shows an example of a nonrigid transformation
minus1 minus05 0 05 1minus1
minus05
0
05
1Grid pattern
minus1 minus05 0 05 1minus1
minus05
0
05
1Warped pattern
A professor A warped professor
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 43: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/43.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN43
Interpolation
The descriptions above are in terms of hypothetical continuous-space images g(x y) and f(x y) In practice we have two digitalimages g[nm] and f [nm] that we want to align To apply any of the preceding spatial transformations we first interpolate oneof the images say f [nm] to define a continuous-space image eg using B-spline based interpolation
f(x y) =Nminus1sumn=0
Mminus1summ=0
c[nm] b(x∆X minus n) b(y∆Y minusm)
We then define a warped version of this image according to some candidate registration parameters α
fα(x y) = f(TX(x yα) TY(x yα))
Then we sample this defined warped image to compute a warped digital image
fα[nm] = fα(n∆Xm∆Y)
Similarity metrics
To perform image registration we must measure the similarity (or dissimilarity) of g[nm] and fα[nm] to find the ldquobestrdquo trans-formation There are many similarity metrics in the literature and this remains an active research areabull sum of squared differences
Nminus1sumn=0
Mminus1summ=0
∣∣∣g[nm]minus fα[nm]∣∣∣
bull normalized correlation coefficient sumNminus1n=0
sumMminus1m=0 g[nm] fα[nm]sumNminus1
n=0
sumMminus1m=0 |g[nm]|2
bull Mutual information and its variantsbull
Optimization algorithms
Finally we need an algorithm that optimizes the free parameters α of the allowable transformations to find the best match (highestsimilarity) This too is an active research area It is a challenging problem because (dis)similarity metrics are non-convex functionsof α and algorithms often converge to local optima
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 44: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/44.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN44
Image rotation is a separable operation
Image rotation is used in applications such as digital image stabilization for hand-held camerasThe obvious approach is to use 2D interpolation (IN2) as used in MATLABrsquos imrotate commandTo save computation one can use a 2-pass factorization of a coordinate rotation matrix [27][
cosϕ sinϕminus sinϕ cosϕ
]=
[1 0
minus tanϕ 1 cosϕ
] [cosϕ sinϕ0 1
]
This factorization allows image rotation to be implemented using the following two-pass approach
f(x y)rarr Sv rarr s(x y) = f(xminusx tanϕ+ y cosϕ)
s(x y)rarr Sh rarr g(x y) = s(x cosϕ+ y sinϕ y)
= f(x cosϕ+ y sinϕminus (x cosϕ+ y sinϕ) tanϕ+ y cosϕ)
= f(x cosϕ+ y sinϕminusx sinϕ+ y cosϕ)
Examplef(xy)
x
y
minus2 0 2
minus2
0
2
s(xy)
minus2 0 2
minus2
0
2
g(xy)
minus2 0 2
minus2
0
2
There is also a 3-pass factorization where each term has unity determinant[cosϕ sinϕminus sinϕ cosϕ
]=
[1 tan(ϕ2)0 1
] [1 0
minus sinϕ 1
] [1 tan(ϕ2)0 1
]
These factorizations allow one to implement image rotation using a set of 1D interpolation operations saving computation
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 45: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/45.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN45
Image zooming (up-sampling) using the FFT
Given a N timesM digital image f [nm] for n = 0 N minus 1 m = 0 M minus 1 often we need to ldquoupsamplerdquo the image to say alarger size K times L for display
One way to do this is to use the interpolation methods described previously ie represent
fa(x y) =Nminus1sumn=0
Mminus1summ=0
f [nm]h(xminus n y minusm)
and then evaluate fa(x y) using a fine grid ie
fa
(nN
Km
M
L
) n = 0 K minus 1 m = 0 Lminus 1
An alternative approach is to use the FFT to compute the (NM)-point DFT of the image f [nm] then appropriately zero pad theDFT coefficients into a K times L array and then compute the inverse (KL)-point DFT (using an inverse FFT) to get an upsampledimage g[nm] for n = 0 K minus 1 m = 0 Lminus 1 One must ldquozero padrdquo quite carefully for this to work properly
MATLABrsquos interpft routine does this for 1D vectors or for each column of a 2D array Unfortunately interpft (in MATLABand octave as of 2013-02-17) contains a subtle error that yields a nonzero imaginary part when interpolating a real signal
For a 1D signal f [n] the interpft routine first uses the FFT to compute F [k] for k = 0 N minus 1 and then defines a newvector G[k] of length K ge N as follows
G[k] ≜
F [k] k = 0 N2minus 10 k = N2 K minusN2minus 1F [k minusK +N ] k = K minusN2 K minus 1
(IN24)
where we assume N is even for simplicity of exposition Then interpft uses the inverse FFT of G[k] and multiplies by KNto compute g[n] corresponding to fa(nNK) for n = 0 K minus 1
Unfortunately the definition (IN24) does not satisfy the DFT Hermitian symmetry property corresponding to real signals leadingto a nonzero imaginary part as illustrated in the figures
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 46: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/46.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN46
-2
6
18
0 1 2 3 4
Real
Re f(x)Re f(n)Re g[n]Re h[n]
-10
-5
0
5
10
0 1 2 3 4
Imag
x
Im f(x)Im f(n)Im g[n]Im h[n]
-12
0
20
28
0 1 2 3
F[k]
-12
0
20
28
0 1 2 3 4 5 6 7 8 9
G[k]
-12
0
14
20
0 1 2 3 4 5 6 7 8 9
H[k]
k
To use FFTs properly for DFT-based upsampling we define (again assuming N is even for simplicity)
H[k] =
F [k] k = 0 N2minus 112 F [N2] k = N20 k = N2 K minusN2minus 112 F [N2] k = K minusN2F [k minusK +N ] k = K minusN2 + 1 K minus 1
(IN25)
Then we use the inverse FFT and multiply by KN to compute h[n] corresponding to fa(nNK) for n = 0 K minus 1
Exercise Analyze the relationship between h[n] and f [n] mathematically
It is essential to insert the zeros in the ldquomiddlerdquo of the DFT coefficients (ie around π in the DSFT) where the high spatialfrequencies are located The following example illustrates this importance
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 47: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/47.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN47
Example
In the following figure we formed Y [k] from X[k] by adding zeros at the end of the N = 16 DFT samples and we formed Z[k]from X[k] by inserting zeros in the middle of the DFT samples The latter works correctly (if we treat X[N2] properly)
0
6
0 15
n
x[n]
0
40
0 15
k
|X[k]|
0
40
0 16 47
k
|Y[k]|
0
6
0 16 47
n
|y[n]|
0
40
0 16 47
k
|Z[k]|
0
6
0 16 47
n
z[n]
Clearly g = ifft(fft(f) K) does not work
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 48: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/48.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN48
Using the separability of 2D Fourier transforms we can achieve 2D upsampling via the FFT in MATLAB using
g = ir_interpft( ir_interpft(f K)rsquo L )rsquo
where ir_interpft is an improved version of interpft based on (IN25)
Example Here is 2D image zooming (by a factor of 8) using 2D FFT
What caused the dark smudge along the lower right edge
Harmon amp Jules 1973
Zero-order interpolation
Lincoln illusion1
19
1 15
FFT-based up-sampling
Lincoln delusion1
152
1 120
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 49: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/49.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN49
Motion estimation for video temporal interpolation
Reference [28]
The preceding ideas generalize readily to 3D interpolation problems such as image registration of 3D medical images g(x y z)Because that generalization is straightforward we instead move beyond 2D by turning to motion estimation and video interpolationA video sequence
f(x y t)
is a function of 3 arguments so in principle ldquo2D+timerdquo could be viewed as a kind of ldquo3D problemrdquo However the time dimensionoften has different practical considerations than the spatial dimensions Different applications need different models and methodsbut many applications involving motion are real time eg in video processing so the temporal part of the processing should becausal or nearly so Thus in practice we often use techniques for video that differ from 3D spatial interpolation problems
A typical problem is the following Given two image frames f(x y tminus) and f(x y t0) find (or approximate) f(x y t) fortminus lt t lt t0 (Of course in practice we will only have a finite number of spatial samples but because time sampling is the largerissue here we focus on that aspect)
Example Converting 24 framessec movie images to 60 framesec NTSC TV images Conventional 32 approach shows 1 cineframe for 3 TV frames and the next for 2 TV frames etc This is a form of nearest neighbor interpolation in time This simpleinterpolation method can cause unnatural motion (temporal ldquoblockingrdquo)
Example Smooth cross-fade between two images [29]
Example ldquoSteadyCamrdquo features of camcorders
In principle we could develop space-time interpolation theory for moving objects f(x y t) that are bandlimited both in terms ofspatial frequencies and in terms of time variations eg [30 31] Such a theory may have specialized applications such as cardiacimaging but in general such ldquobandlimitedrdquo space-time theory may have limited practical use for video processing becausebull moving objects may not be bandlimitedbull even if the objects are bandlimited the sampling rates may be inadequate (video sampling rates are governed more by human
perception than by object properties as evidenced by backwards turning wagon wheels in western movies)bull even if the sampling rates were adequate the theoretical interpolation methods are both IIR and noncausal hence are impractical
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 50: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/50.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN50
Example
Time-varying amplitude f1(x y t) = (15 + sin(2πt)) sinc2
( x
10y
10
) Is this signal space-time band-limited
Time-varying position f2(x y t) = sinc2
(xminus sin(2πt)
10y
10
) Is this signal space-time band-limited
f1(x y t) f2(x y t)
0 02 04 06 08 1
0
05
1
15
2
25
t [s]
f(0
0t)
amplitude sinposition sin
0 5 10 15 20minus200
minus150
minus100
minus50
0
ν [1s]
|Spe
ctru
m| [
dB]
amplitude sinposition sin
![Page 51: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/51.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN51
Example Using linear interpolation (in time) between frames can cause artifacts too for moving objects as shown below
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
32 interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Linear interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
We need more sophisticated but still simple (and probably suboptimal) interpolation methods that take into account interframemotion of objects within the scene
c⃝ J Fessler February 18 2013 1754 (student version) IN52
Translation motion estimation
Having a model for an objectrsquos temporal variations helps in the design of a motion-compensated temporal interpolation method
The simplest model isf(x y t0) = f(xminus dX y minus dY tminus) (IN26)
wherebull f(x y t0) denotes the current framebull f(x y tminus) denotes the past framebull (dXdY) denotes the object shift that occurs between the two frames
This is a global displacement model but in practice it is applied only locally to small regions (eg 8 by 8 pixels) in the image
What about t isin (tminus t0) Because distance is the time integral of velocity we can write
dX = x(tminus)minus x(t0) =
int t0
tminus
vX(t) dt
To simplify further we assume uniform velocity between tminus and t0 so
dX = vX middot (t0 minus tminus)
Thus under the above two assumptions we have the following model that forms the basis of an interpolation method
f(x y t) = f(xminus vX middot (tminus tminus) y minus vY middot (tminus tminus) tminus) = s(α(x y t) β(x y t)) tminus le t le t0 (IN27)
wherebull s(x y) = f(x y tminus) is the ldquostarting imagerdquobull α(x y t) = xminus vX middot (tminus tminus) is the x-coordinate transformationbull β(x y t) = y minus vY middot (tminus tminus) is the y-coordinate transformation
By construction we have f(x y tminus) = f(x y tminus) But if the actual motion departs from the above simple model then in generalf(x y t0) = f(x y t0) for some values of x y regardless of how we choose vX and vY
Our next goal thus becomes choosing vX and vY or equivalently dX and dY so that
f(x y t0) asymp f(x y t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN53
Region-matching methods (based on (IN26))
We want to determine vX and vY (or equivalently dX and dY) to minimize some measure of ldquoerrorrdquo between the current framef(x y t0) and its prediction
f(x y t0) = f(xminus dX y minus dY tminus)
A natural approach is to select a region R and find the (dXdY) pair that minimizes the integrated squared difference between thetwo images over that region
mindXdY
intint(xy)isinR
|f(x y t0)minus f(xminus dX y minus dY tminus)|2 dxdy (IN28)
Metrics other than squared error are also useful
Unfortunately finding the best dXdY pair is a very nonlinear minimization problem fraught with difficulties such as local minimaThere is no closed-form analytical solution so one must apply iterative methods
One approach to help reduce computation and avoid local minima is to use a multiresolution search strategy one begins searchingon a coarse grid of dX and dY values and then refines the search over successively finer grids With such strategies (and appropriateinterpolation) one can achieve subpixel accuracy
To achieve subpixel matching (which is often desirable because objects do not usually move by integer amounts between frames)one must apply some type of spatial interpolation to evaluate the above integral (which will of course be replaced by a summationin practice) A standard choice is bilinear interpolation
One can also apply a coordinate descent algorithm where one alternates between minimizing over dX with dY fixed to itsmost recent estimate and then updating dY with dX fixed to its most recent estimate
Region-matching or block-matching methods are used routinely for video coding
c⃝ J Fessler February 18 2013 1754 (student version) IN54
ExampleThe left figure illustrates a coarse-to-fine or multigrid search for the best displacement parameters dXdY
The right figure illustrates the kind of estimated displacement vectors that result from such an approach
-7
0
7
-7 0 7
dY
dX
Multigrid search (coarse to fine)
httpwwwyoutubecomwatchv=9-v8O1bv6A0
c⃝ J Fessler February 18 2013 1754 (student version) IN55
Example To illustrate how one can perform block matching (without interpolation) with minimal MATLAB coding it is helpful tofirst illustrate use of the colfilt function
demo_colfiltm illustrate colfilt()f = single(imread(rsquocameramantifrsquo)rsquo)fun1 = (x) mean(x) fun2 = (x) max(x)g1 = colfilt(f [7 7] rsquoslidingrsquo fun1)g2 = colfilt(f [7 7] rsquoslidingrsquo fun2)im plc 1 3 im(1 f rsquofrsquo) im(2 g1 rsquomeanrsquo) im(3 g2 rsquomaxrsquo)
f
1 256
1
256
mean
1 256
1
256
max
1 256
1
256
Here is an illustration of how to use colfilt to find the ldquobest matchrdquo for a given region (a face in this case)
fig_region_match1mf = single(imread(rsquocameramantifrsquo)rsquo)h = f(120+[-1515]61+[-1515])fun = (x) sum(abs(x - repmat(h() 1 ncol(x)))ˆ2)fit = colfilt(f size(h) rsquoslidingrsquo fun)im plc 1 3 im(1 f rsquofrsquo) im(2 h rsquohrsquo) axis([0 1 0 1]256-128) im(3 fit rsquofitrsquo)
f
1 256
1
256
h
1 31
131
fit
1 256
1
256
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 52: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/52.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN52
Translation motion estimation
Having a model for an objectrsquos temporal variations helps in the design of a motion-compensated temporal interpolation method
The simplest model isf(x y t0) = f(xminus dX y minus dY tminus) (IN26)
wherebull f(x y t0) denotes the current framebull f(x y tminus) denotes the past framebull (dXdY) denotes the object shift that occurs between the two frames
This is a global displacement model but in practice it is applied only locally to small regions (eg 8 by 8 pixels) in the image
What about t isin (tminus t0) Because distance is the time integral of velocity we can write
dX = x(tminus)minus x(t0) =
int t0
tminus
vX(t) dt
To simplify further we assume uniform velocity between tminus and t0 so
dX = vX middot (t0 minus tminus)
Thus under the above two assumptions we have the following model that forms the basis of an interpolation method
f(x y t) = f(xminus vX middot (tminus tminus) y minus vY middot (tminus tminus) tminus) = s(α(x y t) β(x y t)) tminus le t le t0 (IN27)
wherebull s(x y) = f(x y tminus) is the ldquostarting imagerdquobull α(x y t) = xminus vX middot (tminus tminus) is the x-coordinate transformationbull β(x y t) = y minus vY middot (tminus tminus) is the y-coordinate transformation
By construction we have f(x y tminus) = f(x y tminus) But if the actual motion departs from the above simple model then in generalf(x y t0) = f(x y t0) for some values of x y regardless of how we choose vX and vY
Our next goal thus becomes choosing vX and vY or equivalently dX and dY so that
f(x y t0) asymp f(x y t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN53
Region-matching methods (based on (IN26))
We want to determine vX and vY (or equivalently dX and dY) to minimize some measure of ldquoerrorrdquo between the current framef(x y t0) and its prediction
f(x y t0) = f(xminus dX y minus dY tminus)
A natural approach is to select a region R and find the (dXdY) pair that minimizes the integrated squared difference between thetwo images over that region
mindXdY
intint(xy)isinR
|f(x y t0)minus f(xminus dX y minus dY tminus)|2 dxdy (IN28)
Metrics other than squared error are also useful
Unfortunately finding the best dXdY pair is a very nonlinear minimization problem fraught with difficulties such as local minimaThere is no closed-form analytical solution so one must apply iterative methods
One approach to help reduce computation and avoid local minima is to use a multiresolution search strategy one begins searchingon a coarse grid of dX and dY values and then refines the search over successively finer grids With such strategies (and appropriateinterpolation) one can achieve subpixel accuracy
To achieve subpixel matching (which is often desirable because objects do not usually move by integer amounts between frames)one must apply some type of spatial interpolation to evaluate the above integral (which will of course be replaced by a summationin practice) A standard choice is bilinear interpolation
One can also apply a coordinate descent algorithm where one alternates between minimizing over dX with dY fixed to itsmost recent estimate and then updating dY with dX fixed to its most recent estimate
Region-matching or block-matching methods are used routinely for video coding
c⃝ J Fessler February 18 2013 1754 (student version) IN54
ExampleThe left figure illustrates a coarse-to-fine or multigrid search for the best displacement parameters dXdY
The right figure illustrates the kind of estimated displacement vectors that result from such an approach
-7
0
7
-7 0 7
dY
dX
Multigrid search (coarse to fine)
httpwwwyoutubecomwatchv=9-v8O1bv6A0
c⃝ J Fessler February 18 2013 1754 (student version) IN55
Example To illustrate how one can perform block matching (without interpolation) with minimal MATLAB coding it is helpful tofirst illustrate use of the colfilt function
demo_colfiltm illustrate colfilt()f = single(imread(rsquocameramantifrsquo)rsquo)fun1 = (x) mean(x) fun2 = (x) max(x)g1 = colfilt(f [7 7] rsquoslidingrsquo fun1)g2 = colfilt(f [7 7] rsquoslidingrsquo fun2)im plc 1 3 im(1 f rsquofrsquo) im(2 g1 rsquomeanrsquo) im(3 g2 rsquomaxrsquo)
f
1 256
1
256
mean
1 256
1
256
max
1 256
1
256
Here is an illustration of how to use colfilt to find the ldquobest matchrdquo for a given region (a face in this case)
fig_region_match1mf = single(imread(rsquocameramantifrsquo)rsquo)h = f(120+[-1515]61+[-1515])fun = (x) sum(abs(x - repmat(h() 1 ncol(x)))ˆ2)fit = colfilt(f size(h) rsquoslidingrsquo fun)im plc 1 3 im(1 f rsquofrsquo) im(2 h rsquohrsquo) axis([0 1 0 1]256-128) im(3 fit rsquofitrsquo)
f
1 256
1
256
h
1 31
131
fit
1 256
1
256
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 53: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/53.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN53
Region-matching methods (based on (IN26))
We want to determine vX and vY (or equivalently dX and dY) to minimize some measure of ldquoerrorrdquo between the current framef(x y t0) and its prediction
f(x y t0) = f(xminus dX y minus dY tminus)
A natural approach is to select a region R and find the (dXdY) pair that minimizes the integrated squared difference between thetwo images over that region
mindXdY
intint(xy)isinR
|f(x y t0)minus f(xminus dX y minus dY tminus)|2 dxdy (IN28)
Metrics other than squared error are also useful
Unfortunately finding the best dXdY pair is a very nonlinear minimization problem fraught with difficulties such as local minimaThere is no closed-form analytical solution so one must apply iterative methods
One approach to help reduce computation and avoid local minima is to use a multiresolution search strategy one begins searchingon a coarse grid of dX and dY values and then refines the search over successively finer grids With such strategies (and appropriateinterpolation) one can achieve subpixel accuracy
To achieve subpixel matching (which is often desirable because objects do not usually move by integer amounts between frames)one must apply some type of spatial interpolation to evaluate the above integral (which will of course be replaced by a summationin practice) A standard choice is bilinear interpolation
One can also apply a coordinate descent algorithm where one alternates between minimizing over dX with dY fixed to itsmost recent estimate and then updating dY with dX fixed to its most recent estimate
Region-matching or block-matching methods are used routinely for video coding
c⃝ J Fessler February 18 2013 1754 (student version) IN54
ExampleThe left figure illustrates a coarse-to-fine or multigrid search for the best displacement parameters dXdY
The right figure illustrates the kind of estimated displacement vectors that result from such an approach
-7
0
7
-7 0 7
dY
dX
Multigrid search (coarse to fine)
httpwwwyoutubecomwatchv=9-v8O1bv6A0
c⃝ J Fessler February 18 2013 1754 (student version) IN55
Example To illustrate how one can perform block matching (without interpolation) with minimal MATLAB coding it is helpful tofirst illustrate use of the colfilt function
demo_colfiltm illustrate colfilt()f = single(imread(rsquocameramantifrsquo)rsquo)fun1 = (x) mean(x) fun2 = (x) max(x)g1 = colfilt(f [7 7] rsquoslidingrsquo fun1)g2 = colfilt(f [7 7] rsquoslidingrsquo fun2)im plc 1 3 im(1 f rsquofrsquo) im(2 g1 rsquomeanrsquo) im(3 g2 rsquomaxrsquo)
f
1 256
1
256
mean
1 256
1
256
max
1 256
1
256
Here is an illustration of how to use colfilt to find the ldquobest matchrdquo for a given region (a face in this case)
fig_region_match1mf = single(imread(rsquocameramantifrsquo)rsquo)h = f(120+[-1515]61+[-1515])fun = (x) sum(abs(x - repmat(h() 1 ncol(x)))ˆ2)fit = colfilt(f size(h) rsquoslidingrsquo fun)im plc 1 3 im(1 f rsquofrsquo) im(2 h rsquohrsquo) axis([0 1 0 1]256-128) im(3 fit rsquofitrsquo)
f
1 256
1
256
h
1 31
131
fit
1 256
1
256
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 54: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/54.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN54
ExampleThe left figure illustrates a coarse-to-fine or multigrid search for the best displacement parameters dXdY
The right figure illustrates the kind of estimated displacement vectors that result from such an approach
-7
0
7
-7 0 7
dY
dX
Multigrid search (coarse to fine)
httpwwwyoutubecomwatchv=9-v8O1bv6A0
c⃝ J Fessler February 18 2013 1754 (student version) IN55
Example To illustrate how one can perform block matching (without interpolation) with minimal MATLAB coding it is helpful tofirst illustrate use of the colfilt function
demo_colfiltm illustrate colfilt()f = single(imread(rsquocameramantifrsquo)rsquo)fun1 = (x) mean(x) fun2 = (x) max(x)g1 = colfilt(f [7 7] rsquoslidingrsquo fun1)g2 = colfilt(f [7 7] rsquoslidingrsquo fun2)im plc 1 3 im(1 f rsquofrsquo) im(2 g1 rsquomeanrsquo) im(3 g2 rsquomaxrsquo)
f
1 256
1
256
mean
1 256
1
256
max
1 256
1
256
Here is an illustration of how to use colfilt to find the ldquobest matchrdquo for a given region (a face in this case)
fig_region_match1mf = single(imread(rsquocameramantifrsquo)rsquo)h = f(120+[-1515]61+[-1515])fun = (x) sum(abs(x - repmat(h() 1 ncol(x)))ˆ2)fit = colfilt(f size(h) rsquoslidingrsquo fun)im plc 1 3 im(1 f rsquofrsquo) im(2 h rsquohrsquo) axis([0 1 0 1]256-128) im(3 fit rsquofitrsquo)
f
1 256
1
256
h
1 31
131
fit
1 256
1
256
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 55: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/55.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN55
Example To illustrate how one can perform block matching (without interpolation) with minimal MATLAB coding it is helpful tofirst illustrate use of the colfilt function
demo_colfiltm illustrate colfilt()f = single(imread(rsquocameramantifrsquo)rsquo)fun1 = (x) mean(x) fun2 = (x) max(x)g1 = colfilt(f [7 7] rsquoslidingrsquo fun1)g2 = colfilt(f [7 7] rsquoslidingrsquo fun2)im plc 1 3 im(1 f rsquofrsquo) im(2 g1 rsquomeanrsquo) im(3 g2 rsquomaxrsquo)
f
1 256
1
256
mean
1 256
1
256
max
1 256
1
256
Here is an illustration of how to use colfilt to find the ldquobest matchrdquo for a given region (a face in this case)
fig_region_match1mf = single(imread(rsquocameramantifrsquo)rsquo)h = f(120+[-1515]61+[-1515])fun = (x) sum(abs(x - repmat(h() 1 ncol(x)))ˆ2)fit = colfilt(f size(h) rsquoslidingrsquo fun)im plc 1 3 im(1 f rsquofrsquo) im(2 h rsquohrsquo) axis([0 1 0 1]256-128) im(3 fit rsquofitrsquo)
f
1 256
1
256
h
1 31
131
fit
1 256
1
256
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 56: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/56.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN56
Space-time constraint methods (based on (IN29))
Using the relationship (IN27) ie f(x y t) = s(α(x y t) β(x y t)) by the chain rule the partial derivatives are
part
partxf(x y t) =
parts
partα
partα
partx+
parts
partβ
partβ
partx=
parts
partα
part
partyf(x y t) =
parts
partα
partα
party+
parts
partβ
partβ
party=
parts
partβ
part
parttf(x y t) =
parts
partα
partα
partt+
parts
partβ
partβ
partt= minusvX
parts
partαminus vY
parts
partβ
When combined these equalities lead to the following space-time constraint equation
vX
part f(x y t)
partx+ vY
part f(x y t)
party+
part f(x y t)
partt= 0 (IN29)
This equation disregards other object motion such as rotations zooms or multiple objects with different velocities although it canbe generalized
Instead of using (IN26) and (IN28) we can also estimate the motion using (IN29)
From (IN29) we can find vX and vY (from which we can determine dX and dY) as follows
How many equations and how many unknowns per pixel Physically we cannot determine the component of motion perpendicular to the gradient (ie parallel to the edge) without additionalassumptions
This problem is related to the barber pole illusionhttpenwikipediaorgwikiBarberpole_illusion
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 57: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/57.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN57
One solution is to use a spatial coherence constraint introduced by Lucas and Kanade [32]bull Choose a space-time region over which (vXvY) is assumed to be constantbull Let (xi yi ti)Ni=1 denote a set of samples within that regionbull Use these samples to form estimates of the three types of partial derivatives of f(x y t) using any of the derivative calculation
methods described previouslyDenote these derivative estimates as follows
gX
i ≜ part f(x y t)
partx
∣∣∣∣(xiyiti)
gY
i ≜ part f(x y t)
party
∣∣∣∣(xiyiti)
gT
i ≜ part f(x y t)
partt
∣∣∣∣(xiyiti)
Then from (IN29)vXg
X
i + vYgY
i + gT
i asymp 0
We would like to find the velocity estimates vX and vY that ldquobest agreerdquo with the constraint equation
A natural approach is to use a least-squares criterion
(vX vY) = argminvXvY
Nsumi=1
(vXgX
i + vYgY
i + gT
i )2
This is a linear least-squares problem which can also be written
minθ∥Aθ minus b∥2
where θ =
[vX
vY
] A is a N times 2 matrix and b is a vector of length N with entries
A =
gX1 gY
1
gX
N gY
N
b =
minusgT1
minusgT
N
The solution to such problems is given by the normal equations
(AprimeA)θ = Aprimeb
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 58: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/58.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN58
If A has full column rank ie if the two columns are linearly independent then the LS solution is
θ = (AprimeA)minus1Aprimeb
However it is possible for the columns of A to be linearly dependent When(Over a spatially uniform region all spatial derivatives are zero)In this case one usually adopts a minimum-norm LS solution or used the following strategy based on the eigenvalues of AprimeAbull If λ1 and λ2 fall below a threshold use θ = 0 (No motion or uniform intensity region)bull If λ1 ≫ λ2 then the motion appears to be primarily along one direction (1D)
Let the Gram matrix G = AprimeA have the eigenvector decomposition
G = [u1 u2]
[λ1 00 λ2
][u1 u2]
prime
where u1 and u2 denote orthonormal eigenvectors (because G is symmetric nonnegative definite) Then the LS solution gives
θ = Gminus1Aprimeb = [u1 u2]
[ 1λ1
0
0 1λ2
][u1 u2]
primeAprimeb
The following figure illustrates the behavior of the eigenvalues for different types of image patches The Gram matrix can beinterpreted as the empirical covariance of the gradient vectors (gX
i gYi ) In an edge region most of the gradient vectors will
point along a single direction
(Figure by Prof Savarese)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 59: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/59.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN59
The reciprocal of λ2 will be very unstable in the presence of noise if λ1 ≫ λ2 A solution is to zero out the component alongu2 which is a kind of pseudo inverse
θ = [u1 u2]
[1λ1
0
0 0
][u1 u2]
primeAprimeb =uprime1A
primeb
λ1u1
Of course to implement such methods one must estimate the three partial derivatives Local polynomial fitting (in 3D) is a reason-able choice
To reduce computation one can use variable sized grids and select the region accordingly
Region selection
In both the region matching method and the space-time constraint method reducing region size will reduce computation
However if the region is too small then the object may move out of the region on the next frame One method used to minimizethe size of regions while reducing the influence of large displacements is to prediction In this approach the region is notconstant between frames Based on estimates of the displacement in the previous frame the center of the region is translated Thisprocedure allows tracking of displacements comparable to or greater than the extent of the region Although reducing computationsthis method is susceptible to ldquorun awayrdquo if the original estimates of the displacement are inaccurate
Advanced topics
More general optical flow approaches are available [33] including from photon-limited measurements [34] See [35] for a niceunified treatment A complication that arises in practice is discontinuities [36ndash39] See also [40] and [41]
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 60: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/60.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN60
Motion compensated temporal interpolation
By matching the rectangle and triangle in the previous frame to the current frame we can estimate their displacementsThen we interpolate intermediate frames using the model (IN27) This nonlinear interpolation method is quite effective
0 1 2 3 4 5 6 7
minus04
minus02
0
02
04
06Displacements
0 1 2 3 4 5 6 70
2
4
6
8New positions
xold
x new
0 1 2 3 4 5 6 70
05
1
f(x
tminus)
Motionminuscompensated interpolation
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
0 1 2 3 4 5 6 70
05
1
x
f(x
t0)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 61: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/61.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN61
Example Here is another example illustrating that if we can find corresponding features in two frames then we can estimate thedisplacements and interpolate between those frames using the estimated motion
f(x t)
f(x t0)
f(x tminus)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 62: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/62.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN62
Application of motion estimation methods to spatial interpolation
Consider the following spatial interpolation problemGiven f(x y0) and f(x y1) find f(x y) for y0 lt y lt y1
This problem is essentially identical to the problem considered by (IN26) so similar approaches apply
In fact in the 1D interpolation figures shown previously the running dimension could be either time or y and the results would beidentical
It is sometimes claimed that such an approach can outperform sinc interpolationHow is this possible Because this type of approach is nonlinear and shift varying
We can use 1D region matching or a 1D version of the space-time constraint equation to estimate the correspondences betweenf(x y0) and f(x y1)
This approach is explored in a HW problem and compared to conventional linear interpolation methods
Summary
bull There are numerous applications of image interpolation and numerous available methodsbull Classical methods use interpolation kernels Modern methods use shift invariant spacesbull Methods based on B-splines offer a particularly good tradeoff between interpolation quality and computationbull FFT can be used for zoomingbull Temporal interpolation of video sequences benefits from motion estimation
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 63: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/63.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN63
Bibliography[1] D Shepard A two-dimensional interpolation function for irregularly-spaced data In ACM Conf pages 517ndash24 1968 IN2[2] O V Morozov M Unser and P Hunziker Reconstruction of large irregularly sampled multidimensional images A tensor-
based approach IEEE Trans Med Imag 30(2)366ndash74 February 2011[3] V Chatzis and I Pitas Interpolation of 3-D binary images based on morphological skeletonization IEEE Trans Med Imag
19(7)699ndash710 July 2000 IN2[4] E Meijering A chronology of interpolation from ancient astronomy to modern signal and image processing Proc IEEE
90(3)319ndash42 March 2002 IN2[5] P Thevenaz T Blu and M Unser Interpolation revisited IEEE Trans Med Imag 19(7)739ndash58 July 2000 IN13 IN14[6] N A Dodgson Quadratic interpolation for image resampling IEEE Trans Im Proc 6(9)1322ndash6 September 1997 IN16[7] M Unser Splines A perfect fit for signal and image processing IEEE Sig Proc Mag 16(6)22ndash38 November 1999 IN18
IN31 IN33[8] M Unser A Aldroubi and M Eden B-spline signal processing Part Imdashtheory IEEE Trans Sig Proc 41(2)821ndash33
February 1993 IN18[9] E Maeland On the comparison of interpolation methods IEEE Trans Med Imag 7(3)213ndash7 September 1988 IN18
[10] R G Keys Cubic convolution interpolation for digital image processing IEEE Trans Acoust Sp Sig Proc 29(6)1153ndash60December 1981 IN18 IN19
[11] F N Fritsch and R E Carlson Monotone piecewise cubic interpolation SIAM J Numer Anal 17(2)238ndash46 April 1980IN18
[12] M Unser Sampling-50 years after Shannon Proc IEEE 88(4)569ndash87 April 2000 IN34[13] A C Evans C Beil S Marrett C J Thompson and A Hakim Anatomical-functional correlation using an adjustable MRI-
based region of interest atlas with positron emission tomography J Cerebral Blood Flow and Metabolism 8(4)513ndash301988 IN40
[14] M Holden A review of geometric transformations for nonrigid body registration IEEE Trans Med Imag 27(1)111ndash28January 2008 IN40 IN41
[15] J B A Maintz and M A Viergever A survey of medical image registration Med Im Anal 2(1)1ndash36 March 1998 IN40[16] J M Fitzpatrick D L G Hill and C R Maurer Image registration In M Sonka and J Michael Fitzpatrick editors
Handbook of Medical Imaging Volume 2 Medical Image Processing and Analysis pages 477ndash513 SPIE Bellingham 2000IN40
[17] D L G Hill P G Batchelor M Holden and D J Hawkes Medical image registration Phys Med Biol 46(3)R1ndash47March 2001 IN40
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 64: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/64.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN64
[18] J P W Pluim J B A Maintz and M A Viergever Mutual-information-based registration of medical images a surveyIEEE Trans Med Imag 22(8)986ndash1004 August 2003 IN40
[19] B Zitova and J Flusser Image registration methods a survey Im and Vision Computing 21(11)977ndash1000 October 2003[20] J Modersitzki Numerical methods for image registration Oxford 2003 IN40[21] J Modersitzki FAIR Flexible algorithms for image registration SIAM 2009 IN40[22] S Y Chun and J A Fessler A simple regularizer for B-spline nonrigid image registration that encourages local invertibility
IEEE J Sel Top Sig Proc 3(1)159ndash69 February 2009 Special Issue on Digital Image Processing Techniques for OncologyIN40
[23] F L Bookstein Principal warps thin-plate splines and the decomposition of deformations IEEE Trans Patt Anal MachInt 11(6)567ndash87 June 1989 IN41
[24] Z Wu Multivariate compactly supported positive definite radial functions Advances in Computational Mathematics 4283ndash92 1995 IN41
[25] H Wendland Piecewise polynomial positive definite and compactly supported radial functions of minimal degree Advancesin Computational Mathematics 4(1)389ndash96 1995 IN41
[26] Z Wu Generalized bochnerrsquos theorem for radial function Approximation Theory and its Applications 13(3)47ndash57 1997IN41
[27] M Unser P Thevenaz and L Yaroslavsky Convolution-based interpolation for fast high quality rotation of images IEEETrans Im Proc 4(10)1371ndash81 October 1995 IN44
[28] M A Tekalp Digital video processing Prentice-Hall New York 1996 IN49[29] I Kemelmacher-Shlizerman E Shechtman R Garg and S M Seitz Exploring photobios ACM Trans on Graphics
30(4)61 July 2011 Proc of ACM SIGGRAPH IN49[30] N P Willis and Y Bresler Optimal scan for time-varying tomography I Theoretical analysis and fundamental limitations
IEEE Trans Im Proc 4(5)642ndash53 May 1995 IN49[31] N P Willis and Y Bresler Lattice-theoretic analysis of time-sequential sampling of spatiotemporal signals II Large space-
bandwidth product asymptotics IEEE Trans Info Theory 43(1)208ndash20 January 1997 IN49[32] B Lucas and T Kanade An iterative image registration technique with an application to stereo vision In Proc Seventh
International Joint Conference on Artificial Intelligence pages 674ndash9 1981 IN57[33] B Horn and B G Schunck Determining optical flow Artif Intell 18(1-3)185ndash203 August 1981 IN59[34] C L Chan and A K Katsaggelos Iterative maximum likelihood displacement field estimation in quantum-limited image
sequences IEEE Trans Im Proc 4(6)743ndash51 June 1995 IN59[35] A Bruhn J Weickert and C Schnrr LucasKanade meets HornSchunck combining local and global optic flow methods
Intl J Comp Vision 61(3)211ndash31 February 2005 IN59
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-
![Page 65: Fessler_2D Interpolation [2013]](https://reader034.vdocuments.site/reader034/viewer/2022052603/55cf9ae8550346d033a3f554/html5/thumbnails/65.jpg)
c⃝ J Fessler February 18 2013 1754 (student version) IN65
[36] L Blanc-Feraud M Barlaud and T Gaidon Motion estimation involving discontinuities in a multiresolution scheme OpticalEngineering 32(7)1475ndash82 July 1993 IN59
[37] F Heitz and P Bouthemy Multimodal estimation of discontinuous optical flow using Markov random fields IEEE TransPatt Anal Mach Int 15(12)1217ndash32 December 1993 IN59
[38] P Nesi Variational approach to optical flow estimation managing discontinuities Im and Vision Computing 11(7)419ndash39September 1993 IN59
[39] R Deriche P Kornprobst and G Aubert Optical-flow estimation while preserving its discontinuities A variational approachIn Proc 2nd Asian Conference on Computer Vision volume 2 pages 71ndash80 1995
[40] M Dawood F Buther X Jiang and K P Schafers Respiratory motion correction in 3-D PET data with advanced opticalflow algorithms IEEE Trans Med Imag 27(8)1164ndash75 August 2008 IN59
[41] D Cremers and S Soatto Variational space-time motion segmentation In Proc Intl Conf Comp Vision volume 2 pages886ndash93 2003 IN59
- 2D Interpolation
-
- Introduction
-
- Problem statement and background
- Ideal sinc interpolation
-
- Polynomial interpolation
-
- Zero-order or nearest neighbor (NN) interpolation
- 1D linear interpolation
- 2D linear or bilinear interpolation
- Interpolator properties
- Lagrange interpolation
- Quadratic interpolation
- Cubic interpolation
- Non-polynomial interpolation Lanczos kernel
-
- Interpolation in shift-invariant subspaces
-
- Basis kernels supported on [-22]
- B-spline interpolation
- 2D interpolation in shift-invariant spaces
-
- Applications
-
- Image registration
- Image rotation is a separable operation
- Image zooming (up-sampling) using the FFT
-
- Motion estimation for video temporal interpolation
-
- Translation motion estimation
- Region-matching methods
- Space-time constraint methods
- Region selection
- Advanced topics
- Motion compensated temporal interpolation
- Application of motion estimation methods to spatial interpolation
-
- Summary
-