fourier series, convolution, and filters - university of st....

Today’s Schedule Fourier Series Convolution Filters Convolution as a Matrix Product In the Classroom

Fourier Series, Convolution, and Filters

Patrick J. Van Fleet

Center for Applied MathematicsUniversity of St. Thomas

St. Paul, MN USA

PREP - Wavelet Workshop, 2006

Wednesday, 7 June, 2006 Lecture 3

OutlineToday’s ScheduleFourier Series

Classical ResultsStudent DifficultiesFinite Length Fourier Series

ConvolutionConvolution DefinedThe Convolution Theorem

FiltersTypes of FiltersLowpass and Highpass Filters

Convolution as a Matrix ProductIn the Classroom

Teaching IdeasComputer UsageStudent Difficulties

Today’s Schedule

9:00-10:15 Lecture One: Why Wavelets?10:15-10:30 Coffee Break (OSS 235)10:30-11:45 Lecture Two: Digital Images, Measures, and

Huffman Codes12:00-1:00 Lunch (Cafeteria)1:30-2:45 ⇒Lecture Three: Fourier Series, Convolution and

Filters2:45-3:00 Coffee Break (OSS 235)3:00-4:15 Lecture Four: 1D and 2D Haar Transforms5:30-6:30 Dinner (Cafeteria)

Classical Results

Fourier SeriesClassical Results

I Let’s start by recalling Euler’s formula:

eiω = cos ω + i sin ω

I It can be shown that the set of functions ek (ω) = eikω,k ∈ Z, satisfy ∫ π

−πeikωeijω dω =

{2π j = k

0 j 6= k

Classical Results

{2π j = k

0 j 6= k

Classical Results

{2π j = k

0 j 6= k

Classical Results

I The family {eikω}k∈Z forms a basis for all suitably regular2π-periodic functions.

I A Fourier Series for 2π-periodic function f (ω) is

f (ω) =∑

ckeikω

whereck =

∫ π

−πf (ω)e−ikω dω

are called the Fourier coefficients.

Classical Results

f (ω) =∑

ckeikω

whereck =

∫ π

Classical Results

f (ω) =∑

ckeikω

whereck =

∫ π

Classical Results

If we take f (ω) = ω on the interval [−π, π] and then2π-periodically extend it, we can use integration by parts towrite

f (ω) = i∑k 6=0

(−1)k

keikω = −2

∞∑k=1

(−1)k

ksin(kω)

Classical Results

Here are some partial Fourier series:

Original 0 Terms 3 Terms

10 Terms 20 Terms 50 Terms

Classical Results

I Calculate some Fourier series by hand.I Discover and prove some rules for Fourier series:I Translation Rule: If

f (ω) =∑

ckeikω

and g(ω) = f (ω − a), then the Fourier coefficients for g(ω)are e−ikack .

Classical Results

f (ω) =∑

ckeikω

Classical Results

f (ω) =∑

ckeikω

Classical Results

f (ω) =∑

ckeikω

Student Difficulties

Fourier SeriesStudent Difficulties

I Students, at least at this level, never seem to grasp theideas behind why one would want to compute a Fourierseries.

I You can talk about them in terms of solving differentialequations . . .

I But it’s kind of like Taylor’s series to them: Why take aperfectly good function and make an infinite series out ofit?

I This class gives a wonderful arena for demonstrating theusefulness of Fourier series.

I There are three important uses for Fourier series in thiscourse. Students need:

I to build and manipulate finite length Fourier series and whatthey say about filters. (i.e. what do engineers do with them)

I the ability to extract the coefficients from a Fourier series toobtain a filter.

I to know how to manipulate one Fourier series (say byconjugation, multiplication by a complex exponential,translation) to write down another.

Finite Length Fourier Series

Fourier SeriesFinite Length Fourier Series

I In a typical math class (like the start of this one!), we give astudent a 2π-periodic function f (ω), they integrate by parts,simplify, and obtain the Fourier coefficients for the Fourierseries of f (ω).

I The engineers do it just the opposite way: They know thatthe coefficients are what’s used to process signals orimages, so they will create a (usually finite) list ofcoefficients (by various means), plug them into a Fourierseries, and analyze the result.

I Let’s look at an example:

I Suppose we have number ck , k ∈ Z, with c0 = c2 = 1/4,c1 = 1/2, and all other ck = 0. Find the Fourier series andplot its modulus.

I We have

C(ω) =14

eiω +14

= eiω(14

e−iω +12

= eiω(12

+12· eiω + e−iω

= eiω 12(1 + cos ω)

I We have

C(ω) =14

eiω +14

= eiω(14

e−iω +12

= eiω(12

= eiω 12(1 + cos ω)

I We have

C(ω) =14

eiω +14

= eiω(14

e−iω +12

= eiω(12

= eiω 12(1 + cos ω)

|C(ω)| = 12(1 + cos ω) = cos2(ω/2) ≥ 0

|C(ω)|

I The modulus of C(ω) tells the engineer much about thesequence {ck}.

I As we will see in later lectures, these ck ’s are used in amatrix to transform signals or images.

I In the case of this particular sequence, the modulus ismaximized at ω = 0 (lowest oscillation in C(ω)) andminimized at ω = π (highest oscillation in C(ω)). So whenthis sequence is used to process data, it will tend to leavelow oscillations in data largely unchanged and dampenhigh oscillations in data.

I More on this in the filters lecture . . .

I It is also very important that students learn to “peel” theFourier coefficients off a 2π-periodic function withouthaving to integrate. For example, consider

I What are the Fourier coefficients of

H(ω) = cos2(ω/2)(

1 + sin2(ω/2)(a0 + a1 eiω))

H(ω) = cos2(ω/2)(

1 + sin2(ω/2)(a0 + a1 eiω))

H(ω) = cos2(ω/2)(

1 + sin2(ω/2)(a0 + a1 eiω))

Using the identities

cos(ω) =eiω + e−iω

2and sin(ω) =

eiω − e−iω

and lots of algebra, we have

H(ω) = −a0

16e−2iω +

4 − a1

16e−iω +

8 + 2a0

+4 + 2a1

16eiω − a0

16e2iω − a1

16e3iω

so the Fourier coefficients are

(h−2, h−1, h0, h1, h2, h3) = (−a0

16,4 − a1

16,8 + 2a0

16,4 + 2a1

16,−a1

I It is also important to be able to manipulate Fourier series.For example,

I Suppose (h0, h1, h2, h3) are nonzero and let

H(ω) = h0 + h1 eiω + h2 e2iω + h3 e3iω

I What do we have to do to produce the Fourier series

H1(ω) = h0 − h1 eiω + h2 e2iω + h3 e3iω

I OrH2(ω) = h3 + h2 eiω + h1 e2iω + h0 e3iω

Answers:

H1(ω) = H(ω + π)

H2(ω) = e3iωH(ω)

Convolution Defined

ConvolutionConvolution Defined

Let h = (. . . , h−1, h0, h1, h2, . . .) and x(. . . , x−1, x0, x1, x2, . . .) betwo bi-infinite sequences. Then the convolution producty = h ∗ x is the bi-infinite sequence whose components aregiven by

yn =∑

hkxn−k

Convolution Defined

I In many applications, we think of h as a processor and x isthe input signal.

I Convolution is a standard tool used to process signals andimages.

I The convolution product is commutative.I Modulo a carry algorithm convolution is exactly how we

were taught to multiply numbers in grade school.

Convolution Defined

Let’s look at an example. Let x be any bi-infinite sequence andsuppose h is the bi-infinite sequence with h0 = h1 = 1

2 and allother hk = 0. Find

y = h ∗ x

Convolution Defined

We have

yn =∑

hkxn−k =1∑

xn−k =12(xn + xn−1)

So y is a bi-infinite sequence whose components are averagesof consecutive values of x.

Convolution Defined

I Another important convolution product will use g whereg0 = 1

2 , g1 = −12 and all other gk = 0.

I It should be easy to verify that y = g ∗ x where

yn =12(xn − xn−1)

Convolution Defined

yn =12(xn − xn−1)

Convolution Defined

yn =12(xn − xn−1)

Convolution Defined

I Students often have difficulty computing convolutionproducts.

I In this workshop, the processor h will have only finitelymany nonzero elements.

I When this is the case, it is usually better to show them thesliding strip method for convolution:

Convolution Defined

I The component y0 =∑

k hkx−k can be viewed as an innerproduct of h with a reflection of x.

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x2 x1 x0 x−1 x−2 · · ·

k hkx1−k can be viewed as an innerproduct of h with the reflection of x shifted one unit right:

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x3 x2 x1 x0 x−1 · · ·

Convolution Defined

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x2 x1 x0 x−1 x−2 · · ·

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x3 x2 x1 x0 x−1 · · ·

Convolution Defined

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x2 x1 x0 x−1 x−2 · · ·

h : · · · h−2 h−1 h0 h1 h2 · · ·· · · x3 x2 x1 x0 x−1 · · ·

The Convolution Theorem

ConvolutionThe Convolution Theorem

I Convolution is a useful tool for processing signals.I We saw two processors, h and g, that computed averages

and differences of consecutive terms.I While convolution is a useful tool, it is a bit tedious to

analyze. Fortunately there is a result that makes it reallyeasy to analyze in the transform domain:

Theorem (The Convolution Theorem)Let h and x be bi-infinite sequences with Fourier series H(ω)and X (ω), respectively. If y = h ∗ x, then

Y (ω) = H(ω)X (ω)

I The Convolution Theorem takes convolution to simplemultiplication in the Fourier domain!

I The proof is straightforward and we do it in class.I You can also use the Convolution Theorem to compute

convolutions.

Suppose h is the bi-infinite sequence with h5 = 1 and all otherhk = 0. Let x be any bi-infinite sequence. Compute

y = h ∗ x

The Fourier series for H(ω) is simply H(ω) = e5iω so

Y (ω) = H(ω)X (ω)

xkei(k+5)ω

xk−5eikω

so yn = xn−5.

Convolve h where h0 = h1 = 1 and all other hk = 0 with itself.

The Fourier series for H(ω) is H(ω) = 1 + eiω so

Y (ω) = H(ω)H(ω)

= (1 + eiω)(1 + eiω)

= 1 + 2eiω + e2iω

so y0 = y2 = 1, y1 = 2, and all other yn = 0.

Types of Filters

FiltersTypes of Filters

I For our purposes, we will think of a filter as the processor hin the definition of convolution.

I We will identify filters by some characteristics:I A filter h is said to be causal if hk = 0 whenever k < 0.

Types of Filters

I If we convolve x with causal filter h, we obtain

yn =∑

hkxn−k

=∞∑

hkxn−k

= h0xn + h1xn−1 + . . .

I So when h is causal, yn is formed from xn and thepredecessors of xn.

Types of Filters

yn =∑

hkxn−k

=∞∑

hkxn−k

= h0xn + h1xn−1 + . . .

Types of Filters

yn =∑

hkxn−k

=∞∑

hkxn−k

= h0xn + h1xn−1 + . . .

Types of Filters

I Until we consider biorthogonal wavelet transforms, we willonly work with so-called Finite Impulse Response filters.

I Suppose h is causal and let L > 0, L ∈ Z. If hk = 0 fork > L, h0, hL 6= 0, then we say h is a finite impulseresponse or FIR filter.

I If h is FIR with hk ≥ 0 and∑k

hk = 1, then it is often

convenient to think of h as moving averages - we arecomputing the weighted average of xn, xn−1, . . . , xn−L.

Types of Filters

Lowpass and Highpass Filters

FiltersLowpass and Highpass Filters

I Recall the averaging filter h = (h0, h1) = (12 , 1

2).I If we convolve h with a bi-infinite sequence u consisting

entirely of 1’s, we obtain

yn =∑

hkun−k =12(un + un−1) = 1 = un

I If we convolve h with v where vk = (−1)k , then we obtain

yn =∑

hkvn−k =12(vn + vn−1) = 0

yn =∑

hkun−k =12(un + un−1) = 1 = un

yn =∑

hkvn−k =12(vn + vn−1) = 0

yn =∑

hkun−k =12(un + un−1) = 1 = un

yn =∑

hkvn−k =12(vn + vn−1) = 0

yn =∑

hkun−k =12(un + un−1) = 1 = un

yn =∑

hkvn−k =12(vn + vn−1) = 0

I u and v are important testing sequences for filters. Thesequence u contains no oscillations whatsoever - it isconstantly 1.

I On the other hand, elements of v change sign at eachelement - it is highly oscillatory.

I Note that h applied to u returned u while h applied to vproduced the 0 sequence.

I Filters that generally reproduce low oscillatory signals andannihilate (or dampen) high oscillatory signals are knownas lowpass filters.

The modulus |H(ω)| of a lowpass filter typically looks like:

In general, we will ask that if h is a lowpass filter, then

H(0) = 1 and H(π) = 0

This is equivalent to

L∑k=0

hk = 1 andL∑

(−1)khk = 0

Note that the ideal lowpass filter would look like

But it is easy to show that this 2π-periodic functions has aFourier series consisting of infinitely many nonzero values so itis not desirable for applications.

I The filter g = (g0, g1) = (12 ,−1

2), when convolved with uproduces

yn =∑

gkun−k =12(un − un−1) = 0

I If we convolve g with v we have

yn =∑

gkvn−k =12(vn − vn−1) = (−1)n = vn

I The filter g = (g0, g1) = (12 ,−1

yn =∑

gkun−k =12(un − un−1) = 0

yn =∑

gkvn−k =12(vn − vn−1) = (−1)n = vn

I The filter g = (g0, g1) = (12 ,−1

yn =∑

gkun−k =12(un − un−1) = 0

yn =∑

gkvn−k =12(vn − vn−1) = (−1)n = vn

I Thus g annihilates the non-oscillatory signal u andpreserves the highly oscillatory signal v.

I Such a filter is called a highpass filter.I The Fourier series for g is

G(ω) =12− 1

and note that G(0) = 0 and G(π) = 1.

G(ω) =12− 1

The graph of |G(ω)| is

I It is easy to build a highpass filter from a lowpass filter.I If h is a lowpass filter with H(0) = 1 and H(π) = 0, thenI Take G(ω) = H(ω + π). Then G(0) = H(π) = 0 and

G(π) = H(2π) = H(0) = 1.I Note that

G(ω) = H(ω + π) =∑

hkeik(ω+π) =∑

hk (−1)keikω

G(ω) = H(ω + π) =∑

hkeik(ω+π) =∑

hk (−1)keikω

G(ω) = H(ω + π) =∑

hkeik(ω+π) =∑

hk (−1)keikω

G(ω) = H(ω + π) =∑

hkeik(ω+π) =∑

hk (−1)keikω

G(ω) = H(ω + π) =∑

hkeik(ω+π) =∑

hk (−1)keikω

I We will build our wavelet transforms from lowpass andhighpass filters.

I Lowpass filters tend to produce a good approximation ofthe original signal in areas where the values of the signalsare homogeneous.

I Highpass filters tend to show us where significant changein the signal is taking place.

Convolution as a Matrix ProductThe Convolution Matrix

I Since we will build our wavelet transforms usingconvolutions with lowpass and highpass filters, it is naturalto ask what convolution looks like as a matrix.

I Let’s consider a FIR filter (h0, . . . , hL).I If we think of y = Hx, we have

...y−2y−1y0y1...

...x−2x−1x0x1...

. . .. . .

. . .hL . . . h2 h1 h0 0 0 0 0 0 0 0 00 hL . . . h2 h1 h0 0 0 0 0 0 0 0

. . . 0 0 hL . . . h2 h1 h0 0 0 0 0 0 0 . . .0 0 0 hL . . . h2 h1 h0 0 0 0 0 00 0 0 0 hL . . . h2 h1 h0 0 0 0 0

. . .. . .

So in the case where h is an FIR filter, the matrix H is a lowertriangular matrix with h0 down the main diagonal, h1 on the firstsubdiagonal and so on.

Convolution as a Matrix ProductDe-Convolution

I It is natural to ask if we can invert or de-convolve theprocess.

I We can appeal to the convolution theorem here:

Y (ω) = H(ω)X (ω) ⇒ X (ω) =Y (ω)

I As long as H(ω) 6= 0, then theoretically we could write 1H(ω)

as a Fourier series M(ω) and the Fourier coefficients mkconvolved with yk would produce xk .

Y (ω) = H(ω)X (ω) ⇒ X (ω) =Y (ω)

I There’s only one small hitch here . . .

I We want to convolve with lowpass filter h and highpassfilter g, but

I h lowpass ⇒ H(π) = 0I g highpass ⇒ G(0) = 0

I So we can’t de-convolve when using lowpass or highpassfilters!

I This should make sense - if you are given a list of averagesof consecutive numbers, you have no way of knowing whatthe original numbers are.

I We’ve hit a roadblock - that means . . .

I Workshop Over! We’ll just goof off for the next 3 days.I Just kidding . . . let’s look at some student issues before

breaking.

Teaching Ideas

In The ClassroomTeaching Ideas

I We work lots and lots of basic convolution products.I I make sure to show students that multiplication is

convolution - they really like this - see Exercise 3 on page158.

I The conditions H(0) = 1 ⇒∑

hk = 1 andH(π) = 0 ⇒

∑(−1)khk = 0 come up repeatedly during the

remainder of the course.

Teaching Ideas

hk = 1 andH(π) = 0 ⇒

Teaching Ideas

hk = 1 andH(π) = 0 ⇒

Teaching Ideas

hk = 1 andH(π) = 0 ⇒

Teaching Ideas

I I usually have them try to build their own lowpass/highpassfilters. You can add some conditions too - like a lowpassfilter with H ′(π) = 0.

I It is very important that they make the connect - thenumbers in the convolution matrix are the filter elementsreversed and they are also the Fourier coefficients. There’sa lot going on here.

I For future work, it is important to have them work throughproblems 6-11 in Section 5.1.

Teaching Ideas

Computer Usage

In The ClassroomComputer Usage

I I am in the process of developing an animation routine forconvolving vectors, but other than that I don’t use thecomputer for much more than graphing moduli of Fourierseries of filters and manipulating Fourier series and seeingthe results visually.

In The ClassroomStudent Difficulties

I Getting the convolutions correct when both h and x are offinite length takes a lot of practice.

I The properties on H(ω), G(ω) to make the filters lowpass,highpass, respectively, never seems to seek in. This isused throughout the course and it gets frustrating having torepeat them.

I Students seem generally uncomfortable manipulating finitelength Fourier series. They are fine with looking at a graphof the modulus and determining if a filter islowpass/highpass, but they are hesitant to manipulate aFourier series to produce say a highpass filter from alowpass filter.

Today’s Schedule

9:00-10:15 Lecture One: Why Wavelets?10:15-10:30 Coffee Break (OSS 235)10:30-11:45 Lecture Two: Digital Images, Measures, and

Huffman Codes12:00-1:00 Lunch (Cafeteria)1:30-2:45 Lecture Three: Fourier Series, Convolution and

Filters2:45-3:00 ⇒Coffee Break (OSS 235)3:00-4:15 Lecture Four: 1D and 2D Haar Transforms5:30-6:30 Dinner (Cafeteria)

fourier series, convolution, and filters - university of st....

Documents