A1 2011 1 / 1

A1 Time-Frequency Analysis

David Murray

david.murray@eng.ox.ac.ukwww.robots.ox.ac.uk/∼dwm/Courses/2TF

Hilary 2011

A1 2011 2 / 1

Course Contents

1 From Signals to Complex Fourier Series

2 From Complex Fourier Series to the Fourier Transform

3 Convolution. The impulse response and transfer functions

4 Sampling, Aliasing

5 Energy and Power & Energy Spectra, and Correlation

6 Random Processes and Random Signals

A1 2011 3 / 1

Random Processes and Random Signals

Recall our description in Lecture 1 of random vs deterministicsignals.

A deterministic signal is one that can be described by afunction, mapping, or some other recipe or algorithm — if youknow t , you can work out f (t).

A random signal is determined by some underyling randomprocess.

Although its statistical properties might be known (e.g. you mightknow its mean and variance) you cannot evaluate its value at time tYou might by able to say something about the likelihood of itstaking some value at time t , but not more.

A1 2011 4 / 1

Why are we particularly bothered with randomsignals?

The random signals might be noise, but ...... being able to determine noise properties in the frequencydomain can be very useful in diminishing the effect of noise on the“proper” information-bearing signals.

Second, the signals from a random process might be theinformation-bearing signals themselves.

E.g., electromagnetic signals generated from solar activity.To the solar physicist they tell something about underlying nuclearprocesses in the sun.To the communications engineer they might just be noise.But to the engineer designing the satellite, understanding themmight improve comms quality, and reduce the risk of damage to thesatellite during a solar storm.

A1 2011 5 / 1

However ... we have a problem

You may just have noticed that all our analysis has involved

integration over all time —∫ ∞−∞

f (t) · · · dt pops up everywhere

This is fine for a deterministic signal ...

... but for a random signal it implies having to wait for all timecollecting the random signal before we can work out, say, itsauto-correlation.

Fortunately there is a way around this difficulty.

A1 2011 6 / 1

Probability review (1)

Consider a random variable x .

Probability DISTRIBUTION function

Px(ξ) = Prob[x ≤ ξ]

Varies from 0 at −∞ to 1 at∞ and has anon-negative gradient.

Probability DENSITY function (pdf)

px(ξ)dξ = Prob[ξ < x ≤ ξ + dξ]

orpx(ξ) =

ddξ

Px(ξ) .

For all ξ, px(ξ) ≥ 0 and∫∞−∞ px(ξ)dξ = 1.

Convenient to write p(x), rather thanpx(ξ).

P ( ) 1

0

x

x

ξ ξ+δξ

p(x)

x

A1 2011 7 / 1

♣ Example: Gaussian or Normal distributionA 1-d Gaussian or normal pdf with mean µ and variance σ2 is

p(x) =1√2πσ

exp(− (x − µ)

2

2σ2

)For µ = 0, σ = 1 ...

−3 −2 −1 0 1 2 30

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

x

p(x

)

−2 −1 0 1 20

0.2

0.4

0.6

0.8

1

x

erf

(x)

The probability distribution function is

P(x) =∫ x−∞

p(X )dX

A1 2011 8 / 1

Describing pdfs: moments and central moments

No better description of the pdf than the pdf itself

But useful to have rougher descriptors — of where the centre of gravityof the pdf lies, how spread out it is, etc

The mean or first moment of a discrete random variable is

µx = E [x ] =∑

i

xiP(xi)

and of continuous random variable is

µx = E [x ] =∫ ∞−∞

xp(x)dx

The n-th moment is

E [xn] =∫ ∞−∞

xnp(x)dx

A1 2011 9 / 1

Moments and central moments

Function spread is captured by central moments ...

The n-th central moment is

E [(x − µ)n] =∫ ∞−∞

(x − µ)np(x)dx

The 2nd central moment is the variance

σ2 = var(x) = E [(x − µ)2] =∫ ∞−∞

(x − µ)2p(x)dx

The standard deviation is the root of the variance

σ =√

var(x)

A1 2011 10 / 1

The expectation value of any function

The expection of any function f of x is

E [f (x)] =∫ ∞−∞

f (x)p(x)dx

Another useful relationship

E [(x − µx )2] = E [x2]− 2E [x ]µx + µ2x= E [x2]− µ2x

A1 2011 11 / 1

Random processes and ensembles

The random variable x might be regarded as random in itself, orregarded as the mapping from the random output θ of anexperiment x ≡ x(θ).

We will be particularly concerned with random processesx(t) ≡ x(t , θ), which are functions of time.

BUT when we refer to x(t1) — or, more fully, x(t1, θ) — it isimportant to realize that this is NOT the unique value of therandom variable at some time.

Instead it is just one of an ensemble of values that could beobtained in repeated trials

A1 2011 12 / 1

Random processes and ensemblesTo repeat: it is just one of an ensemble of values that could beobtained in repeated trials

t

t

t

t

t1

Thus x(t1) has an expected value of

E [x(t1)] =∫

xx(t1)p(x)dx =

∫θ

x(t1, θ)p(θ)dθ .

This expected value is called an ensemble average.

A1 2011 13 / 1

Random processes and ensembles

Just suppose that E [x(t)] wasthe same for any value of t .

It might be re-written as justE [x ].

A very reasonable question toask is whether this is thesame as the temporal average

limT→∞

1T

∫ T/2−T/2

x(t)dt .

We will return to this shortly.

t

t

t

t

t1

A1 2011 14 / 1

The ensemble autocorrelation and stationarity

In general, the definition of theensemble autocorrelation ofa random process involvesboth the times t1 and t2 thatsamples are made.That is

Rxx (t1, t2) = E [x(t1)x(t2)] .

t1

t2

t

t

t

t

which meansTake the product of values x(t1) and x(t2) from one trial, thenaverage over all trials

Rxx (τ) = E [x(t1)x(t2)] =∫ ∞−∞

x(t1, θ)x(t2, θ)p(θ)dθ .

A1 2011 15 / 1

The ensemble autocorrelation and stationarity

But ...IF the expectation value E [x(t)] is independent of time

AND the ensemble value for autocorrelation depends only on thedifference τ between t2 and t1

THEN the random process is said to be wide-sense stationary .

Then the autocorrelation depends only on the τ = t2 − t1 ...For a wide-sense stationary process,the ensemble autocorrelation is

Rxx (τ) = E [x(t)x(t + τ)] =∫ ∞−∞

x(t , θ)x(t + τ, θ)p(θ)dθ .

A1 2011 16 / 1

The ensemble autocorrelation /ctd

Notice anything?The ensemble autocorrelationhas units of powerSo the comparison we shouldmake is with definition ofautocorrelation for power signals

is aPOWER

E[x(t)x(t+ τ)]

This makes sense!A random signal can have no deterministic mechanism formaking itself smaller as time progresses, and so must haveinfinite energy ...

This means that our random processes cannot be guaranteed tohave Fourier Transforms

A1 2011 17 / 1

Ensembles and ErgodicityWe now have two definitions of the autocorrelation function of astationary random process.

The ensemble definition The integral over time

RExx (τ) = E [x(t)x(t + τ)] , RTxx = limT→∞

12T

∫ T−T

x(t)x(t + τ)dt

Both are perfectly proper. One is a snapshot at one time of awhole ensemble of processes. The other looks at the behaviourof one process over all time.

If the results are the same, the process is ergodic.

An ergodic random processis a stationary random process whose

ensemble autocorrelation = its temporal autocorrelation

A1 2011 18 / 1

Ergodicity exemplified

A village has two pubs. Suppose you wished to know which wasthe more popular.

The ensemble approach would be to go into both on the sameevening and count the number of people in each.

The temporal approach would be follow one villager for manyevenings and log which (s)he chose to go to.

If you reached the same result, you will have shown the drinkingpreferences of villagers is an ergodic process.

Of course, humans rarely behave ergodically, but engineeringsystems do.

A1 2011 19 / 1

Power Spectral Density of a Random Process

Because they are power signals, random processes don’tnecessarily have Fourier Transforms

But the power spectral density still exists ...

The power spectral density S(ω) of a stationary random processis defined exactly as in Lecture 5 for a power signal—

Sxx (ω) = FT [Rxx ]

We will be concerned exclusively with ergodic processes ...

... so that the above statement does not give rise to ambiguity.

A1 2011 20 / 1

Descriptors of a random process

We are about at the point where we can discuss how to analyzethe response of systems to a random process as input.

Howeverwe cannot give “exact” results for such processes in the timedomain.We will be able to make statements about the the autocorrelationand power spectral density, and in addition have access to otherstatistical descriptors, such as the mean, variance, and secondmoment.

µ = E [x ] =∫θ

x(θ)p(θ)dθ

σ2 = E [(x − µ)2] = E [x2]− (E [x ])2

E [x2] =∫θ

x2(θ)p(θ)dθ

A1 2011 21 / 1

♣ Application to a white noise process(1)

White noise describes a randomprocess whose mean is zero andwhose autocorrelation is adelta-function.

0 100 200 300 400 500−1

−0.5

0

0.5

1Zero−Mean Random Noise

time (milliseconds)

Can we explain both?White noise has the property it is equally likely to take positive ornegative values from instant to instant.

⇒ the mean is obviously µ = 0

What about the auto-correlation? ...

A1 2011 22 / 1

Application to a white noise process (2)

When you shift the signal by even an infinitesimal amount to find

Rxx (∆τ) = limT→∞

12T

∫ T−T

x(t)x(t + ∆τ)dt

the individual products x(t)x(t + ∆τ) being integrated areequally likely to be positive and negative, large and small.Summing these by integration must give zero.

When there is no shift, Rxx (0) = limT→∞

12T

∫ T−T|x(t)|2dt must be

finite for a non-zero signal.

Thus Rxx (τ) = Aδ(τ)

A1 2011 23 / 1

Application to a white noise process (3)

Rxx (τ) = Aδ(τ) but what is A?

The earlier expression actually indicates Rxx (0) = E [x2].

But for a process with µ = 0,

σ2 = E [x2]− µ2 = E [x2] .

So A = σ2.

For zero-mean white noise with variance σ2,

Rxx (τ) = σ2δ(τ)⇔ Sxx (ω) = σ2

The power spectral density, which is the Fourier transform of theautocorrelation function, is uniform and has the value of thesignal’s variance.

A1 2011 24 / 1

Response of system to random signals

The final part of this story is to work out how systems affect thedescriptors.That is, given the mean and variance of the input, what are themean and variance of the output?

Below, input x(t) and output y(t)are random. The system howeveris deterministic!

*h(t)x(t) y(t)

X( ω Y() )ω ω )H(

Given a temporal input x(t), the output y(t) is

y(t) = x(t) ∗ h(t) =∫ ∞−∞

x(τ)h(t − τ)dτ

A1 2011 25 / 1

Response of system to random signals: MEAN (1)The expectation value (mean) of the output y is

E [y ] = E[∫ ∞−∞

x(τ, θ)h(t − τ)dτ]

=

∫ ∞θ=−∞

∫ ∞τ=−∞

x(τ, θ)h(t − τ)dτ p(θ)dθ

=

∫ ∞τ=−∞

[∫ ∞θ=−∞

x(τ, θ)p(θ)dθ]

h(t − τ)dτ

=

∫ ∞τ=−∞

E [x(τ)]h(t − τ)dτ

But for a stationary function E [x(τ)] is independent of τ — it is theconstant E [x ]. Taking this outside the integral, and then substitutingp = t − τ , we find that

E [y ] = E [x ]∫ ∞−∞

h(t − τ)dτ

= −E [x ]∫ −∞∞

h(p) dp = E [x ]∫ ∞−∞

h(p) dp .

A1 2011 26 / 1

Response of system to random signals: MEAN (2)Reminder

E [y ] = E [x ]∫ ∞−∞

h(p) dp .

But we know that FT [h] = H(ω). That is,∫ ∞−∞

h(p)e−iωpdp = H(ω) ⇒∫ ∞−∞

h(p)e−i0pdp = H(0) .

⇒∫ ∞−∞

h(p)dp = H(0) .

For a stationary random input x(t) the mean of the outputis

µy = E [y ] = E [x ] H(0) .

This means the output is also stationary.

A1 2011 27 / 1

Response of system to random signals: variance

A different approach needed to find variance of the output

First remember that for an ergodic signal, Rxx (0) = E [x2].

As σ2y = E [y2]− (E [y ])2, use the mean and autocorrelation tofind the variance of the output

σ2y = E [y2]− (E [y ])2 = Ryy (0)− µ2y

We know µy = H(0)µx ...but we don’t know Ryy (0).

But remember the Wiener-KhinchinTheorem for power signals?

comrade

Search me

A1 2011 28 / 1

Response of system to random signals: variance /ctd

The Fourier Transform of the Auto-Correlation is the PSD

FT [Rxx (τ)] = Sxx (ω)

Now, if the random process x(t) is stationary and ergodic, thisstatement must still hold good for x(t) and for the output y(t) too.

FT [Ryy (τ)] = Syy (ω)

Then (the proof is in the notes)

The PSD of output and input are related by

Syy (ω) = |H(ω)|2 Sxx (ω) = H(ω)H∗(ω) Sxx (ω).

This is true for all power signals signals, not just random ones.

A1 2011 29 / 1

Response of system to random signals: variance /ctd

SoRyy (τ) =

12π

∫ ∞−∞

Syy (ω) eiωτ dω

Hence

Ryy (0) =1

2π

∫ ∞−∞

Syy (ω) dω

=1

2π

∫ ∞−∞|H(ω)|2Sxx (ω) dω .

A1 2011 30 / 1

Response of system to random signals: variance /ctd

Recipe, given input mean µx and autocorrelation Rxx :

Use µx and autoc Rxx(0) to find variance σ2x if needed

Take FT of autoc to find the PSD Sxx of input

Multiply by |H(ω)|2 to find the PSD Syy of output

Take the IFT at τ = 0 to find Ryy (0) = (1/2π)∫∞−∞ Syy dω

Find output mean as µy = H(0)µx

Find output variance as σ2y = Ryy (0)− µ2y

A1 2011 31 / 1

♣ Example 1

[Q] White noise with zero mean and variance σ2 isinput to the circuit in the Figure. (i) Sketch theinput’s autocorrelation and power spectral density,(ii) derive the mean and variance of the output, and(iii) sketch the output’s autocorrelation and powerspectral density.

R

Cx(t) y(t)

[A] (i) The input x(t) is zero mean white noise, with a variance of σ2.

⇒Rxx(τ) = σ2δ(τ) and Sxx(ω) = FT[σ2δ(τ)

]= σ2 .

Sketch later[A] (ii) The transfer function of the RC circuit in the Figure is

H(ω) =(1/jωC)

R + (1/jωC)=

11 + jωRC

.

⇒E [y ] = E [x ]H(0) = 0× 1 = 0 .

A1 2011 32 / 1

♣ Example 1 /ctd ...3. Power spectral density of the output is

Syy (ω) = Sxx |H(ω)|2 =σ2

1 + ω2(RC)2

Recipe

1 use mean and variance to findautocorrelation at zero Rxx (0)= σ2δ(τ)

2 take its FT to get the PSD Sxx ofinput = σ2

3 multiply by |H(ω)|2 to find thePSD Syy of output

4 take the IFT at τ = 0 to findRyy (0).

5 find mean of outputµy = H(0)µx

6 find variance σ2y = Ryy (0)− µ2y

4.

E [y2] = Ryy (0) =1

2π

∫ ∞−∞

Syy (ω)dω =1

2πσ2∫ ∞−∞

11 + ω2(RC)2

dω

(substitute ωRC = tan θ) =1

2πσ2

1RC

∫ π/2−π/2

sec2 θsec2 θ

dθ

= σ2/2RC .

6. The variance of y is therefore σ2y = E [y2]− (E [y ])2 = σ

2

2RC

A1 2011 33 / 1

♣ Example /ctd ...[A](iii) To find the autocorrelation of the output we need the IFT of the PSD

Ryy = FT −1 [Syy (ω)] = FT −1[

σ2

1 + ω2(RC)2

]From HLT

e−a|t| ⇔ 2aa2 + ω2

so that12

e−|t| ⇔ 11 + ω2

.

The parameter scaling property with α = 1/RC It gives

f (t/RC)⇔ |RC|F (ωRC) ⇒ 12|RC| e

−|t/RC| ⇔ 11 + ω2(RC)2

.

Finally, given that RC > 0

Ryy (τ) =σ2

2RCe−|τ |/RC ⇔ σ

2

1 + ω2(RC)2= Syy (ω) .

Note that Ryy (0) = σ2/2RC, which is a nice check.

A1 2011 34 / 1

♣ Example: Can we understand the sketches?

σ2

Rxx

τ( )

σ2

S xx ω( )

ωτ τω

S ω( )yyR τ( )

yy

σ2

σ2/2RC

Autocorrelation and PSD of input x(t) Autocorrelation and PSD of output y(t)

What about the power spectral density?

The circuit was a low pass filter with 1st order roll-off.

The filter will greatly diminish higher frequencies ......but not cut them off completely.

The psd at zero frequency remains at σ2.

A1 2011 35 / 1

♣ Example: Can we understand the sketches? (2)

σ2

Rxx

τ( )

σ2

S xx ω( )

ωτ τω

S ω( )yyR τ( )

yy

σ2

σ2/2RC

The auto-correlation needs a little more thought.The autocorrelation of white noise was a delta-function becausethe next instant’s value was equally likely to be positive or negative,large or small ...... in other words there was no restriction on the rate of change ofthe signal.

But the low pass filter prevents such rapid change — so the nextinstant’s value is more likely to be similar to this instant’s.

However, the correlation will fade away with time.

This is exactly what you see. The decay in the autocorrelation hasa decay constant of 1/RC.

A1 2011 36 / 1

♣ Example 2[Q] Each pulse in a continuous binary pulse train has a fixed durationT , but takes the value 0 or 1 with equal probability (P = 1/2),independently of the previous pulse.

0

1

t

i. If xi is the random variable representing the height of pulse i ,calculate E [xi ], E [x2i ], and E [xixj ] for i=j .

ii. Find and sketch the auto-correlation function Rxx (τ) of the pulsetrain.

iii. Hence find and sketch the power spectral density Sxx (ω) of thepulse train.

A1 2011 37 / 1

♣ Example 2

[A](i)The pulse heights xi are 0 and 1, each with a probability of Pi = 1/2.

E [xi ] =∑

i

xiPi = 0×12

+ 1× 12

=12

E [x2i ] =∑

i

x2i Pi = 02 × 1

2+ 12 × 1

2=

12

Each pulse value xi is independent of any otherHence

E [xixj ] = E [xi ]× E [xj ] =12× 1

2=

14

A1 2011 38 / 1

♣ Example 2

[A](i) To find the ensemble average, take the product x(t)x(t + τ) andaverage over many random trials or experiments

τt+

τt+

t

t

repeated trials

Then

RExx =∫

x(t , θ)x(t + τ, θ)p(θ)dθ

A1 2011 39 / 1

♣ Example 2

τt+

τt+

t

t

repeated trials

RExx =∫

x(t , θ)x(t + τ, θ)p(θ)dθ

= (0× 0) p(x(t) = 0, x(t + τ) = 0)+(0× 1) p(x(t) = 0, x(t + τ) = 1)

+(1× 0) p(x(t) = 1, x(t + τ) = 0)+(1× 1) p(x(t) = 1, x(t + τ) = 1)

= p(x(t) = 1, x(t + τ) = 1)

A1 2011 40 / 1

♣ Example 2

Reminder

RExx = p(x(t) = 1, x(t + τ) = 1)= The probability that x(t) = 1 AND x(t + τ) = 1

When τ > T there is always an independent transition during intervalτ so that the joint probability bcomes a product

RExx = p(x(t) = 1, x(t + τ) = 1)= p(x(t) = 1)× p(x(t + τ) = 1)

=12

× 12

=14

A1 2011 41 / 1

♣ Example 2

When 0 ≤ τ ≤ T ,

RExx = p(x(t) = 1, x(t + τ) = 1)= p(x(t) = 1, no ↓ transition during following τ)= p(x(t) = 1)× p(no ↓ transition during following τ)= p(x(t) = 1)× (1− p(↓ transition during following τ))

=12

(1− 1

2p(ANY transition during following τ)

)=

12

(1− 1

2

( τT

))

The last step multiplies the frequency of transition 1/T by the timeinterval τ to get the probability of transition.

A1 2011 42 / 1

♣ Example 2

So

RExx ={

1/4 for t > T12

(1− 12

(τT

))0 ≤ τ ≤ T

Finally we use the even symmetry property to complete the function

Rxx (τ) =14

+14

ΛT (τ)

where ΛT (τ) is a triangle of unit height, half-width T .Rxx

1/2

1/4

−T T

τ

A1 2011 43 / 1

♣ Example 2Reminder: Rxx (τ) =

14

+14

ΛT (τ)

(Lec 2++) FT [ΛT (τ)] = Tsin2(ωT/2)

(ωT/2)2

(Lec 3) FT [1] = 2πδ(ω)so that the power spectral density becomes

Sxx (ω) = FT [Rxx ] =T4

[sin2(ωT/2)

(ωT/2)2

]+π

2δ(ω)

ω

Sxx

A1 2011 44 / 1

Course Content Reminder

1 From Signals to Complex Fourier Series

2 From Complex Fourier Series to the Fourier Transform

3 Convolution. The impulse response and transfer functions

4 Sampling, Aliasing

5 Energy and Power & Energy Spectra, and Correlation

6 Random Processes and Random Signals