identification of physical systems (applications to condition monitoring, fault diagnosis, soft...

2Characterization of Signals:Correlation and Spectral Density

2.1 IntroductionA very important characterization of signals is correlation and its frequency domain counterpart: powerspectral density. It is a measure of interdependence or similarity between the two waveforms.

The operation of correlation has wide applications in science and engineering including detection ofknown signals buried in noise, pattern recognition, estimation of time delay (e.g., in radar and sonarfor determining the range), identification of an unknown model of a system, synthesis, processingand compression of speech, and medical diagnosis [1–14]. It is employed to compute the measure ofsimilarity or dissimilarity between two waveforms and enjoys the following properties of similaritiesand dissimilarities. Auto-correlation is a special case of correlation when the two signals are the same,that is, the auto-correlation is the correlation of the signal with itself. Auto-correlation is also a measureof similarity between a given time series and a lagged version of itself over successive time intervals. Itis the same as calculating the correlation between two different time series, except that the same timeseries is used twice – once in its original form and once lagged by one or more time periods. For examplethe auto-correlation of a zero mean white noise is a delta function, the auto-correlation of a sum of signaland a zero mean white noise process is the auto-correlation of the signal for all lag values except at zerolag value.

Given a waveform with a rational spectrum, the waveform and its auto-correlation share the samecharacteristics. For example the auto-correlation function of a constant, exponential, sinusoid exponen-tially damped sinusoid is respectively a constant, exponential, and a sinusoid and exponentially dampedsinusoid. The auto-correlation function of a periodic signal is periodic of the same period while the cross-correlation of two signals of different periods is zero. Auto-correlation finds application in many areas,including extraction of the features from the signal, estimation of its model, detection of non-randomnessin data, and the periodicity in a waveform.

Cross-correlation of a zero mean process and a deterministic waveform is zero, and that between twosimilar waveforms is accentuated. These properties are exploited to attenuate noise and enhance a knownsignal corrupted by noise. The delay between two similar waveforms can be found as the lag value (ortime) at which their cross-correlation is maximum.

The power spectral density is the Fourier transform of the auto-correlation function, while the cross-power spectral density is the Fourier transform of the cross-correlation. The power and cross-spectraldensities are widely used as they are computationally fast efficient implements, since the Fast Fourier

Identification of Physical Systems: Applications to Condition Monitoring, Fault Diagnosis, Soft Sensor andController Design, First Edition. Rajamani Doraiswami, Chris Diduch and Maryhelen Stevenson.© 2014 John Wiley & Sons, Ltd. Published 2014 by John Wiley & Sons, Ltd.

58 Identification of Physical Systems

Transform (FFT) algorithm is used to implement them. Further, they provide better insight into theproperties of auto- and cross-correlation. Correlation of two signals is essentially a measure of spectraloverlap between them. A cross-power spectral density indicates how the energy/power is distributedover frequency. The power and cross-power spectral analysis finds application in many fields includingmedical diagnosis, speech recognition, performance monitoring, fault diagnosis, and market analysisto name a few. In medical diagnosis, the biological signals/images such as the electro-encephalogram(EEG), the electrocardiogram (ECG), the phonocardiogram (PCG), the myo-electrical signals, and so on,are classified using the power spectral density. The power spectral density serves as a feature to distinguishone class from the other. In speech processing, power spectral density plays a central role in systemsinvolving the synthesis, recognition, and compression of speech. In radar and sonar the spectral contentis analyzed to determine the location and the type of target, such as a plane, ship, or other. In seismology,spectral analysis of the signals gives useful information on the movement of the ground. The spectralanalysis is used to monitor the health of mechanical system-rotating machinery, vibratory systems, andso on, where the spectral signature is used to determine the wear and tear, breakage, or malfunction ofmechanical parts. In control systems, the spectral analysis is used to design a controller and also helpin performance monitoring and fault diagnosis. In economics, meteorology, and stock market analysis,spectral analysis may reveal trends –hidden periodicities reveal certain recurring or cyclic phenomena.

The relation between cross- and auto-power spectral densities of the input and the output are derived.It is shown that the frequency response of a system may easily be computed from the cross-spectraldensity of the input and the output, and the power spectral density of the input. The cross-correlationbetween the input and the output can be used to estimate the time delay.

The magnitude squared coherence spectrum, which measures similarity or dissimilarity between twosignals, is defined. This measure is the frequency domain counterpart of the correlation coefficient.The coherence spectrum is widely used in many areas including medical diagnosis, performancemonitoring, and fault diagnosis. If the spectral bandwidth of interest is known, then only coherence overthe frequency of interest is considered and the rest of the spectral regions are ignored as they containnoise and other artifacts.

2.2 Definitions of Auto- and Cross-Correlation (and Covariance)If the two signals are identical, that is x(k) = y(k), then the correlation of x(k) with itself, denoted rxx(m),is called the auto-correlation of x(k), and if they are different, that is x(k) ≠ y(k), then rxy(m) is termedthe cross-correlation of x(k) and y(k). In the definition of correlation function, the index, m, is not a timeindex, rather, it is a time-lag, a time lead, or a time-shift index: m can take either positive or negativeinteger values. In general, rxy(m) ≠ ryx(m).

The correlation function of two signals of x(k) and y(k), is denoted by rxy(m), and depending upon thecharacteristics of x(k) and y(k), it is defined as follows:

∙ if x(k) and y(k) are real-valued energy signals, then we need to consider two cases:◦ the signals are of infinite duration

rxy(m) = limN→∞

N∑k=−N

x(k)y(k − m) (2.1)

◦ the signals are of finite duration

rxy(m) =N−1∑k=0

x(k)y(k − m) (2.2)

Characterization of Signals: Correlation and Spectral Density 59

∙ if x(k) and y(k) are real-valued power signals, then

rxy(m) = limN→∞

12N + 1

N∑k=−N

x(k)y(k − m) (2.3)

∙ if x(k) and y(k) are power signals and periodic with period M, then

x(k) = x(k + 𝓁M)

y(k) = y(k + 𝓁M) 𝓁 = ±1,±2,±3,…(2.4)

rxy(m) = 1M

M−1∑k=0

x(k)y(k − m) (2.5)

∙ if x(k) and y(k) are real-valued jointly wide-sense stationary random signals, then

rxy(m) = E[x(k)y(k − m)] =

∞

∫−∞

∞

∫−∞

x(k)y(k − m)fxy(x(k), y(k − m))dx(k)dy(k − m) (2.6)

where E [(.)] is the usual expectation operator indicating an ensemble average of the random processes(.). Assuming that the random process is ergodic in the mean and the correlation, their estimates areobtained from a single realization

𝜇x = E [x(k)] = lim itN→∞

12N + 1

N∑k=−N

x(k) (2.7)

The mean 𝜇y = E [y(k)] can be estimated similarly. The correlation is estimated as:

rxy(m) = E [x(k)y(k − m)] = lim itN→∞

12N + 1

N∑k=−N

x(k)y(k − m) (2.8)

Thanks to the assumption of wide-sense stationarity, the correlation is only a function of the lag m andnot of time index k. Since a random signal is a power signal, the averaging operation is used.

Computation of Correlation Using Truncated SignalsIn practice, the correlation of infinitely long signals (energy or power) is estimated from truncatedversions of the signals. If x(k) and y(k), k = 0, 1, 2,… , N − 1, are truncated signals, then the correlationrxy(m) is:

rxy(m) =N−1∑k=0

x(k)y(k − m) (2.9)

In this case the number of samples N is chosen to be sufficiently large, so that rxy(m) captures completelythe characteristics of the signal including its steady-state and transient behaviors. For example, if it is a


transient signal, N should be much larger than its settling time, and if periodic N should contain severalperiods of the signals. Expanding the summation operation we get

rxy(m) =

⎧⎪⎪⎨⎪⎪⎩

m−1∑k=0

x(k)y(k − m) +N∑

k=m

x(k)y(k − m) m ≥ 0

N−m∑k=0

x(k)y(k − m) +N∑

k=N−m+1

x(k)y(k − m) m < 0

(2.10)

All the data required for computing the correlation function rxy(m) are not available, as can be

deduced from the first summation termm−1∑k=0

x(k)y(k − m) for m ≥ 0, and the second summation term

N∑k=N−m+1

x(k)y(k − m) for m ≥ 0. The unavailable data for m ≥ 0 are y(k) : k = −1,−2,… ,−m, and

y(k) : k = N + 1, N + 2,… , N + m for m < 0. To overcome this problem, the values for the unavailabledata may be assumed equal to: zero, the mean value of the signal, or to a periodic extension of the signal.

The truncated signals may be treated either as power or energy signals, although they are commonlytreated as an energy signals.

Summary of the Definitions of CorrelationConsidering the signal characterization, the definition of correlation may be expressed compactly as:

rxy(m) =

⎧⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎩

limN→∞

N∑k=−N

x(k)y(k − m) infinite duration energy signals

N−1∑k=0

x(k)y(k − m) finite duration energy signals

limN→∞

12N + 1

N∑k=−N

x(k)y(k − m) power signal

rxy(m) = 1M

M−1∑k=0

x(k)y(k − m) M-periodic power signal

limN→∞

12N + 1

N∑k=−N

x(k)y(k − m) ergodic random (power) signals

(2.11)

Comments

∙ If x(k) and y(k) are complex-valued signals, then y(k) should be replaced by its complex conjugatey∗(k), in the definition of rxy(m) above.

∙ Correlation of energy and finite duration signals involves summing operations, while the powersignals, such as the periodic and the random signals, involve averaging operations.

Alternative Expression of CorrelationA change of variables allows one to express the correlation rxy(m) in the alternative form shown below.

Change the variable of summation from k to k where

k = k + m (2.12)

rxy(m) = limN→∞

12N + 1

N∑k=−N

x(k + m)y(k) = lim itN→∞

12N + 1

N∑k=−N

x(k + m)y(k) (2.13)


It important to emphasize that the time shift m is the difference between the time indices associated withthe first signal x(k) and that of the second signal y(k) in the correlation sum.

Covariance FunctionSimilar to correlation, one may define a covariance function of two stationary random signals x(k) andy(k). The covariance function is essentially the correlation function of two zero-mean signals, which areobtained from signals x(k) and y(k) by subtracting their means. The covariance of x(k) and y(k) is denotedby cxy(m). It is defined by

cxy(m) = E[(x(k) − 𝜇x)(y(k − m) − 𝜇y)] (2.14)

where 𝜇x = E [x(k)] and 𝜇y = E [y(k)] are the mean values of x(k) and y(k). Expanding and simplifyingwe get

cxy(m) = E [x(k)y(k − m)] − 𝜇x𝜇y = rxy (m) − 𝜇x𝜇y (2.15)

2.2.1 Properties of CorrelationExcept when noted, we derive all the properties of correlation for the case of energy signals whosedefinitions are given by Eq. (2.11); it can be shown that, when applicable, the resulting properties maybe extended to correlation of other signal types.

Correlation and Convolution

Proposition The correlation x(k) and y(k) is the same as the convolution of x(k) and y(−k)A convolution of x(k) and y(k) denoted conv(x(k), y(k)) is defined as

conv(x(k), y(k)) =∞∑

k=−∞x(k)y(m − k) (2.16)

Proof: Consider the expression for the correlation of the energy signal given in Eq. (2.11)

rxy(m) =∞∑

k=−∞x(k)y(k − m) (2.17)

where N may be finite or infinite. Let s(k) = y(−k). Rewriting y(k − m) = y (−(m − k)) = s(m − k) weget

rxy(m) =∞∑

k=−∞x(k)y (− (m − k)) =

∞∑k=−∞

x(k)s (m − k) = conv(x(k), y(−k)) (2.18)

where conv(x(k), y(−k)) denotes the convolution of x(k) and y(−k).The correlation complex valued signals x(k) and y(k) are the same as the convolution of complex

values x(k) and y∗(−k). A convolution of complex valued x(k) and y(k) is defined as

conv (x(k), y∗(k)) =∞∑

k=−∞x(k)y∗(m − k) (2.19)


Thus rxy(m) is the convolution of x(k) and y(−k). Note that y(−k) is the time reversed sequence of y(k).This means that the same algorithm may be used to compute correlation and convolution simply byreversing one of the sequences. Consequently, if one of the two sequences possesses symmetry such thatthe reversed sequence is a time shifted version of the original sequence (as is the case with constantsand sinusoids), then correlation of the two sequences will be equal to the time shifted versions of theconvolutions of the two sequences. As shown below, the frequency domain provides additional insightregarding the relationship between correlation and convolution.

The frequency domain provides additional insight regarding the relationship between correlation andconvolution. Letting ℑ(.) denote Fourier transform of (.), we have

ℑ{rxy(m)} = X(f )Y∗(f ) = |X(f )| |Y(f )|∠(𝜑x(f ) − 𝜑y(f )) (2.20)

ℑ {conv(x(k), y(k))} = X(f )Y(f ) = |X(f )| |Y(f )|∠(𝜑x(f ) + 𝜑y(f )) (2.21)

Where X(f ) = |X(f )|∠ (𝜑x(f )

); Y(f ) = |Y(f )|∠ (

𝜑y(f )); 𝜑x(f ),𝜑y(f ), |X(f )| and |Y(f )| are the phase and

the magnitude spectra of X(f ) and Y(f ). Comparing the expressions (2.20) and (2.21) we can deducethat the magnitudes of the Fourier transforms of correlation and convolution are identical while theirphase spectra differ. Note that convolution is a filtering operation while correlation is a pattern matchingoperation.

Relation between rxy(m) and ryx(−m)For complex values signals it can be deduced

rxy(m) = r∗yx(−m) (2.22)

Proof: Follows from Eqs. (2.1) and (2.12).

Symmetry Property of the Auto-Correlation Function

rxx(m) = r∗xx(−m) (2.23)

Proof: It is special case of Eq. (2.22) when x(k) = y(k).

This shows that the auto-correlation function is Hermitian. In case of real-valued signal, the autocorre-lation is real and even.

Maximum Values of Auto- and Cross-CorrelationsThe absolute value of the correlation is bounded as follows:

rxx(0) ≥ |rxx(m)| (2.24)√rxx(0)ryy(0) ≥ |rxy(m)| (2.25)|rxy(m) + r∗xy(m)| ≤ rxx(0) + ryy(0) (2.26)


Proof: Consider the following identity

limN→∞

12N + 1

N∑k−N

(ax(k) + by(k − m)) (a∗x∗(k) + b∗y∗(k − m)) ≥ 0 (2.27)

where a and b are complex scalars. Recall the following definitions:

rxy(m) = limN→∞

12N + 1

N∑k−N

x(k)y∗(k − m) (2.28)

rxx(0) = limN→∞

12N + 1

N∑k−N

x(k)x∗(k) (2.29)

ryy(0) = limN→∞

12N + 1

N∑k−N

y(k − m)y∗(k − m) (2.30)

Expanding and using the above definitions, the expression (2.27) becomes

a2rxx (0) + b2ryy (0) + 2abrxy(m) ≥ 0 (2.31)

Expressing in a quadratic form we get

[a b

] [rxx(0) rxy(m)r∗xy(m) ryy(0)

] [a∗

b∗

]≥ 0 for all a and b (2.32)

This shows that the center matrix in Eq. (2.32) is Hermitian and positive semi-definite, which impliesthat its determinant is real-valued. Furthermore, due to the inequality above, the determinant must begreater than or equal to zero:

rxx(0)ryy(0) ≥ |rxy(m)|2 (2.33)

Setting x(k) = y(k) we get

r2xx(0) ≥ |rxx(m)|2 (2.34)

Since rxx(0) ≥ 0 and ryy(0) ≥ 0, the inequalities (2.24) and (2.25) hold. Setting a = 1 and b = −1 in theinequality (2.32) we get Eq. (2.26).

Time Delay Between Two SignalsLet x(k) and y(k) denote two complex-valued signals where y(k) is a delayed version of x(k):y(k) =x (k − 𝜏) and 𝜏 is the time delay. An estimate of the time delay is given by

𝜏 = argmaxm

{ryx(m) + r∗yx(m)} (2.35)

where the cost function ryx(m) + r∗yx(m) = 2real{ryx(m)}The real-valued signal estimate of the time delay becomes

𝜏 = argmaxm

{ryx(m)} (2.36)


Proof: Replacing y(k) by x (k − 𝜏) in the definition of the correlation ryx (m) we find

ryx(m) = limN→∞

N∑k=−N

y(k)x∗(k − m) = limN→∞

N∑k=−N

x(k − 𝜏)x∗(k − m) (2.37)

From Eq. (2.37), it follows that

ryx (m) = rxx (m − 𝜏) (2.38)

Using the maximum value property of auto-correlation from Eq. (2.26), and in conjunction withEq. (2.38), we see that

|rxy(m) + r∗xy(m)| = |rxx (m − 𝜏) + r∗xx (m − 𝜏)| ≤ rxx(0) + ryy(0) (2.39)

From which it follows that the maximum value of |rxy(m) + r∗xy(m)| occurs at m = 𝜏.In the case that x(k) and y(k) are real-valued signals, using the maximum value property of auto-

correlation from Eq. (2.25), and in conjunction with Eq. (2.38), we see that

|ryx (m)| = |rxx (m − 𝜏)| ≤ rxx (0) (2.40)

From which it follows that the maximum value of |ryx (m)| occurs at m = 𝜏.

Additive Property

Consider a signal y(k) =M∑

i=1xi(k) which is formed of a sum of M signals xi(k). The auto-correlation

function of y(k) is the sum of the cross- and auto-correlation functions rxixj(m) of xi(k) and xj(k) given by

ryy(m) =M∑

i=1

M∑j=1

rxixj(m) =

M∑i=1

rxixi(m) +

M∑i=1

M∑j=1j≠i

rxixj(m) (2.41)

Proof: Consider the expression for ryy(m) for the case that y(k) is an energy signal. Substituting

y(k) =M∑

i=1xi(k) we get

ryy(m) = limN→∞

N∑k=−N

y(k)y(k − m) = limN→∞

N∑k=−N

(M∑

i=1

xi(k)

)(M∑

j=1

xj(k − m)

)(2.42)

Changing the order of summation, using the product of the two sums’ formulae given in the Appendix,and simplifying we get

ryy(m) =M∑

i=1

M∑j=1

(lim

N→∞

N∑k=−N

xi(k)xj(k − m)

)(2.43)


Using the definition of correlation we get

ryy(m) =M∑

i=1

M∑j=1

rxixj(m) =

M∑i=1

rxixi(m) +

M∑i=1

M∑j=1j≠i

rxixj(m) (2.44)

Comments The auto-correlation function of a sum of waveforms is not equal to sum of their auto-correlation; instead, it is a sum of the auto- and cross-correlations of the various component waveforms.In the case when the waveforms are uncorrelated, the cross-correlations will be zero, and hence theauto-correlation of a sum of waveforms is the sum of their auto-correlations.

If x(k), y(k) or both x(k), y(k) are periodic with period M, their correlation rxy(m) will also periodicwith a same period M, that is:

rxy(m) = rxy(m + 𝓁M): 𝓁 = ±1, ±2,±3,… (2.45)

Proof: We will assume that y(k) is periodic. Let us consider the definition Eq. (2.5) of a correlationfunction of x(k) and y(k). Substituting m = m + 𝓁M we get

rxy(m + 𝓁M) = 1M

M−1∑k=0

x(k)y(k − m − 𝓁M) (2.46)

Since y(k) = y(k + 𝓁M) 𝓁 = ±1,±2,±3,… , we get y(k − m) = y(k − m − 𝓁M)

rxy(m + 𝓁M) = 1M

M−1∑k=0

x(k)y(k − m) = rxy(m) (2.47)

Hence we have verified Eq. (2.45). Similarly we can prove that rxy(m) is periodic if x(k) or both x(k) andy(k) are periodic.

Noise Annihilation PropertyWe will first show that the cross-correlation of a deterministic signal, s(k), and a zero mean Wide-SenseStationary (WSS ) white noise process,v(k),is zero. The cross-correlation s(k) and v(k) is given by

E [s(k)v(k − m)] = 0 (2.48)

Proof: Consider the cross-correlation of s(k) and v(k) where E [v(k)] = 0. Simplifying we get

E [s(k)v(k − m)] = s(k)E [v(k − m)] = 0 (2.49)

Noise annihilationConsider a noisy signal x(k) = s(k) + v(k) where s(k) is a deterministic signal and v(k) is a zero-meanWSS random process. The cross-correlation of x(k) and s(k) is

rxs(m) = rss(m) (2.50)


Proof:

rxs(k) = E [x(k)s(k − m)] = E [s(k)s(k − m)] + E [v(k)s(k − m)] (2.51)

Since the second term on the right is zero in view of Eq. (2.48), we get

rxs(k) = E [x(k)s(k − m)] = E [s(k)s(k − m)] = rss(m) (2.52)

The noise annihilating property of cross-correlation with a known deterministic signal is widely used fordetecting a known signal in noise. A system which performs this operation is usually termed a correlatoror matched filter. Consider now the auto-correlation noisy signal x(k) = s(k) + v(k)

rxx(k) = E [x(k)x(k − m)] (2.53)

Expanding yields

rxx(k) = E [s(k)s(k − m)] + E [s(k)v(k − m)] + E [v(k)s(k − m)] + E [v(k)v(k − m)] (2.54)

Using the noise annihilation property (2.48) we get

rxx(k) = E [s(k)s(k − m)] + E [v(k)v(k − m)] (2.55)

Thus, assuming the noise, v(k), to be a zero-mean WSS random process, the auto-correlation of a noisysignal, x(k) = s(k) + v(k), will be equal to the sum of the auto-correlations of the signal and the noise.Generally, the auto-correlation of the noise will be dominant over an interval surrounding the lag valuem = 0 but will decay as |m| increases. Assuming the signal to be correlated over a longer interval than thenoise (this assumption will generally be valid for low-frequency signals as well as for periodic signals),the auto-correlation of s(k) will dominate for larger |m|. Thus, an inspection of the auto-correlationfunction of a noisy signal for large |m| will help to determine the characteristics of any buried signalsand will reveal the presence of any buried periodic signals

Generalization of the Properties of CorrelationExcept when noted, we derived all the properties of correlation for the case of energy signals. In general,these properties will hold for the other types of signals. The exceptions are fairly obvious; for example,the periodicity property applies only to the case when at least one of the signals is periodic and thenoise annihilation property applies only to the case when one of the signals is a zero-mean WSS randomprocess.

2.2.2 Normalized Correlation and Correlation CoefficientThe normalized correlation of two signals, x(k) and y(k), is denoted by 𝛾xy (m) and defined as:

𝛾xy(m) =rxy (m)√

rxx (0) ryy (0)(2.56)

Using the inequality (2.25), we find that the normalized cross-correlation is restricted to values between−1 and +1:

− 1 ≤ 𝛾xy(m) ≤ 1 (2.57)


The correlation coefficient is defined as

𝜌xy =cxy (0)√

cxx (0) cyy (0)(2.58)

where cxy (m) is the covariance of x(k) and y(k) given in Eq. (2.14). If x(k) and y(k) are zero-mean signals,then the correlation coefficient is equal to the normalized correlation at the lag index of m = 0. Thecorrelation coefficient x(k) and y(k) satisfies:

∙ −1 ≤ 𝜌xy ≤ 1∙ If x(k) and y(k) are uncorrelated then 𝜌xy = 0.∙ 𝜌xy = 1 if and only if y(k) = ax(k) + b where a > 0.∙ 𝜌xy = −1 if and only if y(k) = −ax(k) + b where a > 0.

Correlation coefficient 𝜌xy is a measure of linear dependence between the signals x(k) and y(k):𝜌xy = 1indicates a positive (increasing) linear relationship, 𝜌xy = −1 indicates decreasing (negative) linearrelationship, and some value between −1 and 1 indicates a lesser degree of linear dependence betweenthe variables. If 𝜌xy approaches zero, there is less dependence, and the closer the coefficient is to either−1 or 1, the stronger the dependence between the variables.

2.3 Spectral Density: Correlation in the Frequency DomainIn practice, the frequency domain representation of a correlation function plays an important role. Thecorrelation of two waveforms is essentially a measure of the degree of overlap in their spectral content.Generally, when the data record is long, the correlation can be computed faster when implemented in thefrequency domain rather than directly in the time domain. This is due mainly to the fast DFT computationusing the FFT algorithm. Frequency domain implementation requires O

(N log2 N

)while time domain

requires O(N2

)arithmetic operations.

If x(k) and y(k) are two signals, then the Discrete Time Fourier Transform (DTFT) of theircross-correlation function is termed the cross-energy spectral density; it is denoted as Pxy (f ) anddefined as:

Pxy(f ) = limN→∞

N∑m=−N

rxy(m)e−j2𝜋fm (2.59)

The definition of spectral density is the same for the energy or the power signals. The thing that changeswith signal type (energy or power) is the expression for the cross-correlation function rxy(m). If thesignals are energy signals, then the spectral density is termed energy spectral density, and if they arepower signals, then the spectral density is termed power spectral density.

Rayleigh TheoremThe energy of the signal, denoted Ex, may be expressed in terms of the energy spectral density

Ex =∞∑

k=−∞|x(k)|2 =

1

∫0

|X(f )|2 df (2.60)


If X(𝓁) is a discrete Fourier transform of x(k) then

N−1∑k=0

x2(k) = 1N

N−1∑𝓁=0

|X(𝓁)|2 (2.61)

Comment In many cases, it is simpler to obtain the correlation function by first obtaining Rxy (z) =ℑ{rxy(m)}, the z-domain equivalent of the power spectral density, and then using the inverse z-transformto obtain the correlation function rxy(m). An example demonstrating this approach will be provided later.

It is convenient to express the energy and power spectral densities in terms of the complex-valuedz-transform variable, z, instead of directly in terms of the Fourier transform variable, f; the use of thevariable, z, emphasizes the discrete-time nature of the associated time-domain. Once the z-transform ofa function is found, the DTFT may be obtained by evaluating the z-transform on the unit circle (i.e.,replacing z by z = ej2𝜋f ). Since z = ej2𝜋f is a periodic function of f (with period equal to 1), so will bethe DTFT, and thus the DTFT need only be specified on an interval of length 1 (e.g., −1 ≤ f ≤ 1 or−0.5 ≤ f ≤ 0.5); this is in contrast to the continuous-time Fourier transform which must be specified forall frequencies. We have expressed the DTFT in terms of z, 𝜔, or f as all of arguments are commonlyemployed.

If x(k), y(k), or both are periodic, then their cross-correlation will also be periodic with this period. Wegenerally don’t talk about the DTFT of periodic functions because they are not absolutely summable;however, we can define a generalized DTFT for periodic functions which will consist of a sum of deltafunctions. For periodic signals, the power spectrum is more commonly defined as a line spectrum wherethe power at each frequency is the squared magnitude of the associated Fourier Series Coefficient. Thegeneralized DTFT simply replaces the delta functions in the power spectrum by delta Functions thathave area equal to the power at a particular frequency.

Figure 2.1 shows the complex vector, ej𝜔, sweeping out a unit circle in the z-plane as the frequencyvariable, omega, varies between -pi −𝜋 to 𝜋. The frequency, 𝜔, is the angle of the vector ej𝜔 as measuredin the counter-clockwise direction from the positive real axis.

ω

je ω

0

/ 2π

π

Real axis

Imaginary axis

2ππ−

/ 2π−

z-planeUnit circle

Figure 2.1 Z-plane, unit circle, and frequency


Similarly, ej2𝜋f sweeps out the unit circle in the z-plane as f varies from 0 to 1; the value of f associatedwith any given point on the unit circle indicates the fraction of the way around the unit circle (whenstarting at the positive real axis and traveling the circle in the counter-clockwise direction) to the point’slocation. For example, the point where the unit circle intercepts the positive imaginary axis correspondsto a frequency f = 1∕4, the point where the unit circle intercepts the negative real axis corresponds tof = 1∕2, the point where the unit circle intercepts the negative imaginary axis corresponds to a frequencyf = 3∕4, which is also equivalent to a frequency of f = −1∕4. Recall that the DTFT, when expressed asa function of the discrete-time frequency variable, f, is periodic with period 1; hence the DTFT will havethe same value at f = 3∕4 as it has at f = −1∕4. When characterizing the frequency content of a signal,it is easiest to focus on the interval −0.5 ≤ f ≤ 0.5. Signals whose frequency components are confinedto the interval 0 ≤ |f | ≤ 0.25 are considered low-pass signals, those with frequency content confinedto the interval 0.25 ≤ |f | ≤ 0.5 are considered high-pass signals, and those whose frequency content iscontained within the interval are considered band-pass signals.

In the following section, we will derive expressions for Rxy(z), the z-transform of the correlationfunction, rxy(m), in terms of the individual z-transforms, X(z) and Y(z). The cross-spectral density Pxy(z)can then be found by evaluating Rxy(z) on the unit circle.

2.3.1 Z-transform of the Correlation FunctionAs explained in the previous section, finding Rxy (z) is sometimes a convenient first step in finding thecross-correlation function and/or the cross-spectral density. In this section we start by defining Rxy (z) asthe z-transform of rxy(m). We then make use of various z-transform properties to express Rxy (z) in termsof X(z) and Y(z). The case of energy signals, power signals, and random signals are treated separatelydue to differences in the definition of rxy(m) for these cases.

Case 1 Energy signalsTheorem: Let x(k) and y(k) be energy signals whose correlation function is given by Eq. (2.1).

Rxy (z) = X (z) Y∗((z∗)−1) (2.62)

In the case of real-values signals:

Rxy (z) = X (z) Y(z−1) (2.63)

Proof: Since rxy(m) is the convolution of x(k) and y∗(−k) we may use the z-transforms properties to findthat:

Rxy(z) = X(z)Y∗((z∗)−1) (2.64)

In the case of real-valued signals rxy(m) is the convolution of x(k) and y(−k) and hence we get:

Rxy(z) = X(x)Y(z−1) (2.65)

Case 2 Power signalsTheorem: Let x(k) and y(k) be deterministic power signals whose correlation function rxy(m) is given byEq. (2.11). Then the z-transform of the correlation function is denoted Rxy(z) and given by:

Rxy (z) = limN→∞

( 12N + 1

XN (z) Y∗N((z∗)−1)

)(2.66)


where XN(z) =N∑

k=−Nx(k)z−k and YN(z) =

N∑k=−N

y(k)z−k are truncated z-transforms of x(k) and y(k) respec-

tively. In the case of real signals:

Rxy (z) = limN→∞

( 12N + 1

XN (z) YN(z−1))

(2.67)

Proof: In the case of deterministic power signals, the cross-correlation, rxy(m), is given by Eq. (2.3).Taking the z-transform yields:

Rxy(z) = limN→∞

12N + 1

∞∑m=−∞

(N∑

k=−N

x(k)y(k − m)

)z−m (2.68)

Let xN(k) and yN(k) denote truncated versions of x(k) and y(k) as defined below:

xN(k) ={

x(k) −N ≤ k ≤ N0 else

yN(k) ={

y(k) −N ≤ k ≤ N0 else

Using the truncated signals, the expression for Rxy (z) becomes

Rxy(z) = limN→∞

12N + 1

N∑m=−N

( ∞∑k=−∞

xN(k)yN(k − m)

)z−m (2.69)

Since the summation term inside the bracket is the convolution of x(k) and y(−k) we get

Rxy(z) = limN→∞

12N + 1

XN (z) YN(z−1) (2.70)

For the case of complex-valued signals we get

Rxy(z) = limN→∞

12N + 1

XN (z) Y∗N((z∗)−1) (2.71)

Case 3 Random signalsLet x(k) and y(k) be jointly WSS processes. Then assuming that the cross-correlation rxy(m) is jointlysummable we may find

Rxy(z) = limN→∞

12N + 1

E[XN (z) YN(z−1)] (2.72)

Where similar to case 2, XN (z) =N∑

k=−Nx(k)z−k and YN

(z−1

)=

N∑k=−N

y(k)zk. If the random signal is jointly

WSS and ergodic in correlation, then Rxy(z) may be computed from a single realization of the jointprocess:

Rxy(z) = limN→∞

XN (z) YN(z−1)

2N + 1(2.73)


For complex-values random signals we get

Rxy(z) = limN→∞

XN (z) Y∗N((z∗)−1)

2N + 1(2.74)

Using Eqs. (2.63), (2.67), and (2.72), the z-transform of the correlation Rxy(z) for the real-valued energy,power, and the random signals may be expressed compactly as:

Rxy(z) =

⎧⎪⎪⎪⎨⎪⎪⎪⎩

X (z) Y(z−1) energy signal

limN→∞

(XN (z) YN(z−1)

2N + 1

)power signal

limN→∞

(E[XN (z) YN(z−1)]

2N + 1

)random signal

(2.75)

Comment Regardless of whether a signal is classified as an energy signal, a deterministic powersignal, or a random power signal the expression for and the properties of the correlation functionare essentially the same; the main difference is that the summation operation required to compute thecorrelation of the energy signal is replaced by an averaging operation when computing the correlationof the power signal.

For pedagogical reason, correlation functions of energy signals are often analyzed to conceptual under-standing of the spectral density.

In practice, it is impractical to acquire an infinite number of samples and, thus, the power signals aretreated as energy signals (i.e., truncated) for the purpose of computing their correlations and spectraldensities. In order for the estimate the correlation/spectral density to be accurate, the record length mustbe larger than the period of the signal’s lowest frequency, and the sampling period must be smaller thanthe half period of the signal’s highest frequency component.

2.3.2 Expressions for Energy and Power Spectral DensitiesAs previously stated, the cross-spectral density Pxy (f ) is defined as the DTFT of Rxy(z) on the unit circlez = ej2𝜋f :

Pxy (f ) = Rxy(ej2𝜋f ) (2.76)

In the case that x(k) and y(k) are real-valued energy signals, it was shown that Rxy(z) = X(z)Y(z−1

)and

hence,

Pxy (f ) = Rxy(ej2𝜋f ) = X(ej2𝜋f )Y(e−j2𝜋f ) (2.77)

Furthermore, letting X (f ) and Y (f ) denote the DTFTs x(k) and y(k) so that X (f ) = X(ej2𝜋f

)and Y (f ) =

Y(ej2𝜋f

), we may rewrite the previous equation as:

Pxy (f ) = X (f ) Y (−f ) (2.78)

Taking the inverse DTFT of the previous relationship reveals the familiar relationship that

Y(−f ) = Y∗(f ) (2.79)


In the case that x(k) and y(k) are complex-valued energy signals, Rxy(z) = X(z)Y((z∗)−1) and hence

Pxy (f ) = Rxy(ej2𝜋f ) = X(ej2𝜋f )Y∗(e−j2𝜋f ) (2.80)

Letting Xf (f ) and Yf (f ) denote the DTFTs x(k) and y(k) we may rewrite the previous equation as:

Pxy (f ) = X (f ) Y∗ (f ) (2.81)

Note that the order of the subscripts is important. The DTFT of the signal is associated with the secondsubscript whose DTFT is conjugated in the expression for the cross-spectral density. Taking the inverseDTFT of the previous relationship yields the previously discovered relationship that shows that thecross-correlation function rxy(k) can be expressed as the convolution of x(k) and y∗(−k),

rxy(k) = conv (x(k), y∗(−k)) (2.82)

where in finding the inverse DTFT of the relation above, we used the DTFT property that multiplicationin frequency is equivalent to convolution in time as well as the DTFT property: y∗(−k) ⇔ Y∗(f ).

Energy and Energy Spectral DensityIf x(k) is an energy signal, its energy spectral density is Pxx (f ) and is defined as the DTFT of the signal’sauto-correlation function. Just as the auto-correlation function is a special case of the cross-correlationfunction, the energy spectral density is a special case of the cross-spectral density. Setting y(k) = x(k) inEq. (2.81), we find:

Pxx (f ) = X (f ) X∗ (f ) (2.83)

The energy spectral density, Pxx (f ) shows how the energy of the signal, x(k), is distributed over frequency.According to Rayleigh’s theorem, the energy, Ex, of the signal x(k) can be found either in the time

domain or in the frequency domain, as shown by the Eq. (2.60). Note that the sum in the middle expressionabove is rxx(0) and the integrand on the rightmost expression is the energy spectral density, Pxx (f ). Hencethe equation above may be rewritten as:

Ex = rxx(0) =

1

∫0

Pxx (f ) df (2.84)

This last relationship illustrates a well-known property of DTFT pairs, namely that the value of thediscrete-time signal at sample no. 0 is equal to the area under its DTFT on the interval [ 0 1 ].

Power and Power Spectral DensityThe development above will hold similarly for power signals. The same limiting and averaging operationsthat were used to define Rxx(z) in terms of the individual z-transforms may be used to express Pxx (f ) interms of the individual DTFTs.

In the case of periodic signals, the power spectrum will be non-zero only at harmonically relateddiscrete frequencies. In particular, if x(k) is periodic with period N, it may be expressed in terms of itsdiscrete-time Fourier series expansion as:

x(k) =N−1∑n=0

cnej2𝜋nk∕N (2.85)

where cn =1N

N−1∑k=0

x(k)e−j2𝜋kn∕N .


Similar to Rayleigh’s theorem for energy signals, the average power,Px, of the signal may be expressedin terms of the discrete-time Fourier series coefficients as:

Px = rxx(0) = 1N

N−1∑k=0

x(k)x∗(k) =N−1∑k=0

cnc∗n (2.86)

Despite the fact that the power spectrum of a periodic signal is a line spectrum (i.e., discrete in frequency),use of the delta function allows it to be represented as a continuous-time spectrum. For example, notingthat the coefficients {cn} are periodic with period N so that cn+N = cn and that the coefficients cn isassociated with discrete-time frequency f = n

N, we may define the power spectral density, Pxx (f ), as

Pxx (f ) =∞∑

n=−∞cnc∗n 𝛿

(f − n

N

)(2.87)

With Pxx (f ) defined this way, we note that the expressions for the average power Px of the periodic signalx(k) may be written as:

Px = rxx(0) =

1

∫0

Pxx (f ) df (2.88)

As the Discrete Fourier Transform (DFT) algorithm is often used as a frequency analysis tool, it is usefulto know that the discrete-time Fourier series coefficients are identically equal to a scaled-version of theN-point DFT; in particular, if {X (n)} denotes the N-point DFT of the N-periodic signal, x(k), then theFourier series coefficients, {cn}, may be expressed in terms of the DFT coefficients as:

cn = 1N

X (n) (2.89)

We may thus rewrite the expressions for the average power of the periodic signal x(k) as:

Px =1N

N−1∑k=0

x (k) x∗ (k) = 1N2

N−1∑n=0

X(n)X∗(n) (2.90)

Summarizing expressions for energy Ex for the energy signal, the average power Px for power signalsand the average power for periodic (power) signals may be expressed compactly as:

rxx(0) =

1

∫0

Pxx (f ) df =

⎧⎪⎪⎨⎪⎪⎩Ex energy signalPx power signalN−1∑k=0

cnc∗n periodic signal(2.91)

Cross-Spectral Density and Degree of Similarity/DissimilarityTwo signals are said to be uncorrelated if there is no spectral overlap between them (i.e., if Pxy (f ) = 0)for all f; otherwise, we say they are correlated.

Pxy(f ) ={

0 for all f uncorrelated≠ 0 for some f correlated

(2.92)

For example, two sinusoids of different frequency are uncorrelated since there is no overlap in theirspectra (whereever one spectrum has a non-zero value, the other has a value of 0).


0 200

0.2

0.4

0.6

0.8

1

Exponential

Time

0.2 0.40

2

4

6

Frequency

0 20

-1

-0.5

0

0.5

1

Sinusoid

Time

0.2 0.40

5

10

15

Frequency

0 20

0

0.2

0.4

0.6

Damped sinusoid

Time

0.2 0.40

0.5

1

1.5

2

2.5

Frequency

0 20

-1

-0.5

0

0.5

1

Square

Time

0.2 0.40

5

10

15

20

Frequency

0 20

-4

-2

0

2

4

Triangular

Time

0 0.50

20

40

60

Frequency

Figure 2.2 Correlated signals

For example, a sine wave, a square wave, and a triangular wave with identical fundamental frequencyf0, are correlated since their spectra have all have non-zero values at f = f0.

Figure 2.2 shows examples of correlated signals. Note that the time domain signals (a decayingexponential, a sinusoid, a damped sinusoid, a square wave, and a triangular wave) are shown along thetop row with the corresponding spectra shown in the bottom row. Since all spectra have a non-zero valueat f = 1∕16, the signals are said to be correlated.

Figure 2.3 shows examples of uncorrelated signals; the signals include a discrete-time sinusoid withfundamental frequency f0 = 1∕32, a square wave with fundamental frequency f0 = 1∕16, and a triangularwave with fundamental frequency f0 = 1∕8. Since the even harmonics of both the square wave and thetriangular wave are all equal to zero, and since the square and triangular waves do not have any oddharmonics in common (i.e., there are no odd integers m and k such that m/16 = k/8), there is no frequencyfor which the spectra of the square and triangular waves both have non-zero values. Furthermore, thespectra of the square and triangular waves both have a value of zero at f=1/16 which is the only valuefor which the spectrum of the sinusoid is non-zero. Hence there is no overlap between the spectra of thesignals shown in Figure 2.3.

Figure 2.3 shows examples of uncorrelated signals. A sinusoid, a square wave, and a triangular waveare shown at the top while their spectra are shown at the bottom. Note that there is no frequency wherethere is a spectral overlap.

A normalized measure of correlation, termed coherence, is introduced in the next section.

2.4 Coherence SpectrumThe coherence spectrum provides a frequency-dependent measure of similarity (or dissimilarity) betweentwo stationary random signals and is derived from the normalized cross-spectral density. The coherence


0 20

-1

-0.5

0

0.5

1

Sinusoid

Time

0.05 0.1 0.15 0.20

5

10

15

Frequency

0 20

-1

-0.5

0

0.5

1

Square

Time

0.05 0.1 0.15 0.20

5

10

15

20

Frequency

0 20

-20

0

20

Triangular

Time

0.05 0.1 0.15 0.20

100

200

300

400

Frequency

Figure 2.3 Uncorrelated signals

is a measure of the correlation between two signals, at each frequency. In this work the coherence spectralmeasure is defined as the cosine-squared of an angle between the Fourier transforms of the two signals.This measure is used widely, including by industries that specialize in test and measurement devices formonitoring the status of a system.

The normalized cross-spectral density of signals x(k) and y(k), denoted PNxy (f ), is given by

PNxy (f ) =

Pxy (f )√Pxx (f ) Pyy (f )

=X(f )Y∗(f )|X(f )| |Y(f )| (2.93)

The coherence spectrum, denoted Ch (x, y), is derived from the normalized cross-spectral density PNxy (f )

so that it is geared for applications including fault diagnosis and performance monitoring [1], It isdefined as:

Ch (x, y) =|real (X (f ) Y∗ (f ))|2

|X (f )|2 |Y(f )|2(2.94)

Expressing X(f ) = Xr(f ) + jXi(f ), Y(f ) = Yr(f ) + jYi(f ) where Xr(f ) = real {X(f )}, Xi(f ) = image {X(f )},Yr(f ) = real {Y(f )}; Yi(f ) = image {Y(f )}, we get:

Ch(x, y) =(Xr(f )Yr(f ) + Xi(f )Yi(f ))2

(X2r (f ) + X2

i (f ))(Y2r (f ) + Y2

i (f ))(2.95)

It is shown in the Appendix that the expression for the coherence spectrum given by Eq. (2.94) is equalto the cos2 (𝜑) where 𝜑 is the angle between X(f ) and Y(f ) as shown in Figure 2.4:

Ch (x, y) = cos2 (𝜑) (2.96)


Real

Imaginary

rX

iX

rY

iY

X

Y

φ

Figure 2.4 Angle between two vectors X and Y

In view of Eq. (2.96), the coherence spectrum will be non-negative and less than or equal to unity:

0 ≤ Ch (x, y) ≤ 1 (2.97)

Measure of Similarity or Dissimilarity between Two SignalsThe coherence spectrum Ch (x, y) is a normalized measure of linear correlation between two signals x(k)and y(k). It is essentially the degree of spectral overlap between spectra X(f ) and Y(f ). The coherencespectrum of two signals will be unity at all frequencies, that is, Ch (x, y) = 1 for all f ∈ [0 0.5] if x(k) andy(k) are identical or collinear, that is, y(k) = 𝛼x(k) (or equivalently Y(f ) = 𝛼X(f )) where 𝛼 is a scalingfactor.

If X(f ) and Y(f ) are not identical or collinear, then the coherence spectrum will be less than unity atsome frequencies

Ch (x, y) < 1 for some or all f (2.98)

The coherence spectrum is widely used in many areas, including medical diagnosis, performance moni-toring, and fault diagnosis [1–3]. Further, it can detect if the measurement is contaminated by correlatednoise or external signal sources, additional inputs present in the system, or system nonlinearity. If thespectral bandwidth of interest is known, then only coherence over the frequency of interest is consideredand the rest of the spectral regions are ignored as they contain noise and other artifacts.

2.5 Illustrative Examples in Correlation and Spectral Density2.5.1 Deterministic Signals: Correlation and Spectral Density

Auto-Correlation

Example 2.1 ConstantCompute the auto-correlation function of s(k) = A, k ∈ (−∞ +∞ )

Solution: Since s(k) is a power signal, its auto-correlation function is given by

rss(m) = lim itN→∞

{1

(2N + 1)

N∑n=−N

s(k)s(k − m) = 1(2N + 1)

N∑k=−N

A2

}= A2


Hence the auto-correlation of a constant is the square of the constant. Furthermore, since rss(m) = A2

the power spectral density is the Fourier transform of the auto-correlation function, we find the powerspectral density of the constant A is a delta function with area A2 positioned at f = 0.

Pss(f ) = A2𝛿 (f )

Example 2.2 SinusoidFind the auto-correlation function of

s(k) = A sin(2𝜋f0k + 𝜑), k ∈ (−∞ +∞), f0 = 1∕M

where s(k) = s(k + M).

Solution: Since s(k) is M-periodic (i.e., s(k) = s(k + M)), its auto-correlation function:

rss(m) = 1M

M−1∑k=0

s(k)s(k − m) = 1M

M−1∑k=0

A2 sin(2𝜋f0k + 𝜙) sin(2𝜋f0(k − m) + 𝜙)

Using the identity 2 sinA sinB = cos (A − B) − cos (A − B) we find:

rss(m) = 12M

M−1∑k=0

A2 cos(2𝜋f0m

)− 1

2M

M−1∑k=0

A2 cos[2𝜋f0 (2k − m) + 2𝜑]

Note that the second term on the right is zero since it is a sum of a sinusoid over two periods. Furthermorethe summand of the first term is constant (does not depend upon the time k). Hence

rss(m) = A2

2cos(2𝜋fom)

This example illustrates that the auto-correlation of a sinusoid of frequency f0, amplitude A, andarbitrary phase 𝜑, s(k) = A sin(2𝜋f0k + 𝜑) is a sinusoid of the same frequency, f0, and ampli-tude A2; the phase information of the original sinusoid is not preserved by the auto-correlationfunction,rss(m) = (A2∕2) cos(2𝜋fom), as per the properties of the auto-correlation function, the maxi-mum value of A2 will occur at a lag of 0. Taking the Fourier transform of rxx(m), we find the powerspectral density of a sinusoid with frequency f0 and amplitude A is:

Pss(f ) = A2

4[𝛿(f − f0) + 𝛿(f + f0)]

Example 2.3 ExponentialDetermine the auto-correlation of s(k) where

s(k) ={

𝜌k 0 ≤ k < ∞0 k < 0

where |𝜌| < 1.


Solution: The signal s(k) is an energy signal. The auto-correlation rss(m) is given by

rss(m) =∞∑

k=0

s(k)s(k − m)

We will use the frequency-domain approach. The energy spectral density is given by

Ess(z) = S (z) S(z−1)

where

S (z) = 11 − 𝜌z−1

Substituting for S(z) and S(z−1) we get

Ess(z) = 1(1 − 𝜌z−1) (1 − 𝜌z)

Using partial fraction expansion we get

Ess(z) =𝛼0

1 − 𝜌z−1+

𝛼0

1 − 𝜌z+ 𝛽0

where 𝛼0 and 𝛽0 are the coefficients of the expansion. The equation relating energy spectral density andits partial fraction expansion becomes

1(1 − 𝜌z−1) (1 − 𝜌z)

=𝛼0

1 − 𝜌z−1+

𝛼0

1 − 𝜌z+ 𝛽0

Note that the first term on the right is the z-transform of rss(m) for m ≥ 0, and the sum of the second andthe third term is the z-transform of rss(m) for m < 0. Simplifying yields

1 = 𝛼0 − 𝛼0𝜌z + 𝛼0 − 𝛼0𝜌z−1 + 𝛽0 − 𝛽0𝜌z − 𝛽0𝜌z−1 + 𝛽0𝜌2

Collecting terms in z−1, z, and constant, and equating them to zero, the unknowns 𝛼0 and 𝛽0 are

𝛼0 =1

1 − 𝜌2

𝛽0 = −𝛼0

Taking the inverse z-transform, the correlation function rss(m) becomes

rss(m) = 𝜌|m|1 − 𝜌2

Figure 2.5 shows the exponential signal s(k), the magnitude of its Fourier transform |S(f )|, the auto-correlation rss(m), and the energy spectral density Ess(f ) = |S(f )|2.

Example 2.4 Damped sinusoidLet us compute the auto-correlation of a damped exponential signal which is defined for ( 0 ∞ ) by

s(k) = 𝜌k sin 2𝜋f0k


0 10 20 300

0.2

0.4

0.6

0.8

1

Time

Exponential signal

-20 0 200

1

2

3

4

5

Time

Auto correlation

0 0.1 0.2 0.3 0.40

2

4

6

8

10

Frequency

Fourier transform of the signal

0 0.1 0.2 0.3 0.40

20

40

60

80

100

Frequency

Spectral density

Figure 2.5 An exponential signal, its Fourier transform, auto-correlation, and spectral density

Solution: It is easier to use the frequency-domain rather than the time-domain approach to obtain thecorrelation.

Ess(z) = S (z) S(z−1)

where

S (z) =b1z−1

Ds(z); Ds(z) = 1 + a1z−1 + a2z−2; a1 = −2𝜌 cos 𝜔0; a2 = 𝜌2; b1 = 𝜌 sin 𝜔0

Substituting for S(z) and S(z−1) we get

Ess(z) =𝜌2b2

1

(1 + a1z−1 + a2z−2)(1 + a1z + a2z2)

Using the partial fraction expansion given in the Appendix we get

b21

Ds(z)Ds(z−1)=

𝛼1z−1 + 𝛼0

Ds(z)+

𝛼1z + 𝛼0

Ds(z−1)+ 𝛽0

where 𝛼1 𝛼0 and 𝛽0 are the coefficients of partial expansion; the first term on the right is the z-transformof rss(m) for m ≥ 0, and the sum of the second and the third term is the z-transform of rss(m) for m < 0.The partial fraction coefficients 𝛼1, 𝛼0, and 𝛽0 are determined from[

b21

0

]=

[2a1 1 − a2

1 − a22

1 + a2 −a1a2

] [𝛼1

𝛼0

]𝛽0 = −𝛼0


0 10 20 30 40

-0.2

0

0.2

0.4

0.6

Signal

Time

-40 -20 0 20 40

-1

0

1

2

Auto-correlation of the signal

Time

0 0.1 0.2 0.3 0.40

1

2

3

4

DTFT of signal

Frequency

0 0.1 0.2 0.3 0.40

5

10

15

20

Frequency

Spectral density

Figure 2.6 A damped sinusoid, its Fourier transform, and auto-correlation spectral density

Taking the inverse z-transform of Ess(z) yields the auto-correlation function rss(m) of s(k). Thus theauto-correlation of a damped sinusoid s(k) = 𝜌k sin 2𝜋f0k is a damped sinusoid given by

rss(m) = 𝛼0𝜌|m| cos 2𝜋f0m

The correlation does not preserve the phase information of the sinusoid. Figure 2.6 shows the exponentialsignal s(k), the magnitude of its Fourier transform |S(f )|, the auto-correlation rss(m), and energy spectraldensity Pss(f ) = |S(f )|2.

Cross-correlation

Example 2.5 Cross-correlation of two sinusoidsCompute the cross-correlation of two sinusoids s1(k) and s2(k) of two different frequencies f1 and f2, andphases 𝜑1 and 𝜑2 which are defined for k ∈ (−∞ ∞ )

s1(k) = sin(2𝜋f1k + 𝜑1)s2(k) = sin(2𝜋f2k + 𝜑2)

Solution: We will use frequency domain approach by computing the DTFTs of the two signals, TheDTFTs of s1(k) and s2(k) are

S1 (f ) =cos 𝜑1

2j[𝛿(f − f1) − 𝛿(f + f1)] +

sin 𝜑1

2[𝛿(f − f1) + 𝛿(f + f1)]

S2 (f ) =cos 𝜑2

2j[𝛿(f − f2) − 𝛿(f + f2)] +

sin 𝜑2

2[𝛿(f − f2) + 𝛿(f + f2)]


The cross-power spectral density P12 (f ) of s1(k) and s2(k) is

P12 (f ) = lim itN→∞

(SN1(f )S∗

N2(f )

N

)Since there is no overlap in the spectral frequencies S1(f ) and S2(f ) we get

P12 (f ) = 0

Thus the cross-spectral density, and hence the cross-correlation of two sinusoids with different frequen-cies, is zero.

Example 2.6 Sinusoid and square waveCompute the cross-correlation of a sinusoid s1(k) and symmetric square wave s2(k), which are definedfor k ∈ (−∞ ∞ ), and assuming that their periods are identical:

s1(k) = sin(2𝜋f0k)s2(k) = square(2𝜋f0k)

Solution: Expanding the symmetric square wave using Fourier series yields

s2(k) = 4𝜋

∞∑n=1,3,5,…

(1n

)sin 2𝜋nf0k

The cross-correlation of s1(k) and s2(k) is

r12(m) = 1M

M−1∑k=0

s1(k)s2(k − m) = 4𝜋M

M−1∑k=0

sin(2𝜋f0k)

( ∞∑n=1,3,5,…

(1n

)sin[2𝜋nf0(k − m)]

)

Since the cross-correlation of two sinusoids of different frequencies is zero, all product terms other thann = 1 are zero, yielding

r12(m) = 4𝜋M

M−1∑k=0

sin(2𝜋f0k) sin[2𝜋f0(k − m)]

Hence we get

r12(m) = 2𝜋cos 2𝜋f0m

The cross-spectral density P12(f ) is given by

P12(f ) = 2𝜋2

[𝛿(f − f0) + 𝛿(f + f0)]


Example 2.7 Waveform and its delayed versionCompute the cross-correlation of the two waveforms x(k) and y(k) where y(k) is a delayed version of x(k)and determine the time delay 𝜏 between the two rectangular pulse waveforms where ck = 1, for all k:

x(k) ={

ck 0 ≤ k ≤ N − 10 else

y(k) = x(k − 𝜏) ={

ck−𝜏 𝜏 ≤ k ≤ N − 1 + 𝜏

0 else

Solution: We will use the frequency-domain approach and consider a general case of a finite durationsignal, and then obtain the solution for the case of a rectangular pulse. The cross-correlation function isrelated to the auto-correlation function as follows

rxy(m) =N−1∑k=0

x(k)y(k − m) =N−1∑k=0

x(k)x(k − 𝜏 − m) = rxx (m + 𝜏)

Using the frequency-domain approach, we will compute rxy(m) by computing first rxx(m) and thenrxy(m) = rxx(m + 𝜏).

Computation of rxx(m)The energy spectral density Exx(z)

Exx(z) = X(z)X(z−1) =N−1∑i=0

N−1∑j=0

cicjz−i+j

Expanding using j = i + m, and since ci = 0 for i ≥ N and i < 0, we get

Exx(z) =N−1∑m=0

N−1−m∑i=0

cici+mz−m+N−1∑m=0

N−1−m∑i=0

cici+mzm −N−1∑i=0

c2i

We will consider the special case when x(k) is a rectangular pulse. In this case since ck = 1 for all k weget

Exx(z) =N−1∑m=0

N−1−m∑i=0

z−m+N−1∑m=0

N−1−m∑i=0

zm −N−1∑i=0

1

Simplifying we get

Exx(z) =N−1∑m=0

(N − m) z−m+N−1∑m=0

(N − m) zm − N

The first term on the right is the z-transform of rxx(m) for m ≥ 0, the second term is for m ≤ 0, and thelast term is for m = 0.

Taking the inverse z-transform the auto-correlation function rxx(m) becomes

rxx(m) ={

N − |m| 0 ≤ |m| ≤ N − 10 |m| ≥ N


0 5 100

0.2

0.4

0.6

0.8

1

Time

Pulse x

0 5 100

0.2

0.4

0.6

0.8

1

Time

Pulse y

-10 0 100

2

4

6

8

Lag index

Correlation

0 0.50

2

4

6

Frequency

DTFT of x

0 0.50

2

4

6

Frequency

DTFT of y

0 0.50

10

20

30

40

50

Frequency

Spectral density

Figure 2.7 A pulse, its delayed version, the cross-correlation, and their magnitude spectra

Hence the cross-correlation function rxy(m) = rxx(m + 𝜏) becomes

rxy(m) = rxx(m + 𝜏) ={

N − |m + 𝜏| 0 ≤ |m + 𝜏| ≤ N − 10 |m| ≥ N

The maximum of the cross-correlation occurs when m = −𝜏 and the maximum value is N.

The maximum value of the cross-correlation occurs at m = −𝜏 = −3, which is the time delay between thetwo waveforms. Figure 2.7 shows a pulse x(k) of duration N = 7 and its delayed version y(k) = x(k − 𝜏)where the delay 𝜏 = 3. The cross-correlation rxx(m) is maximum at m = −𝜏, indicating that the cross-correlation may be used to determine the time delay between two signals. The spectra of the signals andthe energy spectral density are shown at the bottom of the figure. Note that the cross-correlation of twopulse waveforms is a triangular pulse.

Example 2.8 Square wave and its delayed versionWe will use the frequency domain approach and consider a general case of an M-periodic wave form andthen obtain the solution for the case of a square wave. The z-transform of a M-periodic signal x(k) is

X (z) =

M−1∑i=0

ciz−i

1 − z−M


where the denominator 1 − z−M =∞∑

i=0z−M i generates periodic extension of the numerator term

M−1∑i=0

ciz−i.

Using the frequency domain approach, we will compute rxy(m) by computing first rxx(m) and thenrxy(m) = rxx(m + 𝜏).

Computation of rxx(m) = 1M

N−1∑k=0

x(k)x(k − m)

Since x(k) is a power signal, the power spectral density Pxx(z)

Pxx(z) = X(z)X(z−1) =

M−1∑m=0

M−1−m∑i=0

cici+mz−m

(1 − z−M) (1 − zM)

The expression for Pxx(z) indicates that

∙ The numerator term on the rightM−1∑m=0

M−1−m∑i=0

cici+mz−m is the auto-correlation of the square wave in the

interval [ 0 M − 1 ].∙ The denominator 1 − z−M indicates a periodic extension of the numerator for m ≥ 0.∙ The denominator 1 − zM indicates a periodic extension for m < 0.

Let us consider a square wave which has an odd symmetry given by

ci ={

1 0 ≤ i ≤ M1 − 1−1 M1 ≤ i ≤ M − 1

where M1 = M∕2. For this case it can be shown that the expression for the numerator term is

M−1∑m=0

M−1−m∑i=0

cici+mz−m =M−1∑m=0

M − 4 |m|: − M1 ≤ m ≤ M1 − 1

Taking the inverse z-transform of Pxx(z) we get

rxx(m) ={

N − 4 |m| M1 ≤ m ≤ M1 − 1rxx(m + 𝓁M) 𝓁 = ±1,±2,±3,… ,

That is, the auto-correlation of an odd symmetric square wave is an M-periodic triangular wave. Hencethe cross-correlation rxy(m) = rxx(m + 𝜏) of a square wave x(k) and its delayed version y(k) = x(k − 𝜏) is

rxy(m) ={

N − 4 |m + 𝜏| M1 ≤ m + 𝜏 ≤ M1 − 1rxy(m + 𝓁M) 𝓁 = ±1,±2,±3,… ,

Figure 2.8 shows a square wave, its delayed version, and their cross-correlation on the top and theirspectra at the bottom. The cross-correlation rxx(m) is maximum at m = −𝜏, indicating that the cross-correlation may be used to determine the time delay between two signals. Further the cross-correlationof two square waves is a triangular wave.


0 20-1

-0.5

0

0.5

1

Time

Square wave x

0 20-1

-0.5

0

0.5

1

Time

Square wave y

-5 0 5 10 15

-10

0

10

Lag index

Correlation

0 0.50

5

10

15

20

25

Frequency

DTFT of x

0 0.50

5

10

15

20

25

Frequency

DTFT of y

0 0.50

100

200

300

400

500

Frequency

Spectral density

Figure 2.8 A square wave, its delayed version, their cross-correlation, and their spectra

Coherence Spectrum

Example 2.9 Two damped sinusoidsLet x(k), and y(k) be signals defined for k ∈ ( 0 ∞ ) given by

x(k) = 𝜌k1 sin 𝜔1k

y(k) = 𝜌k2 sin 𝜔2k

Compute the coherence spectrum of x(k) and y(k).

Solution:

X(z) =𝜌1z−1 sin 𝜔1

1 − 2𝜌1z−1 cos 𝜔1 + 𝜌21z−2

Y(z) =𝜌2z−1 sin 𝜔2

1 − 2𝜌2z−1 cos 𝜔2 + 𝜌22z−2

Figure 2.9 shows the damped sinusoidal signals x(k) and y(k), auto-correlation functions rxx(m) andryy(m), and their energy spectral densities Exx(f ) and Eyy(f ).

The coherence spectrum Ch(x, y) and the energy spectral density Exy(f ) are shown in Figure 2.10. Thetwo signals differ in the frequency interval (0 0.2), since Ch(x, y) < 1, and are identical over the rest.


0 10 20 30 40 50

-0.5

0

0.5

Signal x

0 10 20 30 40 50

0

0.2

0.4

Signal y

-20 0 20

-1

0

1

2

Correlation of x

-20 0 20

0

1

2Correlation of y

0 0.1 0.2 0.3 0.40

10

20

Spectral density of x

Frequency

0 0.1 0.2 0.3 0.40

10

20

Spectral density of y

Frequency

Figure 2.9 Signals, auto-correlations, and energy spectral densities

-30 -20 -10 0 10 20 30

-0.2

0

0.2

0.4

0.6

Cross-correlation of x and y

Time lag index

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.2

0.4

0.6

0.8

1Coherence spectrum of x and y

Frequency

Figure 2.10 The cross-correlation and the coherence spectrum of the signals


2.5.2 Random Signals: Correlation and Spectral Density

Example 2.10 Noisy signalsLet x0(k) and x(k) be signals and v(k) be a zero-mean white noise process corrupting x(k) defined fork ∈ (−∞ ∞ ):

x0(k) = sin2𝜋f0k

x(k) = x0(k) + v(k)

where v(k) is a zero-mean white noise process with variance 𝜎2v . Using the property that the auto-

correlation of a sum of two uncorrelated signal is the sum of their auto-correlations, as shown in Eq.(2.41), the auto-correlation of the noisy signal is the sum of the auto-correlation of the noise-free signaland the auto-correlation of the white noise given by

rxx(m) = rx0x0(m) + rvv(m)

Since v(k) is a white noise, its auto-correlation is

rvv(m) = 𝜎2v 𝛿(m)

The auto-correlation of the noisy signal is the auto-correlation of the noise free signal except at zero lag.At zero lag the auto-correlation is the sum of auto-correlation of the noise free signal at zero lag and thevariance of the white noise:

rxx(m) ={

rx0x0(0) + 𝜎2

v m = 0rx0x0

(m) m ≠ 0

Figure 2.11 shows the noise free signal x0(k), noisy signal x(k), the noise v(k), their auto-correlationsrx0x0

(m), rxx(m), and rvv(m) respectively, and their spectral densities Px0x0(m), Pxx(m), and Pvv(m). From

the auto-correlation of the noisy signal, we can deduce the amplitude and the frequency (but not thephase of the sinusoid ) of the noise free sinusoidal signal x0(k), and the variance of the white noise. Thevariance of the noise 𝜎2

v is the difference between the correlation at zero lag rxx(0), and the amplitude ofthe sinusoid computed from rxx(m).

Signal Enhancing and Noise Attenuation PropertyThe signal enhancing and noise attenuating properties of correlation are widely used to determine(i) which of the known signals is present in a given noisy observation with applications includingdemodulation, fault isolation, and pattern classification, and (ii) estimated time delay between a noisysignal and a reference signal, with applications in system identification, biomedical signal processing,estimation of distance, and so on.

Example 2.11 Noisy signal with a known deterministic signalLet x0(k),x(k), and y(k) be signals and v(k) be a zero-mean white noise process corrupting x(k) definedfor k ∈ (−∞ ∞ ):

x0(k) = sin(2𝜋f0k + 𝜑)

x(k) = x0(k) + v(k)

y(k) = sin2𝜋f0k


0 50

-1

0

1

Signal

0 50

-2

0

2

Noisy signal

0 50

-2

-1

0

1

Noise

-50 0 50

-0.5

0

0.5

Auto-correlation

-50 0 50

-0.5

0

0.5

1Auto-correlation

-50 0 50

0

0.1

0.2

0.3

Auto-correlation

0 0.50

0.1

0.2

0.3Spectral density

Frequency

0 0.50

0.1

0.2

0.3

Spectral density

Frequency

0 0.50

0.05

0.1

Spectral density

Frequency

Figure 2.11 Signals, correlation, and power spectral densities

The signal y(k) is highly correlated with the signal x0(k) and uncorrelated with the noise v(k). Thecross-correlation rxy(m) of x(k) and y(k) will enhance the signal x0(k) buried in x(k) and attenuate thenoise v(k) corrupting x(k). The cross-correlation of x(k) and y(k) is given by

rxy(m) = rx0y(m) + rvy(m)

Since v(k) is uncorrelated with y(k) we get

rxy(m) = rx0y(m)

The cross-correlation rx0y(m) is the noise free sinusoid

rxy(m) = 12cos 2𝜋f0m

Figure 2.12 shows the noisy signal x(k), a signal y(k) which is matched to x0(k), the cross-correlationrxy(m) which is matched to the filtering operation, the spectrum X, the spectrum Y , and the cross-powerspectral density Pxy(f ).

It is clear from the figure that cross-correlating the noisy signal x(k) by a signal y(k) which ishighly correlated with x0(k) and uncorrelated with the noise v(k), produces a remarkable result. Thecross-correlated signal rxy(m), also termed the matched filter output, is noise free and contains all thecharacteristics of the signal x0(k) except for its phase.


0 50

-6

-4

-2

0

2

4

Signal

0 50

-1

-0.5

0

0.5

1

Noisy signal

-50 0 50

-0.5

0

0.5

Cross-correlation

0 0.50

0.1

0.2

0.3

Spectral density

Frequency

0 0.50

0.1

0.2

Spectral density

Frequency

0 0.50

0.1

0.2

0.3

Spectral density

Frequency

Figure 2.12 Matched filtering: a noisy signal and matching signal

Estimation of the Delay between Two Waveforms

Example 2.12 Two sinusoidsLet x(k), y0(k), and y(k) be signals and v(k) be a zero-mean white noise process corrupting y(k) definedfor k ∈ (−∞ ∞ ) given by

x(k) = sin2𝜋fk

y0(k) = sin2𝜋f0 (k − 𝜏)

y(k) = y0(k) + v(k)

Compute the time delay 𝜏 between the reference signal x(k) and the noisy signal y(k).

Solution: The reference signal x(k) is highly correlated with the signal y0(k) and is uncorrelated with thenoise v(k). Consider the cross-correlation rxy(m) of x(k) and y(k)

rxy(m) = rxy0(m) + rxv(m)

Since v(k) is uncorrelated with y(k) we get

rxy(m) = rxy0(m)


0 50

-1

-0.5

0

0.5

1

Signals x and y0

0 50

-4

-2

0

2

4

Signal x and y

-50 0 50

-0.4

-0.2

0

0.2

0.4

Correlation of x and y

0 0.50

0.05

0.1

0.15

0.2

0.25

Spectral density of x

Frequency

0 0.50

0.05

0.1

0.15

0.2

Spectral density of y

Frequency

0 0.50

0.05

0.1

0.15

0.2

Cross-spectral density

Frequency

Figure 2.13 Signals, cross correlation, and spectral densities

Since y0(k) is the delayed version of x(k), that is y0(k) = x(k − 𝜏), we get

rxy(m) = rxx(m − 𝜏)

Note that the time delay 𝜏 is the value of the time lag m where rxy(m) is maximum, that is 𝜏 = maxm

{rxy(m)}.

Figure 2.13 shows the reference signal x(k) and the noise-free signal y0(k), the noisy signal y(k) whichis a delayed and noise corrupted version of x(k), the cross-correlation rxy(m) of x(k) and y(k), the powerspectral density Pxx(f ) of x(k), the power spectral density Pyy(f ) of y(k), and the cross-power spectraldensity of Pxy(f ) of x(k) and y(k). Thanks to the noise attenuation property of the correlation operator,rxy(m) is practically a noise free sinusoid. An accurate estimate of the delay is given by the first maximumvalue of the correlation, namely 𝜏 = −20, in spite of y0(k) being totally buried in the noise v(k).

Comments The results of simulation verify the following properties of correlation/spectral density:

∙ Similarity properties:◦ A waveform with a rational spectrum preserves its characteristics.

� the auto-correlation of a constant is the square of the constant� an exponential is an exponential� a sinusoid is a sinusoid of the same frequency� an exponentially damped sinusoid is an exponentially damped sinusoid

◦ A periodic signal is a periodic signal.◦ A square waveform is a triangular waveform.◦ A triangular waveform is a parabolic waveform.


u yH

Figure 2.14 Input, output, and transfer function

∙ Dissimilarity properties:◦ Cross-correlation of a deterministic waveform and a zero-mean stochastic process is zero.◦ Cross-correlation function of two waveforms with no spectral overlap is zero. For example. periodic

waveforms of different periods are zero.◦ Cross-correlation function of stochastic processes, x(k) and y(k), is zero if the E [X(f )Y∗(f )] = 0.◦ Cross-correlation of periodic waveforms of different periods are zero.

The properties of similarity/dissimilarity hinge upon the amount of spectral overlap between the twowaveforms.

2.6 Input–Output Correlation and Spectral DensityIn many applications one is faced with the problem of determining the cross-correlation between twowaveforms that are related by some transfer function H, as shown in Figure 2.14 with an input u and anoutput y.

Input and Output Correlations and Spectral DensitiesConsider a linear time invariant and asymptotically stable system whose impulse response is h(k) andtransfer function is H(z), and with input u(k) and output y(k). It is assumed that the impulse response iscausal, that is

h(k) = 0 for k < 0 (2.99)

The output y(k) is a convolution of the input u(k) and the impulse response h(k) given by

y(k) =k∑

i=0

h(i)u(k − i) (2.100)

Taking the z-transform we get

Y(z) = H(z)U(z) (2.101)

Input–output cross-correlation ruy(m) = E [u(k)y(k − m)] is given by

ruy(m) = E

[u(k)

(k∑

i=0

h(i)u(k − i − m)

)](2.102)


Table 2.1 z-transforms of input and output spectral densities

u(k) y(k)

u(k) Puu(z) H(z−1

)Puu(z)

y(k) H(z)Puu(z) H(z)H(z−1

)Puu(z)

Note: The system H(z) is causal whereas the system H(z−1

)is anti-causal.

The impulse response h(k) of H(z) is zero for k < 0, while the impulseresponse of H

(z−1

)is h(−k) which is zero for k > 0. Expressing in terms

of the frequency variable f using z = ej2𝜋f we get.

Similarly cross-correlation ryu(m) = E[y(k)u(k − m)] becomes

ryu(m) = E

[(k∑

i=0

h(i)u(k − i)

)u(k − m)

](2.103)

Using Eq. (2.101), the cross-spectral density Puy(z) = E[U(z)Y(z−1)] becomes

Puy(z) = H(z−1)E[U(z)X(z−1)] = H(z−1)Puu(z) (2.104)

Similarly Pyu(z) = E[Y(z)U(z−1)] and Pyy(z) = E[Y(z)Y(z−1)] are given by

Pyu(z) = H(z)E[U(z)U(z−1)] = H(z)Puu(z) (2.105)

Pyy(z) = E[H(z)U(z)H(z−1)U(z−1)] = H(z)H(z−1)Puu(z) (2.106)

Tables 2.1 and 2.2 give the spectral densities relating all four combinations of U and Y in z-transformand Fourier transform for a general case when the input signal u(k) may be an energy, a power, a finiteduration, or a random signal by appropriately defining spectral density Puu(z).

2.6.1 Generation of Random Signal from White NoiseApplications of the expressions of spectral densities given by Eqs. (2.105) and (2.106) include systemidentification and modeling of a random signal as an output of a non-minimum phase linear system withwhite noise input. Let us choose the input signal to be a zero-mean white noise process v(k) with flatpower spectral density Pvv(z) = 𝜎2

v . Setting u(k) = v(k) and substituting Pvv(z) = 𝜎2v in Eqs. (2.105) and

(2.106) we get

Pyv(z) = 𝜎2v H(z) (2.107)

Pyy(z) = 𝜎2v H(z)H(z−1) (2.108)

Table 2.2 Fourier transforms of input and output spectral densities

u(k) y(k)

u(k) Puu(f ) H∗ (f ) Puu(f )y(k) H(f )Puu(f ) |H(f )|2 Puu(f )


Expressing in terms of the frequency f by setting z = ej2𝜋f , we get

Pyv(f ) = 𝜎2v H(f ) (2.109)

Pyy(f ) = 𝜎2v |H(f )|2 (2.110)

Let us express the power spectral density in terms of the coefficients of the transfer function

H(z) =

nb∑i=0

biz−i

na∑i=0

aiz−i

(2.111)

where ai and bi are the denominator and the numerator coefficients with a0 = 1, and na is the order ofthe system while nb is order of the numerator. The power spectral density becomes

Pyy(z) = 𝜎2v H(z)H(z−1) = 𝜎2

v

nb∑i=0

biz−i

nb∑j=0

bjzj

na∑i=0

aiz−ina∑i=0

ajzj

= 𝜎2v

nb∑i=0

nb∑j=0

bibjz−(i−j)

na∑i=0

na∑j=0

aiajz−(i−j)

(2.112)

Defining m = i − j and simplifying we get

Pyy(z) = 𝜎2v

nb∑m=0

nb−m∑i=0

bibi+m (z−m + zm)

na∑m=0

na−m∑i=0

aiai+m (z−m + zm)

(2.113)

The spectral density Pyy(z) is a function of z−m + zm, which implies

Pyy(z) = Pyy

(z−1

)(2.114)

This shows that the poles and zeros of the Pyy(z) have reciprocal symmetry: if zi is a zero, then 1∕zi is also azero, and if pi is a pole, then 1∕pi is also a pole. Substituting z = ej2𝜋f and noting that z−m + zm = 2cos2𝜋mfwe get

Pyy(f ) = 𝜎2v

nb∑i=0

b2i + 2

nb∑m=0

nb−m∑i=0

bibi+m cos 2𝜋mf

na∑i=0

a2i + 2

na∑m=0

na−m∑i=0

aiai+m cos 2𝜋mf

(2.115)

The spectral density Pyy(f ) expressed in terms of frequency f is a function of cos 2𝜋mf .

2.6.2 Identification of Non-Parametric Model of a SystemA model of the system H(z) may be modeled in (i) parametric form, as given Eq. (2.111) where thetransfer function is expressed in the terms of the numerator coefficients {bi} and denominator coefficients{ai}, the numerator order nb and model order na; or (ii) non-parametric form expressed in terms of itsfrequency response H(𝜔) (or equivalently by its impulse response h(k)).


In practice, one is faced with the problem of identification using a single realization of the input–outputdata record. The expression for the transfer function H(z) of a system given by Eq. (2.105) provides asolution:

H(z) =Pyx(z)

Puu(z)(2.116)

Since there are only a single realizations of U(z) and Y(z), the ensemble average (expectation operation)of a variable is replaced by the variable itself. Using the expression for spectral density in terms of U(z)and Y(z) we get

H(z) =Y (z) U

(z−1

)X (z) U

(z−1

) = Y (z)U (z)

(2.117)

Expressing in terms of the frequency f by setting z = ej2𝜋f , we get

H(f ) =Y (f )U (f )

(2.118)

Thus one can identify a non-parametric model H(𝜔) by (i) computing the Fourier transforms of the inputU(f ) and that of the output Y(f ) and dividing Y(f ) by U(f ). The impulse response h(k) of the system maybe obtained by computing the inverse Fourier transform of U(f ). Let a random signal y(k) be modeled asan output of a linear system H(z) driven by a zero-mean white noise with variance 𝜎2

v given by

Y(z) = H(z)V(z) (2.119)

where V(z) is the z-transform of the white noise process v(k).The cross-power spectral density of y(k)is Pyy(z) = 𝜎2

v H(z)H(z−1). In practice, however, the problem is to find a transfer function H(z) given thepower spectral density Pyy(z)(and not the other way round: given H(z) to find Pyy(z)). The solution to theproblem of determining H(z) given Pyy(z) is not unique. However, if we restrict ourselves to a class of allcausal “minimum phase transfer functions” then there is a unique solution. If the power spectral densityis given explicitly in the form Pyy(z) = 𝜎2

v H(z)H(z−1), a parametric model of H(z) may be computedusing a partial fraction of Pyy(z), as shown in Examples 2.3, 2.4, and 2.8. In practice, however, thepower spectral density has to be estimated from the data record. A popular approach is to employ anauto-correlation function ryy(m) of the output y(k) to estimate the parametric model by expressing H(z)in terms of the numerator coefficients {bi} and denominator coefficients

{ai

}, the numerator order nb

and model order na. The auto-correlation function ryy(m) may be computed by taking the inverse Fouriertransform Pyy (𝜔). In the case where the input to the system x(k) is a zero mean white noise process, theauto-correlation ryy(m) is merely the auto-correlation of the impulse response of the system h(k) scaledby 𝜎v– as can be deduced from Eq. (2.110). Identification of the parametric model of a system will beconsidered in a later chapter. In the next section, however, we will consider the identification of a signalmodel.

2.6.3 Identification of a Parametric Model of a Random SignalWe will consider all-pole (AR), pole-zero (ARMA), and all zero (MA) models and obtain a set ofequations relating the coefficients of the model and the auto-correlation function of the random signalsfrom which the coefficients may be estimated. First an AR, then ARMA, and finally an MA model areconsidered.


Identification of AR CoefficientsIn many applications in modeling of colored noise (or disturbance), it is generally assumed that thecolored noise is generated by filtering the white noise by an all-pole filter, that is the numerator of thefilter transfer function is a constant. Consider the signal model given by Eq. (2.119) where the H(z) is anall-pole transfer function given by

H(z) = Y(z)V(z)

= 11 + a1z−1 + a2z−2 +⋯ + ana

z−na(2.120)

Expressing in the time domain we get

y(k) +na∑i=1

aiy(k − i) = v(k) (2.121)

The model is generally called an Auto-Regressive (AR) model of a signal. We will derive the modelgoverning the auto-correlation of y(k) from the above equation. Multiply both sides of the equation byy(k − m) for m ≥ 0, and take expectation. We get

E [y(k)y(k − m)] +na∑i=1

aiE [y(k − i)y(k − m)] = E [v(k)y(k − m)] (2.122)

Using the definition ryy(m) = E [y(k)y(k − m)] and rvy(m) = E [v(k)y(k − m)] we get

ryy(m) +na∑i=1

airyy(m − i) = rvy(m) (2.123)

Consider the right-hand side of the above equation. The expression for rvy(m) is

rvy(m) ={

E [v(k)y(k)] m = 0E [v(k)y(k − m)] m > 0

(2.124)

From Eq. (2.121), we deduce that the past values of y(k) do not depend upon the present value of v(k),that is E [v(k)y(k − m)] = 0 for m > 0. Hence Eq. (2.124) becomes

rvy(m) ={

E [v(k)y(k)] m = 00 m > 0

(2.125)

Substituting for y(k) using Eq. (2.121) we get

rvy(m) =⎧⎪⎨⎪⎩

E [v(k)y(k)] = −na∑i=1

aiE [v(k)y(k − i)] + E[v2(k)

]m = 0

0 m > 0(2.126)

As E [v(k)y(k − i)] = 0 for i > 0 we get

rvy(m) ={

𝜎2v m = 0

0 m > 0(2.127)


Using Eq. (2.127), Eq. (2.123) becomes

ryy(m) +na∑i=1

airyy(m − i) ={

𝜎2v m = 0

0 m > 0(2.128)

Since ryy(m) = ryy(−m) only m ≥ 0 is considered. Given the auto-correlation functions {ryy(m)}, we maynow estimate the coefficients {ai} using the above equation. Given values m = 1, 2,… na (consideringonly values of m ≥ 1) Eq. (2.128) may be expressed as a linear matrix equation given by

⎡⎢⎢⎢⎢⎢⎣

ryy(0) ryy(1) ryy(2) . ryy(na − 1)ryy(1) ryy(0) ryy(1) . ryy(na − 2)ryy(2) ryy(1) ryy(0) . ryy(na − 3)

. . . . .

ryy(na) ryy(na − 1) ryy(na − 2) . ryy(0)

⎤⎥⎥⎥⎥⎥⎦

⎡⎢⎢⎢⎢⎢⎣

a1

a2

a3

.

ana

⎤⎥⎥⎥⎥⎥⎦= −

⎡⎢⎢⎢⎢⎢⎣

ryy(1)ryy(1)ryy(1)

.

ryy(na)

⎤⎥⎥⎥⎥⎥⎦(2.129)

The linear matrix equation relating the AR model parameters is called the Yule–Walker equation. Thematrix formed of the auto-correlation functions {ryy(m)} is a Toeplitz matrix. The matrix has a specialstructure. The elements along the main and the off diagonals are all equal. It is a constant-diagonalmatrix.

Coefficients of an ARMA ModelUnlike the AR case, a frequency domain rather than a time domain approach is employed to relate theARMA coefficients and the auto-correlation functions, as frequency domain approach is simpler. TheARMA model of the random signal is

y(k) +na∑i=1

aiy(k − i) =nb∑i=0

biv(k − i) (2.130)

Consider the signal model given by Eq. (2.119) where the H(z) is given by

H(z) = Y(z)V(z)

=b0 + b1z−1 + b2z−2 +⋯ + bnb

z−nb

1 + a1z−1 + a2z−2 +⋯ + anaz−na

(2.131)

Cross-multiplying yields

D(z)Y(z) = N(z)V(z) (2.132)

where N(z) = b0 + b1z−1 + b2z−2 +⋯ + bnbz−nb and D(z) = 1 + a1z−1 + a2z−2 +⋯ + ana

z−na . Multiply-

ing by Y(z−1

)on both sides we get

D(z)Y(z)Y(z−1) = N(z)V(z)Y(z−1) (2.133)

Substituting for Y(z−1) using Eq. (2.131) and taking expectation yields

D(z)E[Y(z)Y(z−1)] = N(z)H(z−1)E[V(z)V(z−1)] (2.134)


Taking inverse z-transform we get

ryy(m) +∑

airyy(m − i) = 𝜎2v

nb∑i=0

big(m − i) (2.135)

Since g(m) = h(−m)(where h(m) is the impulse response of H(z)) is anti-causal, g(m) = 0 for m < 0, weget

ryy(m) +na∑i=1

airyy(m − i) =⎧⎪⎨⎪⎩

𝜎2v

nb∑i=0

big(m − i) 0 ≤ m ≤ nb

0 m > nb

(2.136)

Since ryy(m) = ryy(−m) only m ≥ 0 is considered.

Coefficients of an MA ModelAs for the ARMA model, a frequency domain rather than a time domain approach is employed to relatethe MA coefficients and the auto-correlation functions. The MA model of the random signal is

y(k) =nb∑i=0

biv(k − i) (2.137)

The signal model H(z) is given by

H(z) = Y(z)V(z)

= b0 + b1z−1 + b2z−2 +⋯ + bnbz−nb (2.138)

Cross-multiplying and taking expectation gives

Y(z) = N(z)V(z) (2.139)

Substituting for Y(z−1

)using Eq. (2.138) and taking expectation yields

E[Y(z)Y(z−1)] = N(z)N(z−1)E[V(z)V(z−1)] (2.140)

Taking the inverse z-transform and using the Appendix to compute N(z)N(z−1

)we get

ryy(m) =⎧⎪⎨⎪⎩

𝜎2v

nb∑i=m

bibi−m 0 ≤ m ≤ nb

0 m > nb

ryy(−m) m < 0

(2.141)

Comments

∙ Similarity and noise annihilation properties.In all cases – namely the AR, the MA, and the ARMA models – the auto-correlation function ryy(m)generated as an output of the linear system H(z) driven by a zero-mean white noise, and a deterministic


signal y(k) generated as an output of the same system H(z) driven by a delta function, satisfies anidentical homogenous difference equation: the auto-correlation ryy(m) satisfies

ryy(m) +na∑i=1

airyy(m − i) = 0, m > nb

The deterministic signal y(k) satisfies

y(k) +na∑i=1

aiy(k − i) = 0, k > nb

Since the auto-correlation function ryy(m) and y(k) satisfy an identical homogenous difference equa-tion, they both belong to the same class of signals, such as the class of polynomials, exponentials,sinusoids, weighted sinusoids, or their combinations. For example if y(k) belongs to the class of expo-nential, then its auto-correlation ryy(m) will also belong to the same class of exponentials. Further, ifa signal s(k) is buried in a zero-mean white noise v(k), then the auto-correlation ryy(m) of the noisysignal y(k) = s(k) + v(k) belongs to the class of noise-free signal s(k). This justifies the properties ofnoise annihilation and similarity properties of correlation in Section 2.2.2.

∙ The estimated of the AR model using the Yule–alker equation is guaranteed to be stable and, thanksto Toeplitz structure of the correlation matrix, the Yule–Walker equation can be solved in O

(n2

a

)operations instead of O

(n3

a

). Hence it is used widely to estimate the model of a random as well as

a deterministic signal, for example, speech processing, interference cancellation, echo cancellation,and identification.

∙ The MA coefficients of the ARMA and MA models are nonlinear in the coefficients of the coefficientsof the transfer function H(z).

2.7 Illustrative Examples: Modeling and Identification

Example 2.13 Random signalsModel a random signal y(k) as an output of an AR model excited by a zero-mean unity variance whitenoise. Assuming the order of the AR model to be (i) a first order (ii) a second order, and (ii) a third order,determine the AR coefficients for each case.

Solution:

∙ Consider an example of a first-order AR model of random signal

y(k) = a1y(k − 1) + v(k − 1)

The estimated a1 unknown coefficient a1 may be determined using Eq. (2.129).

ryy(1) = −a1ryy(0)

Simplifying we get

a1 = ryy(1)∕ryy(0)

∙ Consider an example of a second-order AR model of random signal

y(k) = a1y(k − 1) + a2y(k − 2) + v(k − 1)


Table 2.3 Estimated and true coefficients

True coefficients Estimated coefficients

First-order model 0.8 0.81Second-order model [−1.5588 0.8100] [−1.5597 0.8088]Third-order model [−2.3588 2.0571 − 0.6480] [−2.3353 2.0124 − 0.6244]

Estimates a1 and a2 of the unknown coefficients may be determined from[ryy(0) ryy(1)ryy(1) ryy(0)

] [a1

a2

]= −

[ryy(1)ryy(2)

]∙ Consider an example of a second-order AR model of random signal.

y(k) = a1y(k − 1) + a2y(k − 2) + a3y(k − 3) + v(k − 1)

Estimates a1, a2, and a3 of the unknown coefficients may be determined from Eq. (2.129)

⎡⎢⎢⎣ryy(0) ryy(1) ryy(2)ryy(1) r(0) ryy(1)ryy(2) ryy(1) r(0)

⎤⎥⎥⎦⎡⎢⎢⎣

a1

a2

a3

⎤⎥⎥⎦ = −⎡⎢⎢⎣

r(1)r(2)r(3)

⎤⎥⎥⎦Simulation ResultsThe coefficients of the first, second, and third were estimated by estimating the auto-correlation functionsof the random signals. Only a single realization of the N = 4096 data samples were employed incomputing the correlation estimates. The estimate and the true coefficients are given below in Table 2.3.

Examples of Modeling and Non-parametric IdentificationFigure 2.15 shows the generation of low-pass, band-pass, and high-pass random signals by filtering azero-mean white noise process using a low-pass filter, a band-pass filter, and a high-pass filter as givenin Eq. (2.119). The input u(k) is a zero-mean white noise process. The outputs of all the filters, and theirestimated auto-correlations and spectral densities, are shown. Since only a single realization of the datais employed, true auto-correlations and power spectral densities are also plotted for comparison. Usingthe Fourier transforms of the input and the output of each of the filters, the non-parametric frequencyresponse model is identified, and identified frequency responses are compared with the true ones. Thetop three figures show the outputs of low-pass, band-pass, and high-pass filters. The mid three figuresshow the estimated and true auto-correlation of the filter outputs, and the bottom three figures show theidentified and true non-parametric models.

Modeling of Random Signals

Example 2.14 Random signalsConsider a stochastic process y(k), which is generated using a zero-mean white noise process with unitvariance, v(k)

y(z) = H(z)𝜈(z) where H(z) = 1 − 0.3z−1

1 − 0.5z−1


0 20 40

0

0.2

0.4

0.6Low-pass

0 20 40

-0.5

0

0.5

Bandpass

0 20 40

-1

0

1

High pass

-40 -20 0 20 40

0

0.05

0.1

Correlation

-20 0 20

-0.1

0

0.1

0.2

Correlation

-10 0 10

-0.2

0

0.2

0.4

Correlation

0 0.2 0.4

0.20.40.60.8

11.2

Non-parametric model

0 0.2 0.40

0.5

1


0 0.2 0.40

0.5

1


True Estimate

Figure 2.15 Non-parametric modeling and identification of random signals

Obtain expressions for the power spectral densities Pyy(z), Pyv(z), Pvy(z), Pvv(z).

Solution:

∙ Pvv(z) = 𝜎2, Pyv(z) = 𝜎2H(z), Pvy(z) = 𝜎2H(z−1), Pyy(z) = 𝜎2H(z)H(z−1)Expanding the expression for Pyy(z) we get

Pyy(z) = 𝜎2

(1 − 0.3z−1

1 − 0.5z−1

)(1 − 0.3z1 − 0.5z

)Expanding we get

Pyy(z) = 𝜎2

(1 − 0.3

(z−1 + z

)+ 0.09

1 − 0.5(z−1 + z

)+ 0.25

)= 𝜎2

(1.09 − 0.3

(z−1 + z

)1.25 − 0.5

(z−1 + z

))

Substituting z = ej2𝜋f we get

Pyy(f ) = 𝜎2

(1.09 − 0.6 cos 2𝜋f

1.25 − cos 2𝜋f

)∙ The poles and the zeros of Pyy(z): poles: 0.5 and 2; zeros: 0.3 and 3.333.∙ The poles and zeros have reciprocal symmetrical: if p is pole then 1/p is also a pole, if z is a zero the

1/z is also a zero.


Example 2.15 Let y(k) be some random process generated as an output of H(z) excited by a zero-meanwhite noise process with unity variance. Compute the power spectral density Pyy(z) if

H(z) =b0 + b1z−1 + b2z−2

a0 + a1z−1 + a2z−2

where the leading coefficient in the denominator a0 = 1.

Solution: Power spectral density Pyy(z) takes the form

Pyy(z) = 𝜎2v H(z)H(z−1)

Substituting for H(z) yields

Pyy(z) = 𝜎2v

(b0 + b1z−1 + b2z−2

a0 + a1z−1 + a2z−2

)(b0 + b1z1 + b2z2

a0 + a1z1 + a2z2

)Simplifying we get

Pyy(z) = 𝜎2v

(b2

0 + b21 + b2

2

)+ (b0b1 + b1b2)(z + z−1) + b0b2(z2 + z−2)(

a20 + a2

1 + a22

)+ (a0a1 + a1a2)(z + z−1) + a0a2(z2 + z−2)

Substituting z = ej2𝜋f we get

z + z−1 = 2 cos 2𝜋f

z2 + z−2 = 2 cos 4𝜋f

Hence

Pyy(𝜔) = 𝜎2v

(b2

0 + b21 + b2

2

)+ 2(b0b1 + b1b2) cos 2𝜋f + 2b0b2cos4𝜋f(

a20 + a2

1 + a22

)+ 2(a0a1 + a1a2) cos 2𝜋f + 2a0a2 cos 4𝜋f

Example 2.16 Let x(k) and y(k) be two random signals modeled respectively as the outputs of G(z)and H(z) excited by zero-mean white noise processes with unit variance where

X(z) = G(z)v(z)

Y(z) = H(z)v(z)

where G(z) = z−1

1 − 𝜌1z−1; H(z) =

b1z−1

1 + a1z−1 + a2z−2; b1 = 𝜌2 sin 𝜔0; a1 = −2𝜌2 cos 𝜔0; a2 = 𝜌2

2; 𝜌1 = 0.8;

𝜌2 = 0.9 and 𝜔0 = 𝜋∕6. Determine the correlation functions rxx(m), ryy(m), and rxy(m), and the corre-sponding spectral densities Exx(f ), Eyy(f ), and Exy(f ).


Solution:

∙ The power spectral density of x(k) is

Exx(z) = 𝜎2G(z)G(z−1) = 1(1 − 𝜌1z−1)(1 − 𝜌1z)

Using the Appendix, the partial fraction expansion becomes

Exx (z) = X(z)X(z−1) = 11 − 𝜌2

1

(1

1 − 𝜌1z−1+ 1

1 − 𝜌1z− 1

)Taking the inverse z-transform, yields

rxx(m) =𝜌|m|1

1 − 𝜌21

∙ The spectral density of y(k) is

Eyy(z) = 𝜎2H(z)H(z−1) =b2

1

(1 + a1z−1 + a2z−2)(1 + a1z + a2z2)

Using the partial fraction expansion we get

Eyy(z) = 𝜎2H(z)H(z−1) =Ayyz

−1 + Byy

1 + a1z−1 + a2z−2+

Ayyz + Byy

1 + a1z + a2z2+ Cyy

Using the Appendix, the coefficients Ayy, Byy, and Cyy are obtained from[b2

10

]=

[2a1 1 − a2

1 − a22

1 + a2 −a1a2

] [Byy

Ayy

]and

Cxy = −Bxy

Taking the inverse z-transform we get

ryy(m) = Byy𝜌|m|2 cos 𝜔0m

∙ The spectral density of x(k) and y(k) is

Exy(z) = X(z)Y(z−1) = 𝜎2G(z)H(z−1)

Using the partial fraction expansion, we get

Exy(z) =Axyz

−1(1 − 𝜌1z−1

) +Bxyx + Cxy(

1 − 2𝜌2 cos 𝜔0z + 𝜌22z2

) + Dxy


Using the Appendix we get

Axy =b0𝜌1 + b1𝜌2

1

𝜌21a2 + 𝜌1a1 + 1

Bxy = b1 − Axya2

Cxy =Axy

𝜌1

Dxy = 0

Taking the inverse z-transform we get

rxy(m) ={

Axy𝜌m1 m ≥ 1

Cxy𝜌m2 cos 𝜔0m m ≤ 0

Simulation ResultFigure 2.16 shows random signals x(k) and y(k) generated using a linear time invariant model drivenby zero-mean white noise process. The auto-correlations rxx(m) and ryy(m), and cross-correlation rxy(m)are (i) estimated using a single realization of the random signal and (ii) computed analytically using themodel which is termed as “true.” Similarly, the spectral densities are computed. The top three figures

-50 0 50

0

1

2

Correlation of x

-20 0 20

-1

0

1

2

Correlation of y

-20 0 20

-0.5

0

0.5

1

1.5

Correlation of x and y

0 0.50

0.5

1Spectral density of x

Frequency

0 0.50

0.5

1Spectral density of y

Frequency

0 0.50

0.5

1Spectral density of x and y

Frequency

Signal Model

Figure 2.16 True and estimated correlations and spectral densities of x(k) and y(k)


give true and estimated correlations of x(k), y(k), and the cross-correlation of x(k) and y(k). The bottomthree figures give true and estimated spectral densities of x(k), y(k), and the cross-spectral density of x(k)and y(k).

System identification

Example 2.17 Estimate from the input and the output data, the non-parametric model in terms offrequency response of a system H(z) given by

H(z) = z−1

1 − 1.7387z−1 + 0.81z−2

Solution: The input is a chosen to be a zero-mean white noise process. The Fourier transforms of theinput and the output are computed and the frequency response is estimated from a single realization ofthe random input and the output. The estimate of the non-parametric model expressed in terms of thefrequency response is

H(f ) =Y (f )U (f )

The estimate and the true frequency responses are given in Figure 2.17.Spectral densities of input and noisy output of a system are shown in Figure 2.18, where u is the input,

s is the noise-free output, v is the measurement noise, and y is the noisy output.

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

0.2

0.4

0.6

Spectrum of input

Frequency

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

0.20.40.60.8

Spectrum of output

Frequency

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45

5101520

Estimate and the true frequency responses

Frequency

Magnitid

e

True

Estimate

Figure 2.17 Estimated and true frequency responses


u sH

v

y

Figure 2.18 Input, noise-free, and noisy output

Example 2.18 Consider the following stochastic process y(k) given by

y(k) = s(k) + v(k)

s(z) = H(z)u(z)

where u(k) is the input and v(k) is the zero-mean white noise process with variance 𝜎2v .

1. Derive the expressions for the power spectral densities Pyy(z), Pyv(z), Pvy(z), and Pvv(z).2. Derive the above expression if u(k) and v(k) are uncorrelated.3. Is it possible to estimate H(z) if u(k) and v(k) are uncorrelated?

Solution:

∙ Consider Pyy(z)Since y(k) = s(k) + v(k)

Pyy(z) = Pss(z) + Pvv(z) + Psv(z) + Pvs(z)

Consider Pss(z)

Pss(z) = H(z)H(z−1)Puu(z)

Consider Psv(z)

Psv(z) = s(z)v(z−1) = H(z)u(z)v(z−1) = H(z)Puv(z)

Consider Pvs(z)

Pvs(z) = v(z)s(z−1) = v(z)H(z−1)u(z−1) = H(z−1)Pvu(z)

Hence

Pyy(z) = H(z)H(z−1)Puu(z) + Pvv(z) + H(z)Puv(z) + H(z−1)Pvu(z)

∙ Consider Pyu(z)

Pyu(z) = Psu(z) + Pvu(z)

Since

Psu(z) = s(z)u(z−1) = H(z)u(z)u(z−1) = H(z)Puu(z)


Hence

Pyu(z) = H(z)Puu(z) + Pvu(z)

∙ Consider Puy(z)

Puy(z) = Pus(z) + Puv(z)

Since

Pus(z) = u(z)s(z−1) = H(z−1)u(z)u(z−1) = H(z−1)Puu(z)

Hence

Puy(z) = H(z−1)Puu(z) + Puv(z)

∙ Substituting Pvv(z) = 𝜎2v and assuming that u(k) and v(k) are uncorrelated, which implies Puv(z) = 0

and Pvu(z) = 0, we get∙ Pyy(z) = H(z)H(z−1)Puu(z) + 𝜎2

v∙ Pyu(z) = H(z)Puu(z)∙ Puy(z) = H(z−1)Puu(z)

Expressing in terms of the frequency 𝜔 we get

∙ Pyy(f ) = |H(f )|2 Puu(f ) + 𝜎2v∙ Pyu(f ) = H(f )Puu(f )

∙ Puy(f ) = H∗(f )Puu(f )∙ Estimation of H(z).

Consider the expression of Pyu(z)

Pyu(z) = H(z)Puu(z)

It is possible to identify H(z) knowing the power spectral densities Pyu(z) and Puu(z)

H(z) =Pyu(z)

Puu(z)

The transfer function is estimated by dividing the cross-power spectral density of y(k) and u(k) by theauto-power spectral density of u(k).

Identifying the Form of a Signal from a Noisy MeasurementThanks to the properties of similarity and noise annihilation, the correlation and spectral density arewidely used to determine the form (or class) of a signal buried in zero-mean uncorrelated noise.

Example 2.19 A random signal y(k) is a sum of a deterministic signal s(k) and a zero-mean whitenoise process v(k) given by

y(k) = s(k) + v(k)

∙ Give expressions for the auto-correlation function and the power spectral density of y(k).


0 50 100 150-5

0

5Signal y1

0 50 100 150

-20

0

20

Signal y2

-50 0 50

-0.50

0.51

1.5

Correlation of y1

-50 0 50 100

0

20

40

Correlation of y2

0 0.1 0.2 0.3 0.40

0.2

Spectral density of y1

Frequency

0 0.1 0.2 0.3 0.40

5Spectral density of y2

Frequency

1.5 41

0.251

1/41/8

1/32

42.25

1/32

160.5

Figure 2.19 Two random signals, auto-correlations, power spectral densities

∙ Consider the random signals y1(k) and y2(k) containing signals s1(k) and s2(k) respectively.

yi(k) = si(k) + vi(k), i = 1, 2

Figure 2.19 shows random signals y1(k) and y2(k). The top, the middle, and the bottom figures on theright give the plots of the output y1(k), its auto-correlation, and its spectra density, and those on the leftshow y2(k), its auto-correlation, and its spectral density. Determine (i) the form s1(k) and the variance𝜎2

1 of v1(k), and (ii) the form of s2(k) and the variance 𝜎22 . The relevant values of auto-correlations and

spectral densities are indicated.

Solution:

∙ The power spectral density of y(k) takes the form.

Pyy(z) = Pss(z) + Psv(z) + Pvs(z) + Pvv(z)

Since s(k) and v(k) are uncorrelated, implying Psv(z) = 0 and Pvs(z) = 0, we get

Pyy(z) = Pss(z) + Pvv(z)

Further, as v(k) is white noise we get

Pyy(z) = Pss(z) + 𝜎2

That is

Pyy(𝜔) = Pss(𝜔) + 𝜎2v

Taking the inverse transform we get

ryy(m) = rss(m) + 𝜎2v 𝛿(m)


Determination of form of the signal buried in the noise

Case 1 The top, middle, and bottom figures on the left of Figure 2.19.The auto-correlation function shown in the middle is a sinusoid of frequency f0 for |m| > 0.

rs1s1(m) = 𝛼2

2cos 2𝜋f0m : |m| > 0

Invoking the property of similarity of correlation, it can be deduced that the signal s1(k) buried in thenoise is also sinusoid is also sinusoid of frequency f0:

s1(k) = 𝛼 sin(2𝜋f0k + 𝜑

)We may cross-check our deduction by analyzing the power spectral density shown in the bottom figure.Since it is a line spectrum at f0 = 1∕32 given by

Py1y1(f ) =

𝛼21

4[𝛿(f − f0) + 𝛿(f + f0)]

this confirms that it is sinusoid. Let us now determine the amplitude of the sinusoid and the variance ofthe noise. From the auto-correlation function we get

ry1y1(0) = rs1s1

(0) + 𝜎21 =

𝛼21

2+ 𝜎2

1 = 1.5

as the amplitude of the auto-correlation, which is a sinusoid, is𝛼2

1

2= 0.5 we get

𝛼1 = 0.5

Hence

𝜎21 = 1

We cannot, however, determine the phase of the sinusoid.

Case 2 The top, the middle and the bottom figures on the right of Figure 2.19The auto-correlation function shown in the middle is periodic. Invoking the property of periodicity, itcan be deduced that the signal s2(k) is a periodic signal buried in the noise. It is, however, difficult todetermine additional characteristics of the signal. Let us analyze the power spectral density shown at thebottom. It is a line spectrum located at three frequencies 1∕32, 1∕8 and 1/4 with magnitude 4, 2.25 and1 respectively given by

Py2y2(f ) = 4

[𝛿

(f − 1

32

)+ 𝛿

(f + 1

32

)]+ 9

4

[𝛿

(f − 1

8

)+ 𝛿

(f − 1

8

)]+

[𝛿

(f − 1

4

)+ 𝛿

(f − 1

4

)]Hence the signal s2(k) is a sum of three sinusoids. The amplitudes of the sinusoids 𝛼1, 𝛼2, and 𝛼3 are

computed from the relations𝛼2

1

4= 4,

𝛼22

4= 9

4, and

𝛼23

4= 1. The general form of s2(k) is

s2(k) = 4 sin(2𝜋k∕32 + 𝜑1) + 3 sin(2𝜋k∕32 + 𝜑2) + sin(2𝜋k∕32 + 𝜑3)


Let us now determine variance of the noise. The auto-correlation of s2(k) is given by

rs2s2(m) = 8 cos (2𝜋m∕32) + 6 cos (2𝜋m∕32) + 2 cos (2𝜋m∕32)

Using the data indicated we get

ry2y2(0) = rs2s2

(0) + 𝜎22 = 8 + 6 + 2 + 𝜎2

2 = 16 + 𝜎22 = 41

Hence

𝜎22 = 25

Comments Correlations and the power spectral density provide a simple scheme to determine theform of the signal buried in uncorrelated noise thanks to the noise annihilation and similarity properties.In simple cases, such as a single sinusoid in zero-mean white noise, the auto-correlation function issufficient to determine the form.

In the presence of colored noise, it is comparatively difficult to determine the form of the signal unlessthe data record is very long.

2.8 Summary

Definitions of Correlation Function

rxy(m) =

⎧⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎩

limN→∞

N∑k=−N

x(k)y(k − m) infinite duration energy signals

N−1∑k=0

x(k)y(k − m) finite duration energy signals

limN→∞

12N + 1

N∑k=−N

x(k)y(k − m) power signal

rxy(m) = 1M

M−1∑k=0

x(k)y(k − m) M-periodic power signal

limN→∞

12N + 1

N∑k=−N

x(k)y(k − m) ergodic random (power) signals

Properties of Correlation Function∙ Correlation and convolution

rxy(m) =N∑

k=−N

x(k)y(− (m − k)) = conv (x(k), y(−k))

ℑ{rxy(m)} = X(f )Y∗(f ) = |X(f )| |Y(f )|∠(𝜑x(f ) − 𝜑y(f ))

ℑ {conv(x(k), y(k))} = X(f )Y(f ) = |X(f )| |Y(f )|∠(𝜑x(f ) + 𝜑y(f ))

∙ Cross-correlation functions rxy(m) and rxy(m):

rxy(m) = ryx(−m)


∙ Symmetry property of auto-correlation

rxx(m) = rxx(−m)

∙ Maximum values of auto- and cross-correlations

rxx(0) ≥ |rxx(m)|√rxx(0)ryy(0) ≥ |rxy(m)|

|rxy(m) + r∗xy(m)| ≤ rxx(0) + ryy(0)

∙ Time delay between two signals𝜏 = argmax

m{rxy(m)} for real signals

𝜏 = argmaxm

{ryx(m) + r∗yx(m)} for complex signals

∙ Additive property

ryy(m) =∑

i,j

rxixj(m) =

M∑i=1

rxixi(m) +

∑i≠j

rxixj(m)

ryy(m) =M∑

i=1

rxixi(m) if rxixj

(m) = 0 for i ≠ j

∙ Periodicity propertyIf x(k) and y(k) are periodic: x(k) = x(k + 𝓁M), then

rxy(m) = rxy(m + 𝓁M) : 𝓁 = ±1,±2,±3,…

∙ Noise annihilation property

E [s(k)v(k − m)] = 0

If x(k) = s(k) + v(k) then rxs(m) = rss(m)

Normalized Correlation and Correlation Coefficient

𝛾xy(m) =rxy (m)√

rxx (0) ryy (0)where − 1 ≤ 𝛾xy(m) ≤ 1

The correlation coefficient

𝜌xy =cxy (0)√

cxx (0) cyy (0)

The correlation coefficient x(k) and y(k) satisfies:

∙ −1 ≤ 𝜌xy ≤ 1∙ if x(k) and y(k) are uncorrelated then 𝜌xy = 0∙ 𝜌xy = 1 if and only if y(k) = ax(k) + b where a > 0∙ 𝜌xy = −1 if and only if y(k) = −ax(k) + b where a > 0.


Energy and Power Spectral Densities

Pxy(f ) = limN→∞

N∑m=−N

rxy(m)e−j2𝜋fm

Z-transform of the Correlation Function

Rxy(z) =

⎧⎪⎪⎪⎨⎪⎪⎪⎩

X (z) Y(z−1

)energy signal

limN→∞

(XN (z) YN

(z−1

)2N + 1

)power signal

limN→∞

(E

[XN (z) YN

(z−1

)]2N + 1

)random signal

Cross-Spectral Density

Pxy (f ) = X (f ) Y (−f )

Note that the order of the subscripts is important. The DTFT of the signal is associated with the secondsubscript whose DTFT is conjugated in the expression for the cross-spectral density.

Spectral Density

Pxx (f ) = |X (f )|2

Energy and Average Power

rxx(0) =

1

∫0

Pxx(f )df =

⎧⎪⎪⎨⎪⎪⎩Ex energy signalPx power signal

N−1∑k=0

cnc∗n periodic signal

Cross-spectral Density and Degree of Similarity/Dissimilarity

Pxy(f ) ={

0 for all f uncorrelated≠ 0 for some f correlated

Similarity properties:

∙ A waveform with a rational spectrum preserves its characteristics.∙ A periodic signal is a periodic signal.∙ A square waveform is a triangular waveform.∙ A triangular waveform is a parabolic waveform.

Dissimilarity properties:

∙ Cross-correlation of a deterministic waveform and a zero-mean stochastic process is zero.∙ Cross-correlation function of two waveforms with no spectral overlap is zero. For example periodic

waveforms of different periods are zero.


∙ Cross-correlation function of stochastic processes, x(k) and y(k), is zero if the E [X(f )Y∗(f )] = 0.∙ Cross-correlations of periodic waveforms of different periods are zero.

Coherence Spectrum

Ch(x, y) =|real(X(f )Y∗(f ))|2|X(f )|2|Y(f )|2

=(XrYr + XiYi)

2

(XrXr + XiXi)2 + (YrYr + YiYi)2

Ch (x, y) ≤ 1 for some or all f

Input–Output Correlation and Spectral DensityGeneration of random signal from white noise

Pyy(z) = 𝜎2v H(z)H(z−1) and Pyy(f ) = 𝜎2

v |H(f )|2

Identification of non-parametric model of a system

H(z) =Pyx(z)

Puu(z)

Identification of a parametric model of a random signal

∙ ARMA model

ryy(m) +na∑i=1

airyy(m − i) =⎧⎪⎨⎪⎩

𝜎2v

nb∑i=0

big(m − i) 0 ≤ m ≤ nb

0 m > nb

∙ AR model

⎡⎢⎢⎢⎢⎢⎣

ryy(0) ryy(1) ryy(2) . ryy(na − 1)ryy(1) ryy(0) ryy(1) . ryy(na − 2)ryy(2) ryy(1) ryy(0) . ryy(na − 3)

. . . . .

ryy(na) ryy(na − 1) ryy(na − 2) . ryy(0)

⎤⎥⎥⎥⎥⎥⎦

⎡⎢⎢⎢⎢⎢⎣

a1

a2

a3

.

ana

⎤⎥⎥⎥⎥⎥⎦= −

⎡⎢⎢⎢⎢⎢⎣

ryy(1)ryy(1)ryy(1)

.

ryy(na)

⎤⎥⎥⎥⎥⎥⎦∙ MA model

ryy(m) =⎧⎪⎨⎪⎩

𝜎2v

nb∑i=m

bibi−m 0 ≤ m ≤ nb

0 m > nb

ryy(−m) m < 0

2.9 Appendix

Formula for the Product of Two Sums

M∑i=1

xi

M∑j=1

xj =M∑

i=1

M∑j=1

xixj


Angle between Two Vectors

Proposition If X and Y are two complex functions X = Xr + jXi and Y = Yr + jYi, where Xr and Xi

are the real and the imaginary components of X, and Yr and Yi are the real and the imaginary componentsof Y . Then

cos2 𝜑 = |real {XY∗}|2|X|2 |Y|2=

(XrYr + XiYi)2(

X2r + X2

i

) (Y2

r + Y2i

) ,

where 𝜑 is the angle between X and Y∗.

Proof: Using [4]

cos 𝜑 = real (XY∗)|X| |Y|Expanding real{XY∗}, |X| and |Y| we get

real{XY∗} = real{(Xr + jXi)(Yr − jYi)} = XrYr + XiYi

|X| = √X2

r + X2i

|Y| = √Y2

r + Y2i

Thus the proposition is established.

Partial Fraction Expansion of Spectral DensityFirst-order systemConsider an energy spectral density Exx (z) = X(z)X(z−1) where

X(z) =b0

1 − 𝜌z−1

The energy spectral density becomes

Exx(z) = X(z)X(z−1) =b2

0

(1 − 𝜌z−1)(1 − 𝜌z)

The partial fraction expansion becomes

Exx(z) = X(z)X(z−1) = A1 − 𝜌z−1

+ A1 − 𝜌z

+ C

Cross-multiplying we get

b20 = A(1 − 𝜌z) + A(1 − 𝜌z−1) + C(1 − 𝜌z)(1 − 𝜌z−1)

Collecting terms in z and z−1 constant we get

b20 = 2A + C + C𝜌2 + (A𝜌 + C𝜌)z − (A𝜌 + C𝜌)z−1


Equating to zero the constant, and the coefficients of z and z−1, we get

C = −A

A =b2

0

(1 − 𝜌2)

Exx(z) = X(z)X(z−1) =b2

0

1 − 𝜌2

(1

1 − 𝜌z−1+ 1

1 − 𝜌z− 1

)Second-order systemConsider an energy spectral density Exx(z) = X(z)X(z−1) where

X(z) =b1z−1 + b0

1 + a1z−1 + a2z−2

The energy spectral density becomes

Exx(z) = X(z)X(z−1) =(b1z−1 + b0)(b1z + b0)

(1 + a1z−1 + a2z−2)(1 + a1z + a2z2)


Exx(z) = X(z)X(z−1) = Az−1 + B1 + a1z−1 + a2z−2

+ Az + B1 + a1z + a2z2

+ C


b21 + b2

0 + b1b0z + b1b0z−1 = (Az−1 + B)(1 + a1z + a2z2) + (Az + B)(1 + a1z−1 + a2z−2)

+C(1 + a1z + a2z2)(1 + a1z−1 + a2z−2)

Consider the terms on the right-hand side

(Az−1 + B)(1 + a1z + a2z2) = (Aa1 + B) + Az−1 + (Aa2 + Ba1)z + Ba2z2

(Az + B)(1 + a1z−1 + a2z−2) = (Aa1 + B) + Az + (Aa2 + Ba1)z−1 + Ba2z−2

(1 + a1z + a2z2)(1 + a1z−1 + a2z−2) =(1 + a2

1 + a22

)+ (a1 + a1a2)z−1 + (a1 + a1a2)z + a2z−2 + a1z2

The partial fraction expansion after cross-multiplication becomes

b21 + b2

0 + b1b0z + b1b0z−1 =[2(Aa1 + B) + C

(1 + a2

1 + a22

)]+ [A(1 + a2) + Ba1 + C(a1 + a1a2)]z−1

+ [A(1 + a2) + Ba1 + C(a1 + a1a2)]z + [Ba1 + Ca1]z−2 + [Ba2 + Ca2]z2

Setting the coefficients of zi, z−i, and the constant to zero we get

b21 + b2

0 = 2(Aa1 + B) + C(1 + a21 + a2

2)b1b0 = A(1 + a2) + Ba1 + C(a1 + a1a2)

Ba1 + Ca1 = 0Ba2 + Ca2 = 0


Hence [b2

0 + b21

b0b1

]=

[2a1 1 − a2

1 − a22

1 + a2 −a1a2

] [AB

]B = −C

Hence

Exx (z) = X(z)X(z−1) = Az−1 + B1 + a1z−1 + a2z−2

+ Az + B1 + a1z + a2z2

− B

Cross-spectral densityThe cross-spectral density is given by

Exy(z) = X(z)Y(z−1) =b0 + b1z

(1 − 𝜌1z−1)(1 + a1z + a2z2)


b0 + b1z

(1 − 𝜌1z−1)(1 + a1z + a2z2)= Az−1

1 − 𝜌1z−1+ Bz + C

1 + a1z + a2z2+ D


b1z + b0 = Az−1 + Aa1 + Aa2z + Bz + C − 𝜌1B − 𝜌1Cz−1

+D(1 + a1z + a2z2 − 𝜌1z−1 − a1𝜌1 − a2𝜌1z)

Collecting terms in powers of z we get

b1z + b0 = Aa1 + C − 𝜌1B + D − Da1𝜌1 + z(Aa2 + B + Da1 − Da2𝜌1)− z−1(A − 𝜌1C + 𝜌1D) + z2Da2

Equating the coefficients of the powers of z to zero yields

D = 0

A = 𝜌1C

Aa1 + C − B𝜌1 − b0 = 0

Aa2 + B − b1 = 0

Simplifying we get

A =b0𝜌1 + b1𝜌2

1

𝜌21a2 + 𝜌1a1 + 1

B = b1 − Aa2

C = A𝜌1



Exy(z) = Az−1

1 − 𝜌1z−1+ Bz + C

1 + a1z + a2z2

References

[1] Neve, A. (2012) Coherence Function: Theory and Visual Engineering Environment Applications. AgilentTechnologies: Education Corner.com Experiments.

[2] Binnie, C., Cooper, R., Mauguiere, F., et al. (eds) (2003) Clinical Neurophysiology: EEG,Paediatric Neuro-physiology, Special Techniques and Applications, Vol. 2, Elsevier Science BV, Amsterdam.

[3] Ropella, K., Sahakian, A., Baerman, J., and Swiryn, S. (1989) Coherence spectrum: a quantitative discriminatorof fibrillatory and non-fibrillatory cardiac rythms. Circulation: Journal of the American Heart Association, 80,112–119.

[4] Kay, S.M. (1998) Fundamentals of Signal Processing: Detection Theory, Prentice Hall, New Jersey.[5] Proakis, J.G. and Manolakis, D.G. (2007) Digital Signal Processing: Principles, Algorithms and Applications,

Pearson Prentice Hall, New Jersey.[6] Haykin, S. (2000) Communication Systems, John Wiley and Sons, New York.[7] Oppenheim, A.V. and Schafer, R.W. (2010) Discrete-Time Signal Processing, Prentice-Hall, New Jersey.[8] Kay, S.M. (1998) Fundamentals of Signal Processing: Detection Theory, Prentice Hall, New Jersey.[9] Ljung, L. (1999) System Identification: Theory for the User, Prentice-Hall, New Jersey.

[10] Kay, S.M. (1988) Modern Spectral Estimation: Theory and Application, Prentice Hall, New Jersey.[11] Mitra, S.K. (2006) Digital Signal Processing: A Computer-Based Approach, McGraw Hill Higher Education,

Boston.[12] Moon, T.K. and Stirling, W.C. (2000) Mathematical Methods and Algorithms for Signal Processing, Prentice

Hall, New Jersey.[13] Verhaegan, M. and Verdult, V. (2007) Filtering and System Identification, Cambridge University Press,

Cambridge.[14] Raol, J., Girija, G. and Singh, J. (2004) Modeling and Parameter Estimation, IEE Control Engineering Series

65, Institution of Electrical Engineers.

identification of physical systems (applications to condition monitoring, fault diagnosis, soft...

Documents