ece 531: detection and estimation theory course outline

ECE 531: Detection and Estimation Theory

Natasha Devroye

[email protected]

http://www.ece.uic.edu/~devroye

Spring 2010

Course outline

ECE 531: Detection and Estimation

University of Illinois at Chicago, ECE

Spring 2010

Instructor: Natasha Devroye, [email protected]

Course coordinates: Tuesday, Thursday from 2-3:15pm in TH 208 (Taft Hall).

Office hours: Monday from 4-5:30pm in SEO 1039, or by appointment

Welcome to ECE 531! This course is a graduate-level introduction to detection and estimation theory,

whose goal is to extract information from signals in noise. A solid background in probability and

some knowledge of signal processing is needed.

Course Textbook: Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by

Steven M. Kay, Prentice Hall, 1993 and (possibly) Fundamentals of Statistical Signal Processing,

Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.

Other useful references:

Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, II, III, IV

H. Vincent Poor, Introduction to Signal Detection and Estimation

Louis L. Scharf and Cedric Demeure, Statistical Signal Processing: Detection, Estimation, and Time

Series Analysis

Carl Helstrom, Elements of Signal Detection and Estimation. It's out of print, so here's my pdf copy.

Notes: I will follow the course textbooks fairly closely, using a mixture of slides (highlighting the

main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I

will post a pdf version of the slides as they become ready here, but the derivations will be given in

class only.

Topics: Estimation Theory:

General Minimum Variance Unbiased Estimation, Ch.2, 5

Cramer-Rao Lower Bound, Ch.3

Linear Models+Unbiased Estimators, Ch.4, 6

Maximum Likelihood Estimation, Ch.7

Least squares estimation, Ch.8

Bayesian Estimation, Ch.10-12

Detection Theory:

Statistical Detection Theory, Ch.3

Deterministic Signals, Ch.4

Random Signals, Ch.5

Statistical Detection Theory 2, Ch.6

Non-parametric and robust detection

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 =

max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

http://www.ece.uic.edu/~devroye/courses/ECE531/

1 of 3 1/11/10 8:50 PM

Fundamentals of Statistical Signal Processing, Volume 1: Estimation Theory, by Steven M. Kay, Prentice Hall, 1993

Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory, by Steven M. Kay, Prentice Hall 1998.



Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM

Problems - attack!

Problems - attack!

Problems - attack!

Problem 2.5In the binary communication system shown in Fig. 5-1, messages m = 0 and m = 1occur with a priori probabilities 1/4 and 3/4 respectively. Suppose that we observer ,

r = n + m,

where n is a continuous valued random variable with the pdf shown in Fig. 5-2. Therandom variable n is statistically independent of whether message m = 0 or m = 1occurs.

source

n

r+

mreceiver m̂

m=0 or 1 message

received signal

m=0 or 1 ^

Figure 5-1

p (n)n

n

-3/4 3/4

2/3

Figure 5-2

(a) Find the minimum probability of error detector, and compute the associatedprobability of error, Pr [m̂ != m].

(b) (practice) Suppose that the receiver does not know the a priori probabilities,so it decides to use a maximum likelihood (ML) detector. Find the ML detectorand the associated probability of error. Is the ML detector unique? Justify youranswer. If your answer is no, find a di!erent ML receiver and the associatedprobability of error.

Problem 2.6Consider the binary discrete-time communication system shown in Fig. 6-1. Assumethat m = 0 and m = 1 are equally likely to occur. Under hypothesis Hm (m = 0, 1),the received signal r [n] is given by

r [n] = ym[n] + w [n] = sm[n] " h[n] + w [n], for all n,

where “"” denotes convolution, and where the w [n]’s are zero-mean statistically in-dependent Gaussian random variables with variance !2. The two signals, s0[n] and

4

Problems - attack!

Massachusetts Institute of TechnologyDepartment of Electrical Engineering and Computer Science

6.432 Stochastic Processes, Detection and Estimation

Problem Set 6Spring 2004

Issued: Tuesday, March 16, 2004 Due: Thursday, April 1, 2004

Reading: This problem set: Sections 4.0-4.5 and Section 4.7 except 4.7.5Next: Sections 4.3-4.6, 4.A, 4.B, Chapter 5

Final Exam is on May 19, 2004, from 9:00am to 12:00 noon in WalkerMemorial (50-340). You are allowed to bring three 8 1

2

!!! 11!! sheets of notes (bothsides). If you have a conflict with this time and need to schedule an alternate time,you must see Prof. Willsky by April 6, 2004 at the absolute latest.

Problem 6.1Consider the estimation of a nonrandom but unknown parameter x from an observa-tion of the form y = x+w where w is a random variable. Two di!erent scenarios areconsidered in parts (a) and (b) below.

(a) Suppose w is a zero-mean Laplacian random variable. i.e.,

pw(w) =!

2e"!|w|

for some ! > 0. Does an unbiased estimate of x exist? Explain. Does an e"centestimate of x exist? If so, determine x̂e!(y). If not, explain.

Hint:

! #

"#w2!

2e"!|w| dw =

2

!2.

(b) Suppose w is a zero-mean random variable with probability density pw(w) > 0depicted in Figure 6-1. Does an e"cient estimate of x exist? If so, determinex̂e!(y). If not, explain.

pw(w )

0 1 2-1-2-3 3 w

Figure 6-1

1

Problems-attack!

3. Suppose that h, x, v are mutually independent random variables with

x ! N (3, 4), v ! N (0, 4)

and where h takes on only two possible values, 1 and 3 with

Pr(h = 1) = Pr(h = 3) = 1/2.

Letz = hx + v

(a) Determine the MMSE estimator of x given z and h (i.e. x̂MMSE).

(b) Determine the MMSE achieved by the above estimator.

4

Course outline



Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM





Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM

Estimation: General Minimum Variance Unbiased Estimation

• Bias: (expected value of estimator - true value of data)

• MVUE:

Estimation: Cramer-Rao lower bound

• Lower bound on variance of ANY unbiased estimator!

• Usage:

• assert whether an estimator is MVUE

• benchmark against which to measure the performance of

an unbiased estimator

• feasibility studies

• Depends on?

NatureTransmission /

measurementProcessing

Noise

The Cramer-Rao Lower Bound (CRLB)

CRLB examples

CRLB T or F

all the way to....

CRLB for General Gaussian Case

• When observations are Gaussian and one knows the dependence of the

mean and covariance matrix on the unknown parameters, we know the

closed form of the CRLB (or Fisher information matrix):

Finding MVUE so far

Finding the MVUE

Estimation: linear models

• What’s a linear model and why is it useful?

• What can be said?

• Best Linear Unbiased Estimators (BLUE)

MVUE for the linear model

Examples

Examples

General linear model

Re-do the derivation

Key idea: whiten the observations,

then apply previous results

• Gauss-Markov theorem for BLUEs:

BLUE - vector

Estimation: Maximum Likelihood Estimation

• Alternative to MVUE which is hard to find in general

• Easy to compute - very widely used and practical

• What is the MLE?

• Properties?

Properties of the MLE

Invariance property

Examples

MLE for linear model

Numerical methods

Estimation: Least Squares

• Alternative estimator with no general optimality properties, but nice and

intuitive and no probabilistic assumptions on data are made - only need a

signal model

• Advantages?

• Disadvantages?

Examples

Linear LSE: theory

Linear model without any noise assumptions!

Linear LSE vs other estimators

7

!"#$%&'()*+,-*.'(-%&*./0*+"*1+,-&*02+'#%+-2

3"4-5 02+'#%+-

-6!7 !"

# $ 7666!!!

"#

1ˆ %"

No Probability Model Needed

# $ 7666!!!

$"%&

1ˆ %"

86!7 !"

PDF Unknown, White

# $ 7666!!

'"

1ˆ %"&

86!7 !"

PDF Gaussian, White

# $ 7666!!!

'(%

1 ˆ %"

86!7 !"

PDF Gaussian, White

9:*;"<*

%22<#-*

=%<22'%(*>*

%$$5;*

+,-2-?*

@AB*;"<*

%&-*

CD1E=?*

;"<*%+*5-%2+*

)-+*+,-*

./0F

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm

Estimation: Bayesian Estimation

• Parameter to be estimated is assumed to be random, according to some prior

distribution which models our knowledge of it

• Bayesian Minimum Mean Squared Error (MMSE):

• Applications to Gaussian noise / linear model

Bayesian estimation on Gaussian priors/noise

This is crucial to know!

Bayesian estimation on Gaussian priors/noise

This is crucial to know!

Bayesian linear model

Examples: MMSE in Gaussian noise

Properties of MMSE

Estimation: Bayesian Estimation

• General risk functions - arbitrary “cost” functions

• Maximum a posteriori (MAP) estimation

• Linear MMSE: constrain estimator to be linear - very practical

MAP properties

Sequential LMMSE

Algebraic approach Geometric approach

LMMSE properties

Course outline



Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM





Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM

Problems - attack!ECE 531 Detection and Estimation - Final Exam Name:

5. The random variables X and Y are uniformly distributed over the trapezoidal region shown in the figurebelow, described by the equations

p(x, y) =!

1 |y| < 1/2, y ! 1/2 < x < y + 1/20 else

Next:

!X|Y (y) = E![x! x̂BLS(y)]2 | y

"=

# !

"!

$x! 2%

y + 1a

&'2

pX|Y (x | y) dx

=# !

0

$x! 2%

y + 1a

&'2

x

(y +

1a

)2

e"x(y+ 1a ) dx

=# !

0

*x3 ! 4x2 1

y + 1a

+ 4x1

%y + 1

a

&2

+ (y +

1a

)2

e"x(y+ 1a ) dx

= 3!(

y +1a

)2"4

! 4 · 2!(

y +1a

)2"1"3

+ 4 · 1!(

y +1a

)2"2"2

=2

(y + 1a )2

Finally:

!BLS = E,!X|Y (y)

-=

# !

"!!X|Y (y) pY (y) dy =

# !

0

2(y + 1

a )21a

1(y + 1

a )2dy =

2a

(!1

3

)(y +

1a

)"3.....

!

0

=2a2

3

(b) Now we want to compute x̂MAP (y). We’ve already done most of the work in part (a).

x̂MAP (y) = argmaxx

pX|Y (x | y) = argmaxx

(y +

1a

)2

xe"x(y+ 1a ) u(x) = argmax

xxe"x(y+ 1

a ), x " 0

Taking derivatives we find:

d

dx

/xe"x(y+ 1

a )0

= e"x(y+ 1a ) ! x(y +

1a)e"x(y+ 1

a ) = 0

=# x =1

y + 1a

Thus x̂MAP (y) = 1y+ 1

a. We really need to also verify that the second derivative is negative, so that we

have actually found a maximum. You can verify that:

d2

dx2

/x e"x(y+ 1

a )0....

x= 1y+ 1

a

= !2(y +1a)e"1 + (y +

1a)e"1 < 0

Problem 9.4 (Old Exam Question)The random variables X and Y are uniformly distributed over the region shown in the figure.

! " #$ ! " #$ ! !

!

"

# " $ ! % % $ & & ' !! " #

$ ! " #

(a) Find 1xBLS the Bayes least square estimate of X given Y , !X|Y the corresponding conditional covari-ance, and !BLS the corresponding mean square error.

5

(a) (5 points) Find the MMSE of X given Y and the corresponding minimum mean squared error.(b) (3 points) Find the Linear MMSE of X.(c) (2 points) Define the Quadratic MMSE as the estimator X̂ = aY 2 + bY + c where the constants

a, b, c, are chosen to minimize the mean squared error. Find the quadratic MMSE estimator of Xbased on Y and the corresponding MMSE. HINT: think first!

(d) (5 points) Find the MAP estimate of X based on Y .

Points earned: out of a possible 15 points

Problems - attack

• T/F: The performance of a detector in colored noise is always worse than that

of a detector in white noise.

• T/F: The Wiener filter is a Bayesian filter than optimizes the MMSE criterion.

• T/F: The Cramer-Rao bound seen in class only applies to unbiased

estimators.

• Explain the difference between MAP and ML estimators.

• Explain the difference between MAP and ML detectors.

• T/F: The sample mean is the MVUE for the DV level of any signal in noise.

Course outline



Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM





Spring 2010














Series Analysis





class only.








Detection Theory:









1 of 3 1/11/10 8:50 PM

Detection: Statistical Detection Theory

• Binary hypothesis testing

Detection: Statistical Detection Theory

Neyman-Pearson hypothesis testing

Useful problem 2.1

Bayesian risk

Multiple hypothesis testing

Detection: Deterministic Signals

• How to detect known signals in noise?

• The famous matched filter!

Xx[n]

T(x)

s[n]

• Generalized matched filter

• > 2 hypotheses

Performance of matched filter

Performance of generalized matched filters

Designing signals

Attack?

Be able to show this!

Multiple Deterministic Signals in Gaussian Noise

Let’s derive this decision rule!

Geometric meaning?

Detection: Random Signals

• What if s[n] is random?

• Key idea behind estimator-correlator:

• Linear model simplifies things again...

The problem

Example: Linear Model

Detection: Statistical Decision Theory II

• Uniformly most powerful test

• Generalized likelihood ratio test

• Bayesian approach

• Wald test

• Rao test

Further resources


ece 531: detection and estimation theory course outline

Documents