ece 6540, lecture 07ece6540/slides/2016_ece6540_class007.pdf · sufficient statistics in the end,...

Variations

ECE 6540, Lecture 07Cramer-Rao Lower Bounds

Last Time Rao-Blackwell-Lehmann-Scheffe Theorem

Complete Sufficient Statistics

2

Sufficient Statistics Rao-Blackwell-Lehmann-Scheffe Theorem

If �𝜽𝜽 is an unbiased estimator of 𝜽𝜽 and 𝑇𝑇 𝒙𝒙 is a sufficient statistic for 𝜽𝜽, then�𝜽𝜽 = E �𝜽𝜽|𝑇𝑇 𝒙𝒙 is

1) A valid estimator for 𝜽𝜽 (not dependent on 𝜽𝜽)2) Unbiased3) Of lesser or equal variance than that of �𝜽𝜽, for all 𝜽𝜽 (each element of �𝜽𝜽 has

lesser or equal variance)4) If the sufficient statistic is complete, then �𝜽𝜽 is the MVU estimator

3

𝑇𝑇 𝒙𝒙 represents more information

Sufficient Statistics A Complete Statistics A statistic is complete if there is only one function𝑔𝑔 ⋅ of the statistic t = 𝑇𝑇 𝑥𝑥

that is unbiased.

That is, A statistic is complete if only one 𝑔𝑔 ⋅ exists such that

𝐸𝐸 𝑔𝑔 𝑇𝑇 𝒙𝒙 = 𝜽𝜽

4

Sufficient Statistics In the end, Applying Neyman-Fisher factorization can be difficult…

Proving that a statistic is complete can be difficult… (unless you have an exponential family PDF)

Rao-Blackwell-Lehmann-Scheffe Theorem can be difficult…

What else can we do????

5

Cramer-Rao Lower Bounds

6

Cramer-Rao Lower Bounds Question: How could we know if an estimator is a MVUB estimator or is close to an MVUB estimator?

7

Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (CRLB) Sets a lower bound on the variance of any unbiased estimator

As a result: An estimator that achieves the CRLB is the MVUB estimator

The CRLB tells sets a benchmark for estimator variance. — Estimators with a variance “close” to the CRLB are almost the MVUB

estimator. — No estimator can have a variance lower than the CRLB.

𝜃𝜃

Estimator variance

Cramer Rao Lower Bound

Common type of analysis

8

Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (CRLB) Sets a lower bound on the variance of any unbiased estimator

Warning about bounds! Bounds usually tell us what we cannot achieve, not what we can achieve.

— i.e., we do not obtain the maximum lower bound

In some scenarios, bounds can be very “loose” — Therefore the Cramer-Rao Lower Bound may not be achievable

(or possibly even that informative)

𝜃𝜃

Estimator variance

Cramer Rao Lower Bound

9

Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (for a single parameter) is defined by

Assume 𝐸𝐸 �̂�𝜃 − 𝐸𝐸 𝜃𝜃 = 0 (unbiased)

Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜃𝜃

= 0 , for all𝜃𝜃 (regularity condition)

Then

var �̂�𝜃 ≥1

𝐼𝐼 𝜃𝜃 , 𝐼𝐼 𝜃𝜃 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃

𝜕𝜕𝜃𝜃2

Fisher Information

Log-Likelihood functionThe score

10

Cramer-Rao Lower Bounds The Cramer-Rao Lower Bound (for a vector parameter) is defined by

Assume 𝐸𝐸 �𝜽𝜽− 𝐸𝐸 𝜽𝜽 = 0 (unbiased)

Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽


Then

𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 ≽ 0 , 𝐼𝐼𝑖𝑖𝑖𝑖 𝜽𝜽 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽

𝜕𝜕𝜃𝜃𝑖𝑖𝜕𝜕𝜃𝜃𝑖𝑖

FisherInformation

Matrix


Positive Semidefinitive

11

Cramer-Rao Lower Bounds Positive Semidefinite Matrices A matrix 𝑴𝑴 is positive semidefinite if

𝒙𝒙𝐻𝐻𝑴𝑴𝒙𝒙 ≥ 0 , for all 𝑥𝑥 ∈ ℂ𝑁𝑁

Also, A matrix 𝑴𝑴 is positive semidefinite if every eigenvalue of 𝑴𝑴≥ 0

Therefore,

𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 ≽ 0 is equivalent to x𝐻𝐻𝐂𝐂�𝜽𝜽𝑥𝑥 ≥ x𝐻𝐻𝑰𝑰−1 𝜽𝜽 𝑥𝑥 for all 𝑥𝑥 ∈ ℂ𝑁𝑁

Or the eigenvalues of 𝐂𝐂�𝜽𝜽 − 𝑰𝑰−1 𝜽𝜽 are all ≥ 0

12

Cramer-Rao Lower Bounds A Looser Cramer-Rao Lower Bound (for a vector parameter) is defined by

Assume 𝐸𝐸 �𝜽𝜽− 𝐸𝐸 𝜽𝜽 = 0 (unbiased)

Assume 𝐸𝐸 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽


Then

var �𝜽𝜽𝑖𝑖 ≥ 𝑰𝑰−1 𝜽𝜽 𝑖𝑖𝑖𝑖, 𝐼𝐼𝑖𝑖𝑖𝑖 𝜽𝜽 = −𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽

𝜕𝜕𝜃𝜃𝑖𝑖𝜕𝜕𝜃𝜃𝑖𝑖

FisherInformation

Matrix


13

Cramer-Rao Lower Bounds A efficient estimator An estimator is said to be efficient if it meets the Cramer-Rao lower bound.

— Covariance matrix = the inverse Fisher information matrix— 𝐂𝐂�𝜽𝜽 = 𝑰𝑰−1 𝜽𝜽

14

Cramer-Rao Lower Bounds Cramer-Rao Lower Bound Proof (for one parameter)

Assume �̂�𝜃 is unbiased and has a PDF 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝐸𝐸 �̂�𝜃 𝑥𝑥 = �−∞

∞

�̂�𝜃 𝑥𝑥 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 𝜃𝜃

�−∞

∞

�̂�𝜃 𝑥𝑥𝜕𝜕𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃 𝑑𝑑𝑥𝑥 =

𝜕𝜕𝜃𝜃𝜕𝜕𝜃𝜃

�−∞

∞

�̂�𝜃 𝑥𝑥𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 1

�−∞

∞

�̂�𝜃 𝑥𝑥 − 𝜃𝜃𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃 𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 = 1

�−∞

∞

�̂�𝜃 𝑥𝑥 − 𝜃𝜃 2𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 �−∞

∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃

2

𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 ≥ 1

𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃

=𝜕𝜕𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃

1𝑝𝑝 𝑥𝑥; 𝜃𝜃

Express as integral

Take derivative of both sides

Cauchy-SchwartzInequality

Apply regularity condition [new term is equal to 0]

15

Cramer-Rao Lower Bounds Cramer-Rao Lower Bound Proof (for one parameter)

Assume �̂�𝜃 is unbiased and has a PDF 𝑝𝑝 𝑥𝑥;𝜃𝜃

�−∞

∞

�̂�𝜃 − 𝜃𝜃 2𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 �−∞


𝜕𝜕𝜃𝜃

2

𝑝𝑝 𝑥𝑥;𝜃𝜃 𝑑𝑑𝑥𝑥 ≥ 1

var �̂�𝜃 𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃

2

≥ 1

var �̂�𝜃 ≥1

𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃

2

var �̂�𝜃 ≥1

𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃2

Express as expectations

Simplify

See next slide for this proof

16

Cramer-Rao Lower Bounds

𝐸𝐸𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃= 0

�−∞

∞𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃

𝜕𝜕𝜃𝜃𝑝𝑝 𝑥𝑥;𝜃𝜃 = 0

𝜕𝜕𝜕𝜕𝜃𝜃

�−∞


𝜕𝜕𝜃𝜃𝑝𝑝 𝑥𝑥;𝜃𝜃 = 0

�−∞

∞𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃

𝜕𝜕𝜃𝜃2𝑝𝑝 𝑥𝑥; 𝜃𝜃 +

𝜕𝜕 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃

𝜕𝜕𝑝𝑝 𝑥𝑥; 𝜃𝜃𝜕𝜕𝜃𝜃

= 0

�−∞

∞𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃

𝜕𝜕𝜃𝜃2𝑝𝑝 𝑥𝑥;𝜃𝜃 +

𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃


𝑝𝑝 𝑥𝑥; 𝜃𝜃 = 0

𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃

𝜕𝜕𝜃𝜃2+ 𝐸𝐸


2

= 0

𝐸𝐸𝜕𝜕2 ln 𝑝𝑝 𝑥𝑥; 𝜃𝜃

𝜕𝜕𝜃𝜃2= −𝐸𝐸


2

Regularity condition

Express as integral

Take derivative of both sides

Evaluate derivative

𝜕𝜕 ln 𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃 =

𝜕𝜕𝑝𝑝 𝑥𝑥;𝜃𝜃𝜕𝜕𝜃𝜃

1𝑝𝑝 𝑥𝑥; 𝜃𝜃

Express as expectation

17

Cramer-Rao Lower Bounds Example: DC signal in Gaussian noise Consider a length 𝑁𝑁 signal defined by

𝒚𝒚 = 𝐴𝐴𝟏𝟏+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰

Determine the Cramer-Rao Lower Bound.

18


𝒚𝒚 = 𝐴𝐴𝟏𝟏+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰


ln 𝑝𝑝 𝒙𝒙;𝐴𝐴 = − ln 2𝜋𝜋 𝑁𝑁𝜎𝜎2 − 12𝜎𝜎2

𝒙𝒙 − 𝐴𝐴𝟏𝟏 𝑇𝑇𝑰𝑰 𝒙𝒙− 𝐴𝐴𝟏𝟏

ln 𝑝𝑝 𝒙𝒙;𝐴𝐴 = − ln 2𝜋𝜋 𝑁𝑁𝜎𝜎2 − 12𝜎𝜎2

𝒙𝒙𝑇𝑇𝒙𝒙− 2𝐴𝐴𝒙𝒙𝑇𝑇𝟏𝟏+ 𝐴𝐴2𝟏𝟏𝑇𝑇𝟏𝟏

𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝐴𝐴

𝜕𝜕𝐴𝐴2= − 1

2𝜎𝜎22𝟏𝟏𝑇𝑇𝟏𝟏 = −𝑁𝑁/𝜎𝜎2

var �̂�𝐴 ≥ 1−𝐸𝐸 𝜕𝜕

2 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝐴𝐴2

= 𝜎𝜎2

𝑁𝑁

19


𝒚𝒚 = 𝐴𝐴+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰

Consider the estimation

�̂�𝐴 =1𝑁𝑁�𝑛𝑛=1

𝑁𝑁

𝑦𝑦𝑛𝑛

Is this an efficient estimator of 𝐴𝐴?

20


𝒚𝒚 = 𝐴𝐴+𝒘𝒘𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰

Consider the estimation

�̂�𝐴 =1𝑁𝑁�𝑛𝑛=1

𝑁𝑁

𝑦𝑦𝑛𝑛

Is this an efficient estimator of 𝐴𝐴?

var �̂�𝐴 = var1𝑁𝑁�𝑛𝑛=1

𝑁𝑁

𝑦𝑦𝑛𝑛 =1𝑁𝑁2

�𝑛𝑛=1

𝑁𝑁

var 𝑦𝑦𝑛𝑛 =1𝑁𝑁2

𝑁𝑁𝜎𝜎2 =𝜎𝜎2

𝑁𝑁

The variance matches the Cramer-Rao Bound, so the estimator is efficient

21

Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a length 2 signal defined by

𝒚𝒚 = 𝑨𝑨+𝒘𝒘 , 𝑨𝑨 = 𝐴𝐴1 𝐴𝐴2 𝑻𝑻

𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2

5 Determine the Cramer-Rao Lower Bound.

22

Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a length 2 signal defined by

𝒚𝒚 = 𝑨𝑨+𝒘𝒘 , 𝑨𝑨 = 𝐴𝐴1 𝐴𝐴2 𝑻𝑻

𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 31

5𝜎𝜎2, 𝑪𝑪−𝟏𝟏 =

1𝜎𝜎2

3 −1−1 2


ln 𝑝𝑝 𝒙𝒙;𝑨𝑨 = − ln 2𝜋𝜋 𝑪𝑪 − 12𝜎𝜎2

𝒙𝒙 − 𝑨𝑨 𝑇𝑇𝑪𝑪−𝟏𝟏 𝒙𝒙− 𝑨𝑨

ln 𝑝𝑝 𝒙𝒙;𝑨𝑨 = − ln 2𝜋𝜋 𝑪𝑪 − 12𝜎𝜎2

𝒙𝒙𝑇𝑇𝑪𝑪−1𝒙𝒙− 2𝐴𝐴𝒙𝒙𝑇𝑇𝑪𝑪−1𝑨𝑨+𝑨𝑨𝑇𝑇𝑪𝑪−1𝑨𝑨

𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝑨𝑨

𝜕𝜕𝐴𝐴12= −3/𝜎𝜎2


𝜕𝜕𝐴𝐴1𝜕𝜕𝐴𝐴2= 1

𝜎𝜎2


𝜕𝜕𝐴𝐴22= −2/𝜎𝜎2

3𝐴𝐴12 − 2𝐴𝐴1𝐴𝐴2 + 2𝐴𝐴22

𝑰𝑰 𝑨𝑨 = 3/𝜎𝜎2 −1/𝜎𝜎2

−1/𝜎𝜎2 2/𝜎𝜎2

𝑰𝑰−1 𝑨𝑨 = 2 11 3𝜎𝜎2

523

Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a signal defined by

𝒚𝒚 = 𝑨𝑨+𝒘𝒘

𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2

5 Consider the estimator

�𝑨𝑨 = 𝒚𝒚

Is this an efficient estimator of 𝑨𝑨?

24

Cramer-Rao Lower Bounds Example: Two values with Gaussian noise Consider a signal defined by

𝒚𝒚 = 𝑨𝑨+𝒘𝒘

𝒘𝒘~𝒩𝒩 0,𝑪𝑪 , 𝑪𝑪 = 2 11 3𝜎𝜎2

5 Consider the estimator

�𝑨𝑨 = 𝒚𝒚

Is this an efficient estimator of 𝑨𝑨?

𝐂𝐂�̂�𝐴 = 𝐂𝐂𝒚𝒚 = 𝑪𝑪 =2 11 3

𝜎𝜎2

5The co-variance matrix matches the Cramer-Rao Bound, so the estimator is efficient

25

Cramer-Rao Lower Bounds and Linear Models

Cramer-Rao Lower Bounds and Linear Models Relationship between the score function and the MVUB estimator

Score function: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽

Regularity condition: E 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽

= 0

Fisher Information: 𝑰𝑰 𝜽𝜽 = E 𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑻𝑻

Score function + efficient estimator: 𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽

= 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽

IF the score function can be represented in this way, then 𝒈𝒈 𝒙𝒙 is the MVUB estimator

27

Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰

Determine the Cramer-Rao Lower Bound for 𝜽𝜽.

28

Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝜎𝜎2𝑰𝑰


𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 =

𝜕𝜕𝜕𝜕𝜽𝜽 − ln 2𝜋𝜋𝜎𝜎

2 𝑁𝑁.2 −1

2𝜎𝜎2 𝒙𝒙 −𝑯𝑯𝜽𝜽𝑇𝑇 𝒙𝒙−𝑯𝑯𝜽𝜽

𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜽𝜽 =

12𝜎𝜎2

𝜕𝜕𝜕𝜕𝜽𝜽 𝒙𝒙

𝑇𝑇𝒙𝒙− 2𝒙𝒙𝑇𝑇𝑯𝑯𝜽𝜽+ 𝜽𝜽𝑇𝑇𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽


12𝜎𝜎2

−2𝑯𝑯𝑇𝑇𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽

𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑇𝑇 = 𝑰𝑰 𝜽𝜽 =

1𝜎𝜎2𝑯𝑯

𝑇𝑇𝑯𝑯

Score function: Gradient of the likelihood function

Hessian of the likelihood function

29



= 1𝜎𝜎2

−𝑯𝑯𝑇𝑇𝒙𝒙+𝑯𝑯𝑇𝑇𝑯𝑯𝜽𝜽


= 0


= 1𝜎𝜎2𝑯𝑯𝑇𝑇𝑯𝑯

Score function + efficient estimator:

𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 = 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽

𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽

=1𝜎𝜎2𝑯𝑯

𝑇𝑇𝑯𝑯 𝒈𝒈 𝒙𝒙 − 𝜽𝜽𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽

𝜕𝜕𝜽𝜽 =1𝜎𝜎2𝑯𝑯

𝑇𝑇𝑯𝑯 𝑯𝑯𝑇𝑇𝑯𝑯 −1𝑯𝑯𝑇𝑇𝒙𝒙− 𝜽𝜽

30

Cramer-Rao Lower Bounds and Linear Models The MVUB Estimator is therefore

�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑯𝑯 −1𝑯𝑯𝑇𝑇𝒙𝒙 = 𝑯𝑯†𝒙𝒙

Question: What is this above? Have we seen it before?

31

Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝑪𝑪


32

Cramer-Rao Lower Bounds and Linear Models Consider the standard linear model signal of length N 𝒙𝒙 = 𝑯𝑯𝜽𝜽+𝒘𝒘 , 𝒘𝒘~𝒩𝒩 0,𝑪𝑪



𝜕𝜕𝜕𝜕𝜽𝜽 − ln 2𝜋𝜋𝜎𝜎

2 𝑁𝑁.2 −1

2𝜎𝜎2 𝒙𝒙 −𝑯𝑯𝜽𝜽𝑇𝑇𝑪𝑪−1 𝒙𝒙−𝑯𝑯𝜽𝜽

𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜃𝜃𝜕𝜕𝜽𝜽 =

12

𝜕𝜕𝜕𝜕𝜽𝜽 𝒙𝒙

𝑇𝑇𝒙𝒙− 2𝒙𝒙𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽+ 𝜽𝜽𝑇𝑇𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽


12−2𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽

𝜕𝜕2 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽𝜕𝜕𝜽𝜽𝑇𝑇 = 𝑰𝑰 𝜽𝜽 = 𝑯𝑯

𝑇𝑇𝑪𝑪−1𝑯𝑯

Score function: Gradient of the likelihood function

Hessian of the likelihood function

33



= 12−2𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙+ 2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯𝜽𝜽


= 0


= 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯

Score function + efficient estimator:

𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽𝜕𝜕𝜽𝜽 = 𝑰𝑰 𝜽𝜽 𝒈𝒈 𝒙𝒙 − 𝜽𝜽


1𝜎𝜎2𝑯𝑯

𝑇𝑇𝑪𝑪−1𝑯𝑯 𝒈𝒈 𝒙𝒙 − 𝜽𝜽𝜕𝜕 ln 𝑝𝑝 𝒙𝒙;𝜽𝜽

𝜕𝜕𝜽𝜽=

1𝜎𝜎2𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙− 𝜽𝜽

34


�𝜽𝜽 𝒙𝒙 = 𝑯𝑯𝑇𝑇𝑪𝑪−1𝑯𝑯 −1𝑯𝑯𝑇𝑇𝑪𝑪−1𝒙𝒙

Question: This is an interesting result. How are some ways we can interpret this?

35



Question: What is our problem when 𝑯𝑯 = 𝟏𝟏 = 1 1 … 1 𝑇𝑇?

36



Question: What is our problem when 𝑯𝑯 = 𝟏𝟏 = 1 1 … 1 𝑇𝑇? 𝒙𝒙 = 𝟏𝟏𝜃𝜃 = 𝒘𝒘

�̂�𝜃 𝒙𝒙 =𝟏𝟏𝑇𝑇𝑪𝑪−1𝒙𝒙𝟏𝟏𝑇𝑇𝑪𝑪−1𝟏𝟏

=𝟏𝟏𝑇𝑇𝑫𝑫𝑇𝑇𝑫𝑫𝒙𝒙𝟏𝟏𝑇𝑇𝑫𝑫𝑇𝑇𝑫𝑫𝟏𝟏

=𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝒙𝒙𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝟏𝟏

=𝑫𝑫𝟏𝟏 𝑇𝑇𝒙𝒙𝒙𝑫𝑫𝟏𝟏 𝑇𝑇𝑫𝑫𝟏𝟏

=𝒅𝒅𝑇𝑇𝒙𝒙𝒙𝒅𝒅 𝟐𝟐

Now a scalar value! Pre-whitenedSince C is symmetric, it has a square root

An averaging operation

37

VariationsLast TimeSufficient Statistics Sufficient Statistics Sufficient Statistics Slide Number 6Cramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsCramer-Rao Lower BoundsSlide Number 26Cramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear ModelsCramer-Rao Lower Bounds and Linear Models

ece 6540, lecture 07ece6540/slides/2016_ece6540_class007.pdf · sufficient statistics in the end,...

Documents