Outline1 Log likelihood and score

2 Fisher information

3 Cramer Rao Lower Bound

4 Hammersley Chapman Robbins IneqReview

Jesinsine quality f EX E Efcx iff is convex if strictly unless HE

Rao Blacku.ee Thm If 210 D convex in dTX sufficient then JCT Eff IT is better

Bias VarianceTradeoff i MSEIO.ro BiasoCS5tVaroCoIf TCX is complete sufficient andglo is U estimable then there existsa unique unbiased estimator of theform JCT and it is UMVU

2 ways to find it1 Find any unbiased SCT2 Rao Blackwellize any unbiased RX

LoglikelihoodyscoreAssume P has densities fo wit m EIRD

Common support x pdx O same to

Recall l 0 x log foldThought of as random function of O

Defy The is TICO x plays a keyrole in many areas of statistics espasymptotes

Can think of as local sufficient statistic i

fq zx ellootz x

Z MCG x

poolx for 2 0

Differentialidentities assuming enough regularity

I faxed dugdF O f eco D ell dµC

E Teco D Ox tonly true if these are theSame value of 0

E o Sc E.io Eo eednEoLEfoo3 iEolEf 8I

VarololcoxB EoL Teco X

JIO X X x 17Same O same 0

Called Fisherinfor mation

It is possible to extend this definition to certain

cases where he is not even differentiable

e g Laplace location family but for our

purposes we can just assume sufficient

regularity

Try with another statistic JCX let

g O Eo Sad unbiased estimator

glo f seederagco fore elder Eoloradrecoix

Codex Teco x s.IE Ere o

Combining these results with Cauchy Schwarz

gives us the CramertowerBoundor Information Lower Bound

1paray Vards varocl.comDZCovoCS Ico xD2

Varo 501 70

OEIRD gCG4R Var d I Tgco Tco bgco

Inter If go is estimand no unbiased estimator

can have smaller variance than Fgco JcojygCo

EI i i d sample

X X dpdC Oe

X pdx Ipo CxiLet L co x logoff xi

LCOix El.co xii

TCO Varocolco xVarocqecoixiNJ 10 where J.CO is Fisher info

in single observationLower bound scales like n t SDI n Ye for regular famili

Efficiency

CR LB is not nec attainable

We define the efficiency of an unbiased estimator as

effs grafts f YI.gg ifgcos oe1R

effoco E 1

We say SCX is efficient if effoco L FO

Depends on Corro Stx Peco X

Corio Kx Ico x

effoldVargo VargCIco

Conf Edo

El

MX is efficient Carrots Eco I FO

Rarely achieved in finite samples but we can

approach it asymptotically as n co

EI Exponential Families

fz x ez Taa Aq Ux

lly x z'Tx Acn log Hx

Fl z x Tix AttyTEX Ez TX

Vargo Varzltan T'AczTH z x PAG

Eal T lez D TAG

So any unbiased est of z has Varzlor Z TAG

ltanmesley.ch RoobinsIneq

CR LB requiresdifferentiation under integral

Can make more general statement if we

replace TKO D with finite difference

Potecx

Fl elcoteix ko

l

I E olio x small E

Eo lot D 1 poder I 1 0

assuming common support or pagespg

Covolo toff D fo eo 1 poder

Eoff Edoglote glo

Varo s z gcote g o2

FEEDCRIB follows from E o but sgp gives begboty

furvedfanity pdx end thx O c IR

ICO x z olTx BCO t log h6

TICO X 7210 Tcx VB co

Tyco TG T A zoo

Oz O TX Eotc x

i meant

n

Doubts about unbiasedness

The 4M VUE might be very inefficientor inadmissible or just dumb in cases where

another approach makes much more sense

Ex X Bin 1000,0Estimate glo IPO Xz soo

UMV4E is 19 25003 why11 500 Conclude gco 100

X 499 Conclude gCO 0

This is not epistemically reasonableCould do much better with e.g ML E or a

Bayes estimator

In fact our theorem should make us

suspicions of UMVU E's everyidiotic function of T is a UMVUE

of its own expectation

Gets worse Ex 4.7 in Keener

X n Truncated Poisson O

foG0 10

X G eo 4 1,3

O 0

EstimategO e

0mass lost 10

truncationKeener shows UMVUE is S C I

Idiotic but we cannot improve using

any unbiased estimator

Sometimes insisting on unbiased ness leadsus to absurd results

Funny example Basu's elephant story on

course website

Unbiasedness has bad reputation but othermethods have their problems too