reading the lindley-smith 1973 paper on linear bayes estimators

.Introduction

. . . . .Exchangeability

. . . .General bayesian linear model

. . . . . .Examples

. . . . . . . . . . . . . . .Estimation with unknown Covariance References

.

.

. ..

.

.

Bayes Estimates for the Linear Model

Reading Seminar in Statistical Classics

Director: C. P. Robert

Presenter: Kaniav Kamary

12 Novembre, 2012

.Introduction



. . . . . .Examples


Outline

.. .1 Introduction

The Model and the bayesian methods

.. .2 Exchangeability

.. .3 General bayesian linear model

.. .4 Examples

.. .5 Estimation with unknown Covariance

.Introduction



. . . . . .Examples


The Model and the bayesian methods

The linear model :

Structure of the linear model:

E(y) = Aθ

y : a vector of the random variablesA: a known design MatrixΘ: unknown parameters

For estimating Θ:The usual estimate by the method of least squares.Unsatisfactory or inadmissibility in demensions greaterthan two.Improved estimates with knowing prior information aboutthe parameters in the bayesian framework

.Introduction



. . . . . .Examples


Outline

.. .1 Introduction


un example


.. .4 Examples


.Introduction



. . . . . .Examples


un example

The concept of exchangeability

In general linear model suppose A = I :E(yi) = Θi for i = 1,2, . . . , n and yi ∼ N(θi , σ

2) iidThe distribution of θi is exchangeable if:The prior opinion of θi is the same of that of θj or any other θkwhere i , j , k = 1,2, . . . , n.In the other hand:A sequence θ1, . . . , θn of random variables is said to beexchangeable if for all k = 2,3, . . .

θ1, . . . , θn ∼= θπ(1), θπ(2), θπ(k)

for all π ∈ S(k) where S(k) is the group of permutation of1,2, . . . , k

.Introduction



. . . . . .Examples


un example

The concept of exchangeability. . .

One way for obtaining an exchangeable distribution p(Θ):

p(Θ) =n∏

i=1

p(θi | µ)dQ(µ) (1)

p(Θ): exchangeable prior knowledge described by a mixtureQ(µ): arbitrary probability distribution for each µµ: the hyperparameters

A linear structure to the parameters:

E(θi) = µ

.Introduction



. . . . . .Examples


un example

Estimate of Θ

If θi ∼ N(µ, τ2):a closer parallelism between the two stage for y and Θ

By assuming that µ have a uniform distribution over the real linethen:

θ∗i =

yiσ2 + y.

τ2

1σ2 + 1

τ2

(2)

where y. =∑n

i=1 yin and θ∗i = E(θi | y).

.Introduction



. . . . . .Examples


un example

The features of θ∗i

A weighted averages of yi = θ̂i , overall mean y. andinversely proportional to the variances of yi and θi

A biased estimate of θi

Use the estimates of τ2 and σ2

An admissible estimate with known σ2, τ2

A bayes estimates as substitution for the usualleast-squares estimates

.Introduction



. . . . . .Examples


un example

The features of θ∗i . . .

.Judging the merit of θi with one of the other estimates..

.

. ..

.

.

The condition that the average M.S.E for θ∗i to be less than thatfor θ̂i is: ∑

(θi − θ.)2

n − 1< 2τ2 + σ2 (3)

s2 =∑

(θi−θ.)2

n−1 is an usual estimate for τ2. Hence, the chance ofunequal (3) being satisfied is high for n as law as 4 and rapidlytends to 1 as n increases.

.Introduction



. . . . . .Examples


Outline

.. .1 Introduction



The posterior distribution of the parameters

.. .4 Examples


.Introduction



. . . . . .Examples



The structure of the model

Let:Y : a column vector

.Lemma..

.

. ..

.

.

Suppose Y ∼ N(A1Θ1,C1) and Θ1 ∼ N(A2Θ2,C2) that Θ1 is avector of P1 parameters, that Θ2 is a vector of P2hyperparameters.Then (a): Y ∼ N(A1A2Θ2,C1 + A1C2AT

1 ),and (b): Θ1 | Y ∼ N(Bb,B) where:

B−1 = AT1 C−1

1 A1 + C−12

b = AT1 C−1

1 y + C−12 A2Θ2 (4)

.Introduction



. . . . . .Examples



The posterior distribution with three stages

.Theorem..

.

. ..

.

.

With the assumptions of the Lemma, suppose that given Θ3,

Θ2 ∼ N(A3Θ3,C3)

then for i = 1,2,3:

Θ1 | {Ai}, {Ci},Θ3,Y ∼ N(Dd ,D)

withD−1 = AT

1 C−11 A1 + {C2 + A2C3AT

2 }−1

(5)

and

d = AT1 C−1

1 y + AT1 C−1

1 A1 + {C2 + A2C3AT2 }

−1A2A3Θ3 (6)

.Introduction



. . . . . .Examples



The properties

.Result of the Lemma..

.

. ..

.

.

For any matrices A1,A2,C1 and C2 of appropriate dimensionsand for witch the inverses stated, we have:

C1 − C−11 A1(AT

1 C−11 A1 + C−1

2 )−1AT1 C−1

1 = (C1 + A1C2AT1 )

−1

(7)

.Properties of the bayesian estimation..

.

. ..

.

.

The E(Θ1 | {Ai}, {Ci},Θ3,Y ) is:A weighed average of the least-squares estimates(AT

1 C−11 A1)

−1AT

1 C−11 y .

A weithed average of the prior mean A2A3Θ3.It may be regarded as a point estimate of Θ1 to replace theusual least-squares estimate.

.Introduction



. . . . . .Examples



Results of the Theorem

.Corollary1..

.

. ..

.

.

An alternative expression for D−1:

AT1 C−1

1 A1 + C−12 − C−1

2 A2{AT2 C−1

2 A2 + C−13 }−1

AT2 C−1

2 (8)

.Corollary2..

.

. ..

.

.

If C−13 = 0, the posterior distribution of Θ1 is N(D0d0,D0) with:

D−10 = AT

1 C−11 A1 + C−1

2 − C−12 A2{AT

2 C−12 A2}

−1AT

2 C−12 (9)

andd0 = AT

1 C−11 y (10)

.Introduction



. . . . . .Examples


Outline

.. .1 Introduction



.. .4 Examples

Two-factor Experimental DesignsExchangeability Between Multiple Regression EquationExchangeability within Multiple Regression Equation


.Introduction



. . . . . .Examples


Two-factor Experimental Designs

The structure of the Two-factor Experimental Designs

The usual model of n observations with the errorsindependent N(0, σ2):

E(yij) = µ+ αi + βj ,1 ≤ i ≤ t ,1 ≤ j ≤ b

ΘT1 = (µ, α1, . . . , αt , β1, . . . , βb) (11)

yij : an observation in the i th treatment and the j th block.The exchangeable prior knowledge of {αi} and {βj} butindependent

αi ∼ N(0, σ2α), βj ∼ N(0, σ2

β), µ ∼ N(w , σ2µ)

The vague prior knowledge of µ and σ2µ → ∞

C2: the diagonal matrix that leading diagonal of C−12 is

(0, σ−2α , . . . , σ−2

α , σ−2β , . . . , σ−2

β )

C1: the unit matrix times σ2

.Introduction



. . . . . .Examples


Two-factor Experimental Designs

Bayesian estimate of the parameters

With substituting the assumptions stated and C3 = 0 in to (5)and (6), then:

D−1 = σ−2AT1 A1 + C−1

2

d = σ−2AT1 y (12)

Hence Θ∗1, the bayes estimate Dd , satisfies the equation as

following(AT

1 A1 + σ2C−12 )Θ∗

1 = AT1 y (13)

by solving (13),

µ = y..

α∗i = (bσ2

α + σ2)−1

bσ2α(yi. − y..)

β∗j = (tσ2

β + σ2)−1

tσ2β(y.j − y..) (14)

.Introduction



. . . . . .Examples


Exchangeability Between Multiple Regression Equation

The structure of the Multiple Regression Equation

The usual model for p regressor variables wherej = 1,2, . . . ,m:

yj ∼ N(Xjβj , Injσ2j ) (15)

A1: a diagonal matrix with xj as the j th diagonal submatrix

ΘT1 = (βT

1 , βT2 , . . . , β

Tm)

Suppose variables X and Y were related with the usual linearregression structure and

βj ∼ N(ξ,Σ),Θ2 = ξ

A2: a matrix of order mp × p, all of whose p × p submatricesare unit matrices

.Introduction



. . . . . .Examples



Bayesian estimation for the parameters of the Multiple RegressionEquation. . .

The equation for the bayes estimates β∗j is σ1

−2X1T X1 +Σ−1 · · · 0

... σ2−2X2

T X2 +Σ−1...

0 · · · σm−2Xm

T Xm +Σ−1

×

β1

∗

β2∗

...βm

∗

− Σ−1

β.

∗

β.∗

...β.

∗

=

σ1

−2X1T y

σ2−2X1

T y...

σm−2X1

T y

(16)

where β.∗ =

∑ βi∗

m .

.Introduction



. . . . . .Examples



Bayesian estimation for the parameters of the Multiple RegressionEquation

By solving equation (16) for βj∗, the bayes estimate is

βj∗ = (σj

−2XjT Xj +Σ−1)

−1(σj

−2XjT y +Σ−1β.

∗) (17)

Noting that D0−1, given in Corollary 2 (9) and the matrix

Lemma 7, we obtain a weighted form of (17) with β.∗ replaced

by∑

wjβj∗:

wi = {m∑

j=1

(σj−2Xj

T Xj +Σ−1)−1

σj−2Xj

T Xj}−1

(σi−2Xi

T Xi +Σ−1)−1

σi−2Xi

T Xi

(18)

.Introduction



. . . . . .Examples


Exchangeability within Multiple Regression Equation

the model and bayes estimates of the parameters

A single multiple regression:

y ∼ N(Xβ, Inσ2) (19)

The individual regression coefficients in βT = (β1, β2, . . . , βp)are exchangeable and βj ∼ N(ξ, σβ

2)..bayes estimate with two possibilities..

.

. ..

.

.

to suppose vague prior knowledge for ξ with k = σ2

σ2β

β∗ = {Ip + k(X T X )−1(Ip − p−1)}−1β̂ (20)

to put ξ = 0, reflecting a feeling that the βi are small

β∗ = {Ip + k(X T X )−1}−1β̂ (21)

.Introduction



. . . . . .Examples


Outline

.. .1 Introduction



.. .4 Examples


Exposition and methodTwo-factor Experimental Designs(unknown Covariance)Exch between Multiple Regression(unknown Covariance)Exch within Multiple Regression(unknown Covariance)

.Introduction



. . . . . .Examples


Exposition and method

Method

θ: the parameters of interest in the general modelϕ: the nuisance parametersCi : the unknown dispersion matrices

.The method and its defect..

.

. ..

.

.

assign a joint prior distribution to θ and ϕ

provide the joint posterior distribution p(θ, ϕ | y)integrating the joint posterior with respect to ϕ and leavingthe posterior for θfor using loss function, necessity another integration forcalculate the meanrequire the constant of proportionality in bayes’s formulafor calculating the mean

the above argument is technically most complex to execute.

.Introduction



. . . . . .Examples



Solution

For simplified the method:considering an approximationusing the mode of the posterior distribution in place of themeanusing the mode of the joint distribution rather than that ofthe θ-margintaking the estimates derived in section 2 and replace theunknown values of the nuisances parameters by theirmodal estimates

.Introduction



. . . . . .Examples



Solution. . .

The modal value:

∂

∂θp(θ, ϕ | y) = 0,

∂

∂ϕp(θ, ϕ | y) = 0

assuming that p(ϕ | y) ̸= 0 as

∂

∂θp(θ | y , ϕ) = 0 (22)

The approximation is good if:the samples are largethe resulting posterior distributions approximately normal

.Introduction



. . . . . .Examples


Two-factor Experimental Designs(unknown Covariance)

The prior distributions for σ2, σα2 and σβ2 are invers-χ2.

νλ

σ2 ∼ χν2,

ναλα

σα2 ∼ χνα2,

νβλβ

σβ2 ∼ χνβ2

With assuming the three variances independent.The joint distribution of all quantities:

(σ2)−12 (n+ν+2) × exp

−12σ2 {νλ+ S2(µ, α, β)}

×(σ2α)

−12 (t+να+2)exp −1

2σ2α{ναλα +

∑2αi}

×(σ2β)

−12 (b+νβ+2)exp −1

2σ2β

{νβλβ +∑2

βj}

.Introduction



. . . . . .Examples


Two-factor Experimental Designs(unknown Covariance)

Estimates of the parameters of the model

To find the modal estimates:reversing the roles of θ and ϕ with supposing µ, α and βknown

s2 ={νλ+ S2(µ∗, α∗, β∗)}

(n + ν + 2)

sα2 ={ναλα +

∑αi

∗2}(t + να + 2)

sβ2 ={νβλβ +

∑βj

∗2}(b + νβ + 2)

(23)

solving (13) with trial value of σ2, σ2α and σ2

β

inserting the value in to (23)

.Introduction



. . . . . .Examples


Exch between Multiple Regression(unknown Covariance)

Suppositions of the model. . .

In the model (15), suppose σj2 = σ2 with νλ

σ2 ∼ χν2 and Σ−1 has

a Wishart distribution with ρ degree of freedom and matrix Rindependent of σ2.The joint distribution of all the quantities:

(σ2)−1n

2 × exp{ −12σ2

m∑j=1

(yj − Xjβj)T (yj − Xjβj)}

× (| Σ |)−1m

2 exp{−12

m∑j=1

(βj − ξ)TΣ−1(βj − ξ)}

× (| Σ |)−1(ρ−p−1)

2 exp{−12

trΣ−1R}

× (σ2)−1(ν+2)

2 exp{−νλ

2σ2 } (24)

.Introduction



. . . . . .Examples



The joint posterior distribution

The joint posterior density for β, σ2 and Σ−1:

(σ2)−1(n+ν+2)

2 × exp{ −12σ2

m∑j=1

{m−1νλ+ (yj − Xjβj)T(yj − Xjβj)}}

× (| Σ |)−1(m+ρ−p−2)

2

× exp{−12

trΣ−1{R +m∑

j=1

(βj − β.)(βj − β.)T}} (25)

where β. = m−1 ∑mj=1 βj .

.Introduction



. . . . . .Examples



The modal estimates

.The estimates of the parameters..

.

. ..

.

.

s2 =

∑mj=1{m−1νλ+ (yj − Xjβ

∗j )

T (yj − Xjβ∗j )}

(n + ν + 2)(26)

and

Σ∗ ={R +

∑mj=1(β

∗j − β∗

. )(β∗j − β∗

. )T}

(m + ρ− p − 2)(27)

.Introduction



. . . . . .Examples



The modal estimates . . .

The posterior distribution of the βj ’s, free of σ2 and Σ:

{n∑

j=1

{m−1νλ+ (yj − Xjβj)T (yj − Xjβj)}}−

12 (n+ν)

×| R +m∑

j=1

(βj − β.)(βj − β.)T |

− 12 (m+ρ−1)

(28)

The mode of this distribution can be used in place of the modalvalues for the wider distribution.

.Introduction



. . . . . .Examples



Application

.an application in an educational context..

.

. ..

.

.

data from the American Collage Testing Program 1968, 1969prediction of grade-point average at 22 collagesthe results of 4 tests (English, Mathematics, Social Studies,Natural Sciences),p = 5, m = 22, and nj varying from 105 to 739

Table: Comparison of predictive efficiency

reduction the error by under 2 per cent by using the bayesianmethod in the first row but 9 per cent with the quarter sample

.Introduction



. . . . . .Examples


Exch within Multiple Regression(unknown Covariance)

Assumptions of the model regression

In the model 19 and βj ∼ N(ξ, σβ2),

supposeνλ

σ2 ∼ χν2,

νβλβ

σβ2 ∼ χνβ2

The posterior distribution of β, σ2 and σβ2:

(σ2)−1(n+ν+2)

2 × exp{ −12σ2 {νλ+ (y − Xβ)T (y − Xβ)}}

× (σβ2)

−1(p+νβ+1)2

× exp{ −12σβ2 {νβλβ +

p∑j=1

(βj − β.)2}} (29)

that β. = p−1 ∑pj=1 βj .

.Introduction



. . . . . .Examples



The modal estimation. . .

The modal equations:

β∗ = {Ip + k∗X T X−1

(Ip − p−1Jp)}−1β̂

s2 ={νλ+ (y − Xβ∗)T (y − Xβ∗)}

(n + ν + 2)

sβ2 ={νβλβ +

∑pj=1 (βj

∗ − β.∗)2}

(p + νβ + 1)(30)

where k∗ = s2

sβ∗ .

.Introduction



. . . . . .Examples



Comparison between the methods of the estimates

The main difference lies in the choice of kin absolute value, the least-squares procedure produceregression estimates too large, of incorrect sign andunstable with respect to small changes in the dataThe ridge method avoid some of these undesirablefeaturesThe bayesian method reaches the same conclusion buthas the added advantage of dispensing with the ratherarbitrary choice of k and allows the data to estimate it

Table: 10-factor multiple regression example

.Introduction



. . . . . .Examples



A brief explanation of a recent paper

On overview of the Bayesian Linear Model with unknownVariance:

Yn×p = Xp×1 + ξ

The bayesian approache to fitting the linear model consists ofthree steps (S.Kuns, 2009)[4]:

assign priors to all unknown parameterswrite down the likelihood of the data given the parametersdetermine the posterior distribution of the parametersgiven the data using bayes’ theorem

If Y ∼ N(Xβ, k−1) then a conjugate prior distribution for theparameters is: β, k ∼ NG(β0,Σ0,a,b). In other word:

f (β, k) = CK a+ p−22 exp

−12

k{(β − β0)TΣ−1

0 (β − β0) + 2b}

where C = ba

(2π)p2 |Σ0|

12 Γ(a)

.Introduction



. . . . . .Examples



A brief explanation of a recent paper...

The posterior distribution is:

f (β, k | Y ) ∝ ka∗+ p2−1exp{−1

2k((β−β∗)T (Σ∗)−1(β−β∗)+2b∗)}

β∗ = (Σ0−1 + X T X )−1((Σ0

−1β0 + X T y)Σ∗ = (Σ0

−1 + X T X )−1

a∗ = a +n2

b∗ = b +12(β0

TΣ0−1β0 + yT y − (β∗)T (Σ∗)−1β∗)

And β | y follows a multivariate t-distribution:

f (β | y) ∝ (1 +1ν(β − β∗)T (Σ∗)−1(β − β∗))

−12 (ν+p)

.Introduction



. . . . . .Examples


References

L. D. Brown, On the Admissibility of Invariant Estimators of Oneor More Location Parameters, The Annals of MathematicalStatistics, Vol. 37, No. 5 (Oct., 1966), pp. 1087-1136.

A. E. Hoerl, R. W. kennard, Ridge Regression: BiasedEstimation for Nonorthogonal Problems, Technometrics, Vol.12, No. 1. (Feb., 1970), pp. 55-67.

T. Bouche, Formation LaTex, (2007).

S. Kunz, The Bayesian Linear Model with unknown Variance,Seminar for Statistics. ETH Zurich, (2009).

V. Roy, J. P. Hobert, On Monte Carlo methods for Bayesianmultivariate regression models with heavy-tailed errors, (2009).

.Introduction



. . . . . .Examples


References

Thank you

for your Attention

.Introduction



. . . . . .Examples


Result

.Proof...

.

. ..

.

.

To prove (a), suppose y = A1Θ1 + u, Θ1 = A2Θ2 + v that:

u ∼ N(0,C1)

and v ∼ N(0,C2) Then y = A1A2Θ2 + A1v + u that:

A1v + u ∼ N(0,C1 + A1C2AT1 )

To prove (b), by using the Bayesian Theorem:

p(Θ1 | Y ) ∝ e− 12 Q

Q = (Θ1 − Bb)T B−1(Θ1 − Bb)+ yT C−1

1 y +Θ2T A2

T C−12 A2Θ2 − bT Bb (31)

reading the lindley-smith 1973 paper on linear bayes estimators

Real Estate

general linear model

unknown covariance references

distribution of i

linear structure

unknown parameters

bayesian framework

examplethe features

prior opinion of i