more on distributions of quadratic formsdistributions of quadratic forms martin ohlson outline...

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

The 8th Tartu Conference on MULTIVARIATE STATISTICS

The 6th Conference on MULTIVARIATE DISTRIBUTIONS with Fixed Marginals

More on Distributions of Quadratic Forms

Martin Ohlson and Timo KoskiLinkoping University, Sweden

Tartu, EstoniaJune 29, 2007

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Outline

1 Introduction

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Outline

1 Introduction

2 Quadratic FormsUnivariate Quadratic FormsMultivariate Quadratic Forms

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Outline

1 Introduction

2 Quadratic FormsUnivariate Quadratic FormsMultivariate Quadratic Forms

3 Example

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

This talk is based on my licentiate thesis (Ohlson, 2007)

”Testing Spatial Independenceusing a

Separable Covariance Matrix”

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Testing Independence

Let the matrix X : (p × n) have matrix normal distribution witha separable covariance matrix, i.e.,

X ∼ Np,n (M, Σ, Ψ) .

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Testing Independence

Let the matrix X : (p × n) have matrix normal distribution witha separable covariance matrix, i.e.,

X ∼ Np,n (M, Σ, Ψ) .

Assume that Ψ is unknown, but has some structure.

AR(1)

Intraclass structure

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Testing Independence

Let the matrix X : (p × n) have matrix normal distribution witha separable covariance matrix, i.e.,

X ∼ Np,n (M, Σ, Ψ) .

Assume that Ψ is unknown, but has some structure.

AR(1)

Intraclass structure

My problem is to test spatial independence,

H0 : Σij = 0, when i 6= j ,

where

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

X , M and Σ are partition into k parts, as

X =

X1

...Xk

, µ =

µ1...

µk

and

Σ =

Σ11 Σ12 · · · Σ1k

Σ21 Σ22 Σ2k

......

. . .

Σk1 Σk2 Σkk

,

where Xi : (pi × n), µi : (pi × 1) and Σij : (pi × pj) fori , j = 1, . . . , k .

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Introduction

Several authors have investigated the density function for amultivariate quadratic form. The density function involves thehypergeometric function of matrix argument, which can beexpand in different ways.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Introduction

Several authors have investigated the density function for amultivariate quadratic form. The density function involves thehypergeometric function of matrix argument, which can beexpand in different ways.

Khatri (1966) - zonal polynomials

Hayakawa (1966); Shah (1970) - Laguerre polynomials

Gupta and Nagar (2000) - generalized Hayakawapolynomials

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Introduction

Several authors have also investigated under what conditionsthe distribution for a multivariate quadratic form is Wishart.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Introduction

Several authors have also investigated under what conditionsthe distribution for a multivariate quadratic form is Wishart.

Rao (1973)

XAX ′ ∼ Wp ⇔ v′XAX ′v ∼ χ2

Gupta and Nagar (2000)

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Univariate Quadratic Forms

q = X′AX,

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Univariate Quadratic Forms

q = X′AX,

where

X ∼ Np(µ, Σ).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Univariate Quadratic Forms

q = X′AX,

where

X ∼ Np(µ, Σ).

What is the distribution of q?

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Two Theorems from (Graybill, 1976).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Two Theorems from (Graybill, 1976).

Theorem (I)

Suppose that X ∼ Np(µ, Σ), where rank(Σ) = p and let

q = X′AX. Then the distribution of q is noncentral χ2, i.e.,

q ∼ χ2k(δ), where δ = 1

2µ′Aµ, if and only if any of the

following three conditions are satisfied

1. AΣ is an idempotent matrix of rank k.

2. ΣA is an idempotent matrix of rank k.

3. Σ is a c-inverse of A and rank(A) = k.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem (II)

Suppose that X ∼ Np(µ, Σ), where rank(Σ) = p. The random

variable q = X′AX has the same distribution as the random

variable

w =n

∑

i=1

diwi ,

where di are the latent roots of the matrix AΣ, and where wi

are independent noncentral χ2 random variables, each with one

degree of freedom.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

A generalization of the second Theorem from(Graybill, 1976) for the multivariate case can be found

in (Mathai and Provost, 1992).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

A generalization of the second Theorem from(Graybill, 1976) for the multivariate case can be found

in (Mathai and Provost, 1992).

In this talk we will also generalize the second Theorem, but ina slightly different way using the characteristic function.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Multivariate Quadratic Forms

Q = XAX ′,

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Multivariate Quadratic Forms

Q = XAX ′,

where X is a normally distributed matrix.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Multivariate Quadratic Forms

Q = XAX ′,

where X is a normally distributed matrix.

What is the distribution of Q?

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Matrix Normal Distribution

X : (p × n) is a normally distributed matrix with a separablecovariance matrix

X ∼ Np,n (M, Σ, Ψ|m, k) ,

where m = rank(Σ) ≤ p and k = rank(Ψ) ≤ n.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Matrix Normal Distribution

X : (p × n) is a normally distributed matrix with a separablecovariance matrix

X ∼ Np,n (M, Σ, Ψ|m, k) ,

where m = rank(Σ) ≤ p and k = rank(Ψ) ≤ n.

The covariance matrix for X is

cov(X ) = cov(vecX ) = Ψ ⊗ Σ : (pn × pn)

where ⊗ is the Kronecker product.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Matrix Normal Distribution

X : (p × n) is a normally distributed matrix with a separablecovariance matrix

X ∼ Np,n (M, Σ, Ψ|m, k) ,

where m = rank(Σ) ≤ p and k = rank(Ψ) ≤ n.

The covariance matrix for X is

cov(X ) = cov(vecX ) = Ψ ⊗ Σ : (pn × pn)

where ⊗ is the Kronecker product.

Hence, the distribution is singular if m < p and/or k < n.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

We start with the following Theorem.

Theorem

Let Y ∼ Np,n (M, Σ, I |m), where m ≤ p and let A : (n × n) be

a symmetric real matrix of rank r . The characteristic function

of Q = YAY ′ is then

ϕQ(T ) =r

∏

j=1

|I − iλjΓΣ|−1/2etr

1

2iλjΩj(I − iλjΓΣ)−1Γ

,

where T = (tij)pi ,j=1, Γ = (γij) = ((1 + δij) tij)

pi ,j=1

, tij = tji and

δij is the Kronecker delta. The noncentrality parameters are

Ωj = mjm′

j , where mj = M∆j . The vectors ∆j and the values

λj are the latent vectors and roots of A respectively, such that

|∆j | = 1.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem

Suppose Y ∼ Np,n (M, Σ, I |m), where m ≤ p and let

Q = YAY ′, where A : (n × n) is a symmetric real matrix of

rank r . Then the distribution of Q is the same as for

W =r

∑

j=1

λjWj ,

where λj are the nonzero latent roots of A and Wj are

independent noncentral Wishart, i.e.,

Wj ∼ Wp(1, Σ,mjm′

j),

where mj = M∆j and ∆j are the corresponding latent vectors

such that |∆j | = 1 for j = 1, . . . , r . If m < p the noncentral

Wishart distributions are singular.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Definition

If Y ∼ Np,n (M, Σ, I |m), we define the distribution of themultivariate quadratic form Q = YAY ′ to be Qp(A, M, Σ|m).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Now, suppose that

X ∼ Np,n (M, Σ, Ψ|m, n)

i.e., the columns are dependent as well.

Suppose also that the matrix Σ is non-negative definite of rankm ≤ p. Ψ is positive definite since rank(Ψ) = n.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem

Let X ∼ Np,n (M, Σ, Ψ|m, n), m ≤ p and let A : (n × n) be a

symmetric real matrix of rank r . The distribution of Q = XAX ′

is the same as for

W =r

∑

j=1

λjWj ,

where λj are the nonzero latent roots of Ψ1/2AΨ1/2 and Wj

are independent noncentral Wishart, i.e.,

Wj ∼ Wp(1, Σ,mjm′

j),

where mj = MΨ−1/2∆j and ∆j are the corresponding latent

vectors such that |∆j | = 1 for j = 1, . . . , r . If m < p the

noncentral Wishart distributions are singular.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

We see that

Q = XAX ′ ∼ Qp(Ψ1/2AΨ1/2, MΨ−1/2, Σ|m).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem

If AΨ is idempotent, then

Qp(Ψ1/2AΨ1/2, MΨ−1/2, Σ) = Wp

(

r , Σ, MAM ′)

,

where r = rank(AΨ).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem

If AΨ is idempotent, then

Qp(Ψ1/2AΨ1/2, MΨ−1/2, Σ) = Wp

(

r , Σ, MAM ′)

,

where r = rank(AΨ).

Proof.

If AΨ is idempotent, then Ψ1/2AΨ1/2 is idempotent as welland λj = 1 for j = 1, . . . , r and zero otherwise. Use of the factthat the sum of independent Wishart distributed matrices isagain Wishart completes the proof.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Example

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Let us assume that we have a matrix

X =(

X1 X2 · · · Xn

)

∼ Np,n (M, Σ, Ψ) ,

where the expectation is

M = µ1′

and

µ : (p × 1),

1′ = (1, 1, . . . , 1) : (1 × n).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Assume that Ψ is known and that we want to estimate Σ.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Assume that Ψ is known and that we want to estimate Σ.

For some reason we estimate the mean with µ = 1nX1.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Assume that Ψ is known and that we want to estimate Σ.

For some reason we estimate the mean with µ = 1nX1.

robust for large matrices

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Assume that Ψ is known and that we want to estimate Σ.

For some reason we estimate the mean with µ = 1nX1.

robust for large matrices

it is all we have available, we only know (X − µ1′)

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Matrix Normal Density

The matrix normal density function is

f (X ) = (2π)−12pn|Σ|−n/2|Ψ|−p/2

etr

−1

2Σ−1 (X − M)Ψ−1 (X − M)′

.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

ML Estimators

µml = (1′Ψ−11)−1XΨ−11,

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

ML Estimators

µml = (1′Ψ−11)−1XΨ−11,

nΣ =(

X − µml1′)

Ψ−1(

X − µml1′)

′

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

ML Estimators

µml = (1′Ψ−11)−1XΨ−11,

nΣ =(

X − µml1′)

Ψ−1(

X − µml1′)

′

= X(

Ψ−1 − Ψ−11(

1′Ψ−11)

−11′Ψ−1

)

X ′ = XHX ′

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

ML Estimators

µml = (1′Ψ−11)−1XΨ−11,

nΣ =(

X − µml1′)

Ψ−1(

X − µml1′)

′

= X(

Ψ−1 − Ψ−11(

1′Ψ−11)

−11′Ψ−1

)

X ′ = XHX ′

Theorem

Let X ∼ Np,n (M, Σ, Ψ), where M = µ1′ and Ψ is known.

Then XHX ′ ∼ Wp (n − 1, Σ).

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

But now we have the estimator of Σ as

nΣ =(

X − µ1′)

Ψ−1(

X − µ1′)

′

= XCΨ−1CX ′,

where C is the centralization matrix

C = I − 1(1′1)−11′ = I − n−111′.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

But now we have the estimator of Σ as

nΣ =(

X − µ1′)

Ψ−1(

X − µ1′)

′

= XCΨ−1CX ′,

where C is the centralization matrix

C = I − 1(1′1)−11′ = I − n−111′.

What is the distribution of XCΨ−1CX ′?

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

Theorem

Let X ∼ Np,n

(

µ1′, Σ, Ψ)

, where Ψ is known. The distribution

of XCΨ−1CX ′ is the same as the distribution of

W = W1 + λ∗W ∗,

where W1 and W ∗ are independent and

W1 ∼ Wp (n − 2, Σ) ,

W ∗ ∼ Wp (1, Σ)

and λ∗ = 1 − 1n1′ΨCΨ−11.

Distributions

of Quadratic

Forms

Martin Ohlson

Outline

Introduction

Quadratic

Forms

Univariate

Multivariate

Example

References

References

Graybill, F. (1976). Theory and Application of the Linear Model. DuxburyPress, North Scituate, Massachusetts.

Gupta, A. and Nagar, D. (2000). Matrix Variate Distributions. Chapmanand Hall.

Hayakawa, T. (1966). On the distribution of a quadratic form inmultivariate normal sample. Annals of the Institute of Statistical

Mathematics, 18:191–201.

Khatri, C. (1966). On certain distribution problems based on positivedefinite quadratic functions in normal vectors. The Annals of

Mathematical Statistics, 37(2):468–479.

Mathai, A. and Provost, S. (1992). Quadratic forms in random variables.M. Dekker New York.

Ohlson, M. (2007). Testing Spatial Independence using a Separable

Covariance Matrix. Lic thesis, Linkopings universitet, Linkoping.

Rao, C. (1973). Linear Statistical Inference and Its Applications. JohnWiley & Sons, New York, USA.

Shah, B. (1970). Distribution theory of a positive definite quadratic formwith matrix argument. The Annals of Mathematical Statistics,41(2):692–697.