CHAPTER SIXCHAPTER SIX

EigenvaluesEigenvalues

Outlines

System of linear ODE (Omit)DiagonalizationHermitian matricesOuadratic formPositive definite matrices

MotivationsTo simplify a linear dynamics such

that it is as simple as possible.To realize a linear system characteristics

e.g.,

the behavior of system dynamics.

xAx

zecharacteripofroot

bapeqch

byyay

)(

0)(..

0

2

Example: In a town, each year 30% married women get divorced 20% single women get married In the 1st year, 8000 married women 2000 single

women. Total population remains constant

be the women numbers at year i,

where represent married & single women

respectively.

si

mi

iW

WWLet

si

mi WW &

ii WWW

8.03.0

2.07.0&

2000

800010

If

Question: Why does converges?

Why does it converges to the same limit even

when the initial condition is different?

12,&6000

4000,,

4000

6000

2000

8000,

8.03.0

2.07.0

12121

01

nWWWW

WWAWW

n

iii

14,

6000

4000

0

10000

14

14

0

iWW

W

W

i

iW

Ans: Choose a basis

Given an initial for some

for example

Question: How does one know choosing such a basis?

1

1&

3

221 xx

2211 2

1& xxAxxA

1

1)4000(

3

22000

2000

8000

221100 xCxCWW 21 &CC

nasxCxACxACWA nnn1122110

Def: Let . A scalar is said to be an

eigenvalue or characteristic value of A if

such that

The vector is said to be an eigenvector

or characteristic vector belonging to .

is called an eigen pair of A.

Question: Given A, How to compute eigenvalues

& eigenvectors?

xxA

nnFA

0x

x

),( x

is an eigen pair of

is singular

Note that, is a polynomial, called

characteristic polynomial of A, of degree n in

Thus, by FTA, A has exactly n eigenvalues including

multiplicities. is a eigenvector associated with

eigenvalue while is eigenspace of A.

0)det(

),(

0,0)(

0,

IA

IA

xxIA

xxxA

nnFA

.

)det()( AIp

0\)( AINx

),( x

)( IAN

Example: Let

are eigenvalues of A. To find eigenspace of 2:( i.e., )

3&2

)3)(2(65

11

24det

)det()(

11

24

2

AIp

A

1

1)2(

0

0

11

220)2(

2

1

spanIAN

x

xxIA

)2( IAN

To find eigenspaces of 3(i.e., )

Let

1

2)3(

0

0

21

210)3(

2

1

spanIAN

x

xxIA

)3( IAN

30

02

11

21

1App

p

Let . Then

is an eigenvalue of A.

has a nontrivial solution.

is singular.

loses rank.

nIANullityvii

IAvi

IAv

IAiv

IANiii

xIAii

i

)()(

)(

0)det()(

)(

0)()(

0))((

)(

nnFA

Let .

If is an eigenvalue of A with eigenvector

Then

This means that is also an eigen-pair of

A.

xxxAxA

xxA

nnA

.xC

),( x

Let .

Where are eigenvalues of A.

(i) Let

(ii) Compare with the coefficient of , we

have

nnA

)()det()(1

i

n

iAIp

n 1

n

iiA

1

det0

1n)(

11

Atracean

iii

n

ii

Theorem 6.1.1: Let

Then and consequently A & B have the

same eigenvalues.

Pf: Let for some nonsingular matrix S.

)()det(

)det()det()det(

)(det

)det(

)det()(

1

1

1

A

B

PAI

SAIS

SAIS

ASSI

BIP

BA ~

)()( BA PP

ASSB 1

Diagonalization

Goal: Given find nonsingular matrix S

a diagonal matrix.

Question1: Are all matrices diagonalizable?

Question2: What kinds of A are diagonalizable?

Question3: How to find S if A is diagonalizable?

nnFA

DASS 1

NOT all matrices are diagonalizable

e.g., Let

If A is diagonalizable

nonsingular matrix S

00

10A

2

11

0

0

d

dASS

00

0

0

0)det(

0)(

1

2

1

21

21

21

Sd

dSA

dd

ddA

ddAtrace

To answer Q2,

Suppose A is diagonalizable.

nonsingular matrix S

Let

are eigenpair of A for

This gives a condition for and diagonalizability and a way

to find S.

nSSS 1

ni ,,1

SDAS

)( 11

ndddiagDASS

nSASA 1 nn SdSd 11

ii Sd

Theorem 6.3.2: Let is diagonalizable

A has n linear independent eigenvectors.

Note : Similarity transformation

Change of coordinate

diagonalization

)()( EE

WL

A

B

1 SASB

)()( EE

WL

SI

nnFA

xPy

SI

Theorem6.3.1: If are distinct eigenvalues of a

matrix A with corresponding eigenvectors

, then are linear independent.

Pf: Suppose are linear independent

not all zero

Suppose

are distinct.

are linear independent.

.... golw

0)(2

iii

K

ixCIA

n 1

0)(2

11

K

ii xC

nn

01

i

K

ii xC

Kxx 1 Kxx 1

Kxx 1

KCC 1

01 C

K

i

xx

C

x

1

1

1

*0

&0

Remarks: Let , and

(i) is an eigenpair of A for

(ii) The diagonalizing matrix S is not unique because

Its columns can be reordered or multiplied by

an nonzero scalar

(iii) If A has n distinct eigenvalues , A is diagonalizable.

If the eigenvalues are not distinct , then may or may not

diagonalizable depending on whether or not A has n

linear independent eigenvectors.

(iv)

1SDSA )( 1 ndiagD nSSS 1

ii S ni ,,1

11

1 SdiagSSSDA nkk

Example: Let

For

For

Let

112

202

213

A

100

010

0001SDS

1,1,00det AI

1

1

1

)0(,0 SpanIAN

1

2

0

,

0

2

1

)(,1 SpanIAN

101

221

011

S

Def: If an matrix A has fewer than nlinear independent eigenvectors,we say that A is defective

e.g.

(i) is defective

(ii) is defective

10

11A

nn

00

10A

Example: Let A & B both have the same eigenvalues Nullity (A-2I)=1 The eigenspace associated with has only one dimension. A is NOT diagonalizable However, Nullity (B-2I)=2 B is diagonalizable

263

041

002

&

201

040

002

BA

4,2,2

2

001

020

000

)2(

rankIArank

1

063

021

000

)2(

rankIBrank

2

Question:

Is the following matrix diagonalizable ?

00

00

01

)(

00

10

01

)( BiiAi

The Exponential of a Matrix Motiration :

Motiration:The general solution of

is

The unique solution of

is

Question:What is and

how to compute ?

xAx cetx At)(

00 )( xtx

xAx

0)( 0)( xetx ttA

AteAte

Note that

Define

0 !i

ia

i

ae

......!2!

2

0

AAI

i

Ae

i

iA

Suppose A is diagonalizable with

1

00

)(!

SDS

i

Ae

i

i

i

iAt

1SDSA

kSSDA kk ,1

1 SSeDt

1

0

01

S

e

e

St

t

n

Example: Compute

Sol: The eigenvalues A are

with eigenvectors

0,1

1

3

1

221 xandx

00

01

11

32;1 DXXDXA

11

10

0

X

eXXXee DA

231

6623

ee

ee

Hermitian matrices : Let , then A can

be written as

where

e.g. ,

ii

iiA

274

32

21

11

74

32i

nnCA

iCBA

nnRCB ,

Let , then

e.g. ,

ii

ii

ii

iiH

273

42

274

32

nnCA TH AA

HHH

HHH

HH

ACAC

BABA

AA

Def:

A matrix A is said to be Hermitian if

A is said to be skew-Hermitian if

A is said to be unitary if

( → its column vectors form an orthonormal set in )

HAA

AAH

IUU H nC

Theorem6.4.1: Let Then

(i)

(ii) eigenvectors belonging to distinct eigenvalues are orthogonal

Pf ： (i)Let be an eigenpair of A

(ii) Let and be two eigenpairs of A with

HAA RA

x,) õjjxjj2= xHAx = xHAHx = (xHAx)H = õöjjxjj2

) õ = õö ) õ 2 R(õ1;x1) (õ2;x2)

õ16=õ2) (õ1à õ2)xH1x2= õ1xH1x2à õ2xH1x2

= (õ1x1)Hx2à xH1 (õ2x2)

= xH1AHx2à xH1Ax2= 0

) xH1x2= 0) x1 ? x2

(i)

Theorem: Let Then

Pf ： (i) Let be an eigenpair of A

is pure-imaginary

HAA

axisjA

x,

xxxAx

xAxxxxxHH

HHHH

AAH

Theorem6.4.3: (Schur`s Theorem) Let

Then unitary matrix U is

upper triangular

Pf ： Let be an eigenpair of A with

Choose to be such that is unitary

Choose

Chose to be unitary

Continue this process , we have the theorem

nnCA AUU H

11,w 11 w

nww 2 nwwU 11

2

11

1

1

11

*0

*

nTn

T

H wAwA

w

w

AUU

222

211

2112

2

2

*0

*

0

01

ww

w

UAUUU

w

UH

HH

2w

)1()1(4

32

222

0

nn

Hww

0

0

0

2

1

2112

UUAUU HH

Theorem6.4.4: (Spectral Theorem)

If , then unitary matrix U

that diagonalizes A .

Pf ： By previous Theorem , unitary matrix

, where

T is upper triangular .

T is a diagonal matrix

AAH

TAUUU H

TAUU

UAUAUUTH

HHHHH

AAH

Cor: Let A be real symmetric matrix .Then (i) (ii) an orthogonal matrix U is a diagonal matrixRemark ： If A is Hermitian , then , by Th6.4.4 , Complete orthonormal eigenbasis

RA

AUU T

Example:

Find an orthogonal matrix U that diagonalizes A

Sol : (i)

(ii)

(iii)By Gram-Schmidt Process

The columns of

form an orthogonormal eigenbasis (WHY?)

021

232

120

ALet

TT

T

SpanAIN

SpanAIN

0,1,2,1,0,1)(

6

1,

6

2,

6

1)5(

)5()1()det()( 2 AIp5,1,1

TT

SpanAIN3

1,

3

1,

3

1,

2

1,0,

2

1)(

3

1

2

1

6

13

10

6

23

1

2

1

6

1

U

1,1,5 diagAUU T

Question:In addition to Hermitian matrices ,

Is there any other matrices possessing orthonormal

eigenbasis?

Note ： If A has orthonormal eigenbasis

where U is Hermitian &

D is diagonal

HUDUA

HHHH

HHHH

HHHH

AAUUDUDU

UUDDDUUD

UDUUUDAA

Def: A is said to be normal if

Remark ： Hermitian , Skew- Hermitian and Unitary matrices are all normal

HH AAAA

Theorem6.4.6: A is normal A possesses orthonormal eigenbasisPf ： have proved By Th.6.4.3 , unitary U is upper triangular T is also normal Compare the diagonal elements of

T has orthonormal eigenbasis(WHY?)

""""

AUUT HHH AAAA

HH TTTT

HH TTTT &

2

1

2

2

2

2

2

1

2

2

1

2

1

2

11

nn

n

iin

n

jj

ii

n

jj

tt

tt

tt

jiTij ,0

Singular Value Decomposition(SVD) :

Theorem : Let with rank(A)=r

Then unitary matrices

With

Where

nmCA nmCU

rrr

nn RdiagCV 1&021 r

Hi

r

iii

H

nm

rrH

uu

VVUUVUA

1

212100

0

nrr

mrr

vvVvvV

uuUuuU

1211

1211

,

,

Remark:In the SVD

The scalars are called

singular values of A

Columns of U are called

left singular vectors of A

Columns of V are called

right singular vectors of A

HVUA

r 1

Pf ： Note that , is Hermitian

& Positive semidefinite with

unitary matrix V

where

Define

(1)

(2)

Define (3)

Define is unitary

nnH CAA

rAArank H )(

H

rH VdiagVAA 00,22

1 021 r

)(211 )()(& rnnrnr VVVdiag

HH VVAA 1

2

1

00 22 AVAVAH

rmCAVU 1

11

)3)(1(r

H IUU 11

00

0,HVUA

00

02121 UUVVA

mmCUUU 212 )3)(2(

Remark:In the SVD The singular values of A are unique while U&V are not unique Columns of U are orthonormal eigenbasis for Columns of V are orthonormal eigenbasis for

is an orthonormal basis for

is an orthonormal basis for

is an orthonormal basis for

is an orthonormal basis for

HVUA r 1

AAHHAA

VUA

UUAV

H

00

0

nr vv 1

mr uu 1

ruu 1

rvv 1

)( HAN

)( HAR

)(AN

)(AR

rank(A) = number of nonzero singular values

but rank(A) ≠ number of nonzero eigenvalues

for example

0,0)(1)(

00

10

AbutArankA

Example : Find SVD of

Sol :

An orthonormal eigenbasis associate with

can be found as

Find U is orthogonal

A set of candidate for are

Thus

00

11

11

A

0,40,4)( 21 AAT

AAH

11

11

2

1V

TvAu )0,2

1,

2

1(

11

11

)()(, 32 VUAANuu TT

32 &uu

3311

2 ,)0,2

1,

2

1(

1euvAu T

00

00

02

withVUA T

Lemma6.5.2 : Let

be orthogonal . Then

Pf :

mmnm RQRA &

FFAQA

2

1

2

2

1

2

2

21

2

F

n

ii

n

ii

FnF

A

A

QA

QAQAQA

Cor : Let be

the SVD of A . Then

nnH RVUA

n

iiFF

A1

2

We`ll state the next result without proof :

Theorem6.5.3 :

H.(1) be the SVD of A

(2)

C :

nnH RVUA

kSrankRS nm )(

FS

n

kji

F

k

i

Tiii ASvuA

min1

2

1

Application : Digital Image Processing

p. 377

(especially efficient for matrix which

has low rank)

Quadratic Forms :

To classify the type of quadratic surface (line) Optimization : An application to the Calculus

Def : A quadratic equation in two variables x & y

is an equation of the form

02 22 feydxcybxyax

0),(),(

f

y

xed

y

x

dc

bayx

Standard forms of conic sections

(i)

(ii)

(iii)

(iv)

Note : Is there any difference between the eigenvalues

A of the quadratic form ?XAX T

circlery

xyxryx

2222

10

01)(

ellipsey

x

b

ayxb

y

a

x

11

0

01

)(1

2

2

2

2

2

2

hyperbolay

x

b

ayxb

y

a

x

11

0

01

)(1

2

2

2

2

2

2

0010

00)(22

y

x

y

xyxyxorxy

Goal : Try to transform the quadratic equation

into standard form by suitable translation

and rotation

Example : (No xy term)

The eigenvalues of the quadratic terms are 9 , 4

→ ellipse

→

→

011164189 22 yyxx

362419 22 yx

1

3

2

2

12

2

2

2

yx

)21(

Example : (Have xy term)

→

→ By direct computation

is orthogonal

( why does such U exist ? )

→ Let the original equation becomes

08323 22 yxyx

831

13)(

y

xyx

00

00

45cos45sin

45sin45cos

2

1

2

12

1

2

1

,40

02UUUA T

y

xU

y

x

840

02)(

y

xyx

x

y

x

y

045

Example :

→

→ Let

→

or

or

→

0428323 22 yyxyx

04)280(31

13)(

y

x

y

xyx

2

1

2

12

1

2

1

, Uy

xU

y

x

x

y

x

y

4)280(40

02)(

y

xU

y

xyx

48)(4)(2 22 yxyx

8)1(2)2( 22 yx

1&2,

8)(2)( 22

yyxx

yx

x

y

Optimization :

Let

It is known from Taylor’s Theorem of Calculus that

Where is Hessian matrix

is local extremum

If

then is a local minimum

2: CRRf n

...

)()())(()()( 00000

TOH

xxHxxxxxfxfxf T

nnxx RxfHji

))(( 0

))(,( 00 xfx 0)( 0 xf

)0.(

0)()(0&0)( 000

resp

xxHxxxf T

}0{)( 00* xxxxNx

))(,( 00 xfx .)max.(resp

Def : A real symmetric matrix A is said to be

(i) Positive definite denoted by

(ii) Negative definite denoted by

(iii) Positive semidefinite denoted by

(iv) Negative semidefinite denoted by

example :

is indefinite

question : Given a real symmetric matrix , how to determine

its definiteness efficiently ?

0,00 xAxxifA T

0,00 xAxxifA T

0,00 xAxxifA T

0,00 xAxxifA T

040

02

A

040

02

A

040

02

A

Theorem6.5.1: Let Then

Pf : let be eigenpair of A

Suppose

Let be an orthonormal eigen-basis of A

(Why can assume this ? )

""

""

nnT RAA RAA )(0

),(&0 xA 2

xxAxT

02 x

xAxT

RA)(}{ 1 nxx

0A

n

n

iii

n someforxXRx 11

,

)?(0

)()(

1

2

11

why

xxxAx

n

iii

n

iiii

Tn

iii

T

Example: Find local extrema of

Sol :

Thus f has local maximum at while

are saddle points

22 cosy3)( zxzxzyxf

)0,0,0()2siny,3,2( zxzxf

)0,,0()y,,( nzx

201

030

102

)0,2,0( nH

1,3,3)( H

201

030

102

)0,)12(,0( nH

1,3,3)( H)0,2,0( n

)0,)12(,0( n

Positive Definite Matrices :

Property I : P is nonsingular

Property II :

and all the leading principal submatrices of

A are positive definite

0& PRPP nnT

0det(P)& )det(P)( i

iPPRPP iinnT ,00&

Property III :

P can be reduced to upper triangular form

using only row operation III and the pivots

elements will all be positive

Sketch of the proof :

& determinant is invariant under row

operation of type III

Continue this process , the property can be proved

0& PRPP nnT

011 P

22

1211

2221

12112 0 P

PP

PP

PPP

02 P0)1(

22 P

Property IV : Let Then

(i) A can be decompose as A=LU where L is lower triangular & U is upper triangular

(ii) A can be decompose as A=LU where L is lower triangular & U is upper triangular with all the

diagonal element being equal to 1 , D is an

diagonal matrix

Pf : by Gaussian elimination and the fact that the

product of two lower (upper) triangular matrix

is lower (upper) triangular

0& ARAA nnT

Example:

Thus A=LU

Also A=LDU with

522

2102

224

A

430

390

2242

1

2

11213

AEE

UAE

300

390

2243

123

13

1

2

1

012

1000

3

1

2

1

2

1 where 231312 EEEL

1003

110

2

1

2

11

&

300

090

004

,

13

1

2

1

012

1001

UDL

Property V : Let

If

Pf :

LHS is lower triangular & RHS is upper triangular

with diagonal elements 1

0& ARAA nnT

212121

222111

&,

then,

UUDDLL

UDLUDLA

11211

12

12

UUDLLD

121

12 UUIUU

IDLLD 11

12

12

(why?) 11

21

1211

2 ILLDDLL

211

1221 DDIDDLL

Property VI : Let

Then A can be factored into

where D is a diagonal matrix & L is lower triangular

with 1’s along the diagonal

Pf :

Since the LDU representation is unique

0& ARAA nnT

TTAA DLULDU

LDUAT

Let

TLDLA

TLU

Property VII : (Cholesky decomposition)

Let

Then A can be factored into where

L is lower triangular with positive diagonal

Hint :

TT LDLDLDLA ))(( 2

1

2

1

0& ARAA nnT

TLLA

Example: We have seen that

Note that

Define we have the

Choleoky decomposition

LUA

300

390

224

13

1

2

1

012

1001

522

2102

224

LDUA

1003

110

2

1

2

11

300

090

004

13

1

2

1

012

1001

Also

TTT LDLALUAA

LDL 2

1

1

TLLA 11

Theorem6.6.1 : Let , Then the followings are equivalent:

(i) A>0

(ii) All the leading principal submatrices have positive determinants.

(iii) A ～ U only using elementary row operation of type III. And the pivots are all positive , where U is an upper triangular matrix.

(iv) A has Cholesky decomposition LLT.

(v) A can be factored into BTB for some nonsingular matrix B

nnT RAA

row

Pf : We have shown that (i) (ii) (iii) (iv) In addition , (iv) (v) (i) is trivial

Housholder Transformation :

Def : Then the matrix

is called

Housholder transformation

Geometrical lnterpretation:

Q is symmetric ,

Q is orthogonal ,

What is the eigenvalues , eigenvectors

and determinant of Q ?

1Let 2u

TuuIQ 2

QQT 1, QQIQQ TT

Given , Find

x u

1)2( exuuIxH T

2221 xxHe

1

1

ex

exu

QR factorization

xu

1e

....

ˆ0

01,

*0

*0

*...

*0

*......

222

21

HHHH

HHAHH

k

kk

Theorem : Let and be a

SVD of A with

Then

Pf :

Cor : Let be nonsingular with

singular values

Then and

nnRA TVUA )( 1 ndiag

)(max12AAA T

12

max

12

max

1

2

max

12

max

12

yxVyxV

xVUxAA

y

TT

x

T

xx

nnRA

n 1

ii 0n

A1

2

1

Application : In solving What is the effect of the

Solution when present measurement error ?

bxA b

bxAbb~~ and ~

~Let

)1(~

.~ 1 bbAxx

b

bbAA

x

xx

A

bx

xAxAb

~

..~

)2(

. , Also

1)2)(1(

is said to be the condition number of A

If A is orthogonal then

This means that , due to the error in b

the deviation of the associated solution of

is minimum if A is orthogonal

1.)( AAAK

nn

n

RAAK , 1)( 1

1)( AK

bAx

Example :

A is close to singular

Note that , is the solution for and is the solution for

What does this mean ?

Similarly , i.e. small deviation in x results in large deviation in b This is the reason why we use orthogonal factorization in Numerical solving Ax=b

11

001.11A

1.)(& AAAK

1

1x

001.2

001.2b

0

2~x

2

2~b

1~

but22

001.0~

x

xx

b

bb

1

0~ ,

0

1

1001

1001~ , 1000

1000bbxx