• QR Factorization– Direct Method to solve linear systems

• Problems that generate Singular matrices

– Modified Gram-Schmidt Algorithm– QR Pivoting

• Matrix must be singular, move zero column to end.

– Minimization view point Link to Iterative Non stationary Methods (Krylov Subspace)

Outline

1

1

v1 v2 v3 v4

The resulting nodal matrix is SINGULAR, but a solution exists!

LU Factorization fails Singular Example

0

1

1

1

2100

1100

0011

0011

4

3

2

1

v

v

v

v

The resulting nodal matrix is SINGULAR, but a solution exists!

Solution (from picture):v4 = -1v3 = -2v2 = anything you want solutionsv1 = v2 - 1

LU Factorization fails Singular Example

2100

1100

0011

0011

2100

1100

0000

0011One step GE

1 1

2 21 2 N

N N

x b

x bM M M

x b

1 1 2 2 N Nx M x M x M b

Recall weighted sum of columns view of systems of equations

M is singular but b is in the span of the columns of M

QR Factorization Singular Example

1 1 2 2i N N iM x M x M x M M b

0i jM M i j

Orthogonal columns implies:

Multiplying the weighted columns equation by i-th column:

Simplifying using orthogonality:

i

i i i i i

i i

M bx M M M b x

M M

QR Factorization – Key ideaIf M has orthogonal columns

Picture for the two-dimensional case

1M

2Mb

Non-orthogonal Case

1M

2M

b

Orthogonal Case

2x1x

QR Factorization - M orthonormal

0 and 1i j i iM M i j M M M is orthonormal if:

1 1

2 21 2 N

N N

x b

x bM M M

x bOriginal Matrix

1 1

2 21 2 N

N N

y b

y bQ Q Q

y bMatrix withOrthonormal

Columns

TQy b y Q b

How to perform the conversion?

QR Factorization – Key idea

1 2 2 2 12 1Given , find so that, =M M Q M r M

1 2 1 2 12 1 0 M Q M M r M

1 212

1 1

M Mr

M M

2M

2Q

1M

12r

QR Factorization Projection formula

1 2122 2 1=Now find so that 0rQ M Q Q Q

1 1 1 1

1 111

1

111Q M M Q Q

M M r

12 1 2r Q M

Formulas simplify if we normalize

2 2 2

222

2

Fi1

al1

n ly r

Q Q QQ Q

QR Factorization – Normalization

1 11 2 1 1 2 2 1 2 1 1 2 2

2 2

x yM M x M x M Q Q y Q y Q

x y

1 1 2 2 111 22 12M Q M Q Qr r r

Mx=b Qy=b Mx=Qy

11 12 1 1

2 2220

x y

x

r r

r y

QR Factorization – 2x2 case

1 1 11 2 1 2

2 2 2

11 12

220Upper

Triangular

x x bM M Q Q

x x b

Orthonormal

r r

r

Step 1) TQRx b Rx Q b b Two Step Solve Given QR

Step 2) Backsolve Rx b

QR Factorization – 2x2 case

12 11 2 3 1 2 1 3 3 231 2rM M M M M M M rMr M

13 21 3 1 23 0M M M Mr r

13 22 3 1 23 0M M M Mr r

To Insure the third column is orthogonal

QR Factorization – General case

13 21 3 1 23 0M M M Mr r

13 22 3 1 23 0M M M Mr r

1 31 1 1 2

2 32

1

1 2

3

232

M MM M M M

M MM M M

r

M r

QR Factorization – General case

In general, must solve NxN dense linear system for coefficients

1,1 1 1 1 1

1 1 ,1 1 11

N N

N N N N N

N

N N

M M M M M M

M M M M M M

r

r

To Orthogonalize the Nth Vector

2 3 inner products or workN N

QR Factorization – General case

11 2 3 2 13 21 3 1 232 1M M M M M Q M Q Qr r r

1 3 1 213 23 13 1 30Q M Q Q Mr Qr r

To Insure the third column is orthogonal

2 3 1 213 23 23 2 30Q M Q Q Mr Qr r

QR Factorization – General caseModified Gram-Schmidt Algorithm

For i = 1 to N “For each Source Column”

For j = i+1 to N { “For each target Column right of source”

endend

1

ii i

irQ M

jij iMr Q

iii iMr M

Normalize

j jj i iM M Qr

2

1

2 2 operationsN

i

N N

3

1

( )2 operationsN

i

N i N N

QR FactorizationModified Gram-Schmidt Algorithm

(Source-column oriented approach)

4M

3M

2M

1M

12 13 14r r r

1Q

2Q

3Q

4Q

11r

22r 23 24r r

33r

44r

34r

QR Factorization – By picture

1 2 1 1 2 2

1 0 0

0 1

0 0

0 1

N NNx x xx e x e x ex

1 11 1 2 2 22 N N NNx Me x Me x x M x Me xM MMx

Suppose only matrix-vector products were available?

More convenient to use another approach

QR Factorization – Matrix-Vector Product View

For i = 1 to N “For each Target Column”

For j = 1 to i-1 “For each Source Column left of target”

end

end

1

ii i

irQ M

jji iQr M

iii iMr M

Normalize

i ii j jM M Qr

2

1

2 2 operationsN

i

N N

3

1

( )2 operationsN

i

N i N N

"matrix-vector product"i iM Me

QR FactorizationModified Gram-Schmidt Algorithm

(Target-column oriented approach)

4M

3M

2M

1M

1Q

2Q

3Q

4Q

QR Factorization

4M

3M

2M

1M

1Q

2Q

3Q

4Q

r11 r12

r22

r13

r23

r33

r14

r24

r34

r44

r11

r22

r12 r14r13

r23 r24

r33 r34

r44

1 3

0

0

0NMQ M

What if a Column becomes Zero?

Matrix MUST BE Singular!1) Do not try to normalize the column.2) Do not use the column as a source for orthogonalization.3) Perform backward substitution as well as possible

QR Factorization – Zero Column

1 3

0

0

0NQ Q Q

Resulting QR Factorization

11 12 13 1

33 3

0 0 0 0

0 0

0 0 0

0 0 0

N

N

NN

r r r r

r r

r

QR Factorization – Zero Column

1 1

2 21 2 N

N N

x b

x bM M M

x b

1 1 2 2 N Nx M x M x M b

Recall weighted sum of columns view of systems of equations

M is singular but b is in the span of the columns of M

QR Factorization – Zero Column

Reasons for QR Factorization

• QR factorization to solve Mx=b– Mx=b QRx=b Rx=QTb

where Q is orthogonal, R is upper trg

• O(N3) as GE

• Nice for singular matrices– Least-Squares problem

Mx=b where M: mxn and m>n

• Pointer to Krylov-Subspace Methods– through minimization point of view

2

1

Minimize over all xN

T

ii

R x R x R x

Definition of the Residual R: R x b Mx

Find x which satisfies

Mx b

Equivalent if b span cols M

and min 0T

Mx b R x R xx

Minimization More General!

QR Factorization – Minimization View

1 1 1 1 1 1Suppose and therefo rex x e Mx x Me x M

1 1 1 1

T TR x R x b x Me b x Me

One dimensional Minimization

21 1 1 1 12

TT Tb b x b Me x Me Me

1 1 1 12 2 0T TTd

b Me x MR x R Mxx

e ed

11

1 1

T

T T

b Mex

e M Me

Normalization

QR Factorization – Minimization ViewOne-Dimensional Minimization

1 1Me M

b

1e

1x

One dimensional minimization yields same result as projection on the column!

11

1 1

T

T T

b Mex

e M Me

QR Factorization – Minimization ViewOne-Dimensional Minimization: Picture

1 1 2 2 1 1 2 2Now and x x e x e Mx x Me x Me

1 1 2 2 1 1 2 2

T TR x R x b x Me x Me b x Me x Me

Residual Minimization

21 1 1 1 12

TT Tx b Me xb eb Me M

22 2 2 2 22

TTx b Me x Me Me

1 2 1 22T

x x Me MeCoupling

Term

QR Factorization – Minimization ViewTwo-Dimensional Minimization

QR Factorization – Minimization ViewTwo-Dimensional Minimization: Residual Minimization

1 1 2 2 1 1 2 2

T TR x R x b x Me x Me b x Me x Me

22 2 2 2 22

TTx b Me x Me Me

1 2 1 22T

x x Me MeCoupling

Term

21 1 1 1 12

TT Tx b Me xb eb Me M

termcouplingeMeMxeMbdx

xRxdR TTT

)()(220)()(

1111

1

termcouplingeMeMxeMbdx

xRxdR TTT

)()(220)()(

2222

2

To eliminate coupling term: we change search directions !!!

1 1 2 2 1 1 2 2and x v p v p Mx v Mp v Mp

21 1 1 1 12

TTT TR x R x b v b Mp v Mp Mb p

More General Search Directions

22 2 2 2 22

TTv b Mp v Mp Mp

1 2 1 22T

v v Mp MpCoupling

Term

1 2 1 2 span , = span ,p p e e

1 2If Minimization 0 s D ecouple!!T Tp M Mp

QR Factorization – Minimization ViewTwo-Dimensional Minimization

2211 exexx

More General Search Directions

QR Factorization – Minimization ViewTwo-Dimensional Minimization

Goal: find a set of search directions such that

In this case minimization decouples !!!

pi and pj are called MTM orthogonal

ijpMMp jTT

i when 0

1

1

0i

T Ti i ji j i j

j

p e r p p M Mp

i-th search direction equals orthogonalized unit vector

T

j i

ji T

j j

Mp Mer

Mp Mp

Use previous orthogonalized Search directions

QR Factorization – Minimization ViewForming MTM orthogonal Minimization Directions

2Minimize: 2T T

i i i i iv Mp Mp v b Mp

Ti

i T

i i

b Mpv

Mp Mp

Differentiating 2 0 2:T T

i i i iv Mp Mp b Mp

QR Factorization – Minimization ViewMinimizing in the Search Direction

When search directions pj are MTM orthogonal, residual minimization becomes:

For i = 1 to N “For each Target Column”

For j = 1 to i-1 “For each Source Column left of target”

end

end

1

ii i

irp p

iii iMpr Mp

Normalize

i ix x v p

i ip e

Tjj

Tiir p M Mp

i ji i jp p pr Orthogonalize Search Direction

QR Factorization – Minimization ViewMinimization Algorithm

Intuitive summary

• QR factorization Minimization view

(Direct) (Iterative)• Compose vector x along search directions:

– Direct: composition along Qi (orthonormalized columns of M) need to factorize M

– Iterative: composition along certain search directions you can stop half way

• About the search directions:– Chosen so that it is easy to do the minimization

(decoupling) pj are MTM orthogonal

– Each step: try to minimize the residual

1

1

11

1e

p

r

2 112

22

2

1e e

r

p

r

2

1i

N

N

iNN

rr

e e

p

1Q

M M

2Q

M

NQ

MTMOrthonormal

Orthonormal

Compare Minimization and QR

Summary

• Iterative Methods Overview– Stationary– Non Stationary

• QR factorization to solve Mx=b– Modified Gram-Schmidt Algorithm– QR Pivoting– Minimization View of QR

• Basic Minimization approach• Orthogonalized Search Directions• Pointer to Krylov Subspace Methods

Forward Difference Formula

Of order o(h2) forNumerical Differentiation

Y’ = e-x sin(x)

Terima kasihTerima kasih