Linear Algebra and Applications:Numerical Linear Algebra

David S. Watkinswatkins@math.wsu.edu

Department of Mathematics

Washington State University

IMA Summer Program, 2008 – p. 1

My Pledge to You

IMA Summer Program, 2008 – p. 2

My Pledge to YouI promise not to cover as much materialas I previously claimed I would.

IMA Summer Program, 2008 – p. 2

Resources

IMA Summer Program, 2008 – p. 3

Resources (a biased list)

IMA Summer Program, 2008 – p. 3

Resources (a biased list)David S. Watkins,Fundamentals of MatrixComputations, Second Edition, John Wiley andSons, 2002. (FMC)

IMA Summer Program, 2008 – p. 3

Resources (a biased list)David S. Watkins,Fundamentals of MatrixComputations, Second Edition, John Wiley andSons, 2002. (FMC)

David S. Watkins,The QR algorithm revisited,SIAM Review, 50 (2008), pp. 133–145.

IMA Summer Program, 2008 – p. 3

Resources (a biased list)David S. Watkins,Fundamentals of MatrixComputations, Second Edition, John Wiley andSons, 2002. (FMC)

David S. Watkins,The QR algorithm revisited,SIAM Review, 50 (2008), pp. 133–145.

David S. Watkins,The Matrix Eigenvalue Problem,GR and Krylov Subspace Methods, SIAM, 2007.

IMA Summer Program, 2008 – p. 3

Leslie Hogben,Handbook of Linear Algebra,Chapman and Hall/CRC, 2007.

IMA Summer Program, 2008 – p. 4

Leslie Hogben,Handbook of Linear Algebra,Chapman and Hall/CRC, 2007.

Lloyd N. Trefethen and David Bau, III,NumericalLinear Algebra, SIAM, 1997.

IMA Summer Program, 2008 – p. 4

Leslie Hogben,Handbook of Linear Algebra,Chapman and Hall/CRC, 2007.

Lloyd N. Trefethen and David Bau, III,NumericalLinear Algebra, SIAM, 1997.

James W. Demmel,Applied Numerical LinearAlgebra, SIAM, 1997.

IMA Summer Program, 2008 – p. 4

Leslie Hogben,Handbook of Linear Algebra,Chapman and Hall/CRC, 2007.

Lloyd N. Trefethen and David Bau, III,NumericalLinear Algebra, SIAM, 1997.

James W. Demmel,Applied Numerical LinearAlgebra, SIAM, 1997.

G. H. Golub and C. F. Van Loan,MatrixComputations, Third Edition, Johns HopkinsUniversity Press, 1996.

IMA Summer Program, 2008 – p. 4

Common Linear AlgebraComputations

IMA Summer Program, 2008 – p. 5

Common Linear AlgebraComputations

linear systemAx = b

IMA Summer Program, 2008 – p. 5

Common Linear AlgebraComputations

linear systemAx = b

overdetermined linear systemAx = b

IMA Summer Program, 2008 – p. 5

Common Linear AlgebraComputations

linear systemAx = b

overdetermined linear systemAx = b

eigenvalue problemAv = λv

IMA Summer Program, 2008 – p. 5

Common Linear AlgebraComputations

linear systemAx = b

overdetermined linear systemAx = b

eigenvalue problemAv = λv

various generalized eigenvalue problems,e.g.Av = λBv

IMA Summer Program, 2008 – p. 5

Linear Systems

IMA Summer Program, 2008 – p. 6

Linear SystemsAx = b, n × n, nonsingular, real or complex

IMA Summer Program, 2008 – p. 6

Linear SystemsAx = b, n × n, nonsingular, real or complex

Examples: FMC §1.2, 7.1; any linear algebra text

IMA Summer Program, 2008 – p. 6

Linear SystemsAx = b, n × n, nonsingular, real or complex

Examples: FMC §1.2, 7.1; any linear algebra text

Major tools:Gaussian elimination (LU Decomp.)various iterative methods

IMA Summer Program, 2008 – p. 6

Overdetermined Linear Systems

IMA Summer Program, 2008 – p. 7

Overdetermined Linear SystemsAx = b, n × m, n > m

IMA Summer Program, 2008 – p. 7

Overdetermined Linear SystemsAx = b, n × m, n > m

oftenn ≫ m

IMA Summer Program, 2008 – p. 7

Overdetermined Linear SystemsAx = b, n × m, n > m

oftenn ≫ m

Example: fitting data by a straight line

IMA Summer Program, 2008 – p. 7

Overdetermined Linear SystemsAx = b, n × m, n > m

oftenn ≫ m

Example: fitting data by a straight line

minimize‖b − Ax‖2

(least squares)

IMA Summer Program, 2008 – p. 7

Overdetermined Linear SystemsAx = b, n × m, n > m

oftenn ≫ m

Example: fitting data by a straight line

minimize‖b − Ax‖2

(least squares)

Major tools:QR decompositionsingular value decomposition

IMA Summer Program, 2008 – p. 7

Eigenvalue Problems

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

generalized:Av = λBv

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

generalized:Av = λBv

Examples: FMC § 6.7

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

generalized:Av = λBv

Examples: FMC § 6.7

product:A1A2

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

generalized:Av = λBv

Examples: FMC § 6.7

product:A1A2

Examples: generalized (AB−1), SVD (A∗A)

IMA Summer Program, 2008 – p. 8

Eigenvalue Problemsstandard:Av = λv, n × n, real or complex

Examples: FMC § 5.1

generalized:Av = λBv

Examples: FMC § 6.7

product:A1A2

Examples: generalized (AB−1), SVD (A∗A)

quadratic:(λ2K + λG + M)v = 0

IMA Summer Program, 2008 – p. 8

Sizes of Linear Algebra Problems

IMA Summer Program, 2008 – p. 9

Sizes of Linear Algebra Problemssmall

IMA Summer Program, 2008 – p. 9

Sizes of Linear Algebra Problemssmall

medium

IMA Summer Program, 2008 – p. 9

Sizes of Linear Algebra Problemssmall

medium

large

IMA Summer Program, 2008 – p. 9

Solving Linear Systems:

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

A = LU

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

A = LU

PA = LU (partial pivoting)

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

A = LU

PA = LU (partial pivoting)

forward and back substitution

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

A = LU

PA = LU (partial pivoting)

forward and back substitution

Questions: cost?,

IMA Summer Program, 2008 – p. 10

Solving Linear Systems:small problems

Ax = b, n × n, n “small”

storeA conventionally

solve using Gaussian elimination

A = LU

PA = LU (partial pivoting)

forward and back substitution

Questions: cost?, accuracy? (FMC Ch. 2)

IMA Summer Program, 2008 – p. 10

Positive Definite Case

IMA Summer Program, 2008 – p. 11

Positive Definite CaseA = A∗, x∗Ax > 0 for all x 6= 0

IMA Summer Program, 2008 – p. 11

Positive Definite CaseA = A∗, x∗Ax > 0 for all x 6= 0

A = R∗R Cholesky decomposition

IMA Summer Program, 2008 – p. 11

Positive Definite CaseA = A∗, x∗Ax > 0 for all x 6= 0

A = R∗R Cholesky decomposition

symmetric variant of Gaussian elimination

IMA Summer Program, 2008 – p. 11

Positive Definite CaseA = A∗, x∗Ax > 0 for all x 6= 0

A = R∗R Cholesky decomposition

symmetric variant of Gaussian elimination

flop count is halved

IMA Summer Program, 2008 – p. 11

Solving Linear Systems:

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

sparse Gaussian elimination

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

sparse Gaussian elimination

A = LU

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

sparse Gaussian elimination

A = LU

factors “usually” less sparse thanA,

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

sparse Gaussian elimination

A = LU

factors “usually” less sparse thanA, but still sparse

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:medium problems

Larger problems are usually sparser.

Use sparse data structure.

sparse Gaussian elimination

A = LU

factors “usually” less sparse thanA, but still sparse

Crucial question: Can factors fit in main memory?

IMA Summer Program, 2008 – p. 12

Solving Linear Systems:

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

Some buzz words: descent method, conjugategradients (CG), GMRES, . . .

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

Some buzz words: descent method, conjugategradients (CG), GMRES, . . .

preconditioners,

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

Some buzz words: descent method, conjugategradients (CG), GMRES, . . .

preconditioners, and on and on.

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

Some buzz words: descent method, conjugategradients (CG), GMRES, . . .

preconditioners, and on and on.

FMC Chapter 7

IMA Summer Program, 2008 – p. 13

Solving Linear Systems:large problems

L andU factors may be too large to store . . .

Use an iterative method.

direct vs. iterative methods

Some buzz words: descent method, conjugategradients (CG), GMRES, . . .

preconditioners, and on and on.

FMC Chapter 7

Richard Barrett et. al.,Templates for the Solution ofLinear Systems, . . . , SIAM 1994. (FREE!!!)

IMA Summer Program, 2008 – p. 13

Moving On

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

standard inner product: 〈x, y〉 =∑n

j=1xjyj

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

standard inner product: 〈x, y〉 =∑n

j=1xjyj

Euclidean norm: ‖x‖2

=(

∑nj=1

x2

j

)1/2

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

standard inner product: 〈x, y〉 =∑n

j=1xjyj

Euclidean norm: ‖x‖2

=(

∑nj=1

x2

j

)1/2

‖x‖2

=√

〈x, x〉

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

standard inner product: 〈x, y〉 =∑n

j=1xjyj

Euclidean norm: ‖x‖2

=(

∑nj=1

x2

j

)1/2

‖x‖2

=√

〈x, x〉

definition of orthogonal:QT = Q−1

IMA Summer Program, 2008 – p. 14

Moving OnOrthogonal Transformations

generally useful computing tools

sticking to real case for simplicity

standard inner product: 〈x, y〉 =∑n

j=1xjyj

Euclidean norm: ‖x‖2

=(

∑nj=1

x2

j

)1/2

‖x‖2

=√

〈x, x〉

definition of orthogonal:QT = Q−1

properties of orthogonal matricesIMA Summer Program, 2008 – p. 14

Elementary Reflectors

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

geometric action

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

geometric action

Qx = y

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

geometric action

Qx = y

creating zeros

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

geometric action

Qx = y

creating zeros

details: FMC Chapter 3

IMA Summer Program, 2008 – p. 15

Elementary Reflectors= Householder transformations

one of two major classes of computationally usefulorthogonal transformations

Q = I − 2uuT , ‖u‖2

= 1

geometric action

Qx = y

creating zeros

details: FMC Chapter 3

QR decomposition

IMA Summer Program, 2008 – p. 15

Uses of theQR Decomposition

IMA Summer Program, 2008 – p. 16

Uses of theQR DecompositionAx = b, n × n

IMA Summer Program, 2008 – p. 16

Uses of theQR DecompositionAx = b, n × n

overdetermined system

IMA Summer Program, 2008 – p. 16

Uses of theQR DecompositionAx = b, n × n

overdetermined system

orthonormalizing vectors

IMA Summer Program, 2008 – p. 16

The Gram-Schmidt Process

IMA Summer Program, 2008 – p. 17

The Gram-Schmidt Processorthonormalization of vectors

IMA Summer Program, 2008 – p. 17

The Gram-Schmidt Processorthonormalization of vectors

relationship toQR decomposition

IMA Summer Program, 2008 – p. 17

The Gram-Schmidt Processorthonormalization of vectors

relationship toQR decomposition

reorthogonalization

IMA Summer Program, 2008 – p. 17

The SVD

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

A = UΣV T

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

A = UΣV T

product eigenvalue problem

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

A = UΣV T

product eigenvalue problem

FMC Chapter 4

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

A = UΣV T

product eigenvalue problem

FMC Chapter 4

numerical rank determination

IMA Summer Program, 2008 – p. 18

The SVDsingular value decomposition

A = UΣV T

product eigenvalue problem

FMC Chapter 4

numerical rank determination

solution of least-squares problem

IMA Summer Program, 2008 – p. 18

End of Part I

IMA Summer Program, 2008 – p. 19