Notes on Matrix Computation

University of Chicago, 2014

Vivak Patel

September 7, 2014

1

Contents

1 Introduction 31.1 Variations of solving Ax = b . . . . . . . . . . . . . . . . . . . . . 31.2 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Error Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Floating Point Numbers . . . . . . . . . . . . . . . . . . . . . . . 7

2 Eigenvalue Decomposition 102.1 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . 102.2 Jordan Canonical Form . . . . . . . . . . . . . . . . . . . . . . . 102.3 Spectra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.4 Spectral Radius . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.5 Diagonal Dominance and Gerschgorin’s Disk Theorem . . . . . . 13

3 Singular Value Decomposition 153.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Rank Retaining Factorization 254.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5 QR & Complete Orthogonal Factorization 275.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285.3 Givens Rotations . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.4 Householder Reflections . . . . . . . . . . . . . . . . . . . . . . . 30

6 LU, LDU, Cholesky and LDL Decompositions 31

7 Iterative Methods 327.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327.2 Splitting Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 327.3 Semi-Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . 357.4 Krylov Space Methods . . . . . . . . . . . . . . . . . . . . . . . . 36

2

1 Introduction

1.1 Variations of solving Ax = b

1. Linear Regression. A is known and b is known but corrupted by someerror unknown error r. Our goal is to find x such that:

x ∈ arg minx∈Rn

‖Ax− b‖22 = arg min‖r‖22 : Ax+ r = b

2. Data Least Squares. A is known but corrupted by some unknown errorE. We want to determine:

x ∈ arg minx∈Rn‖E‖2F : (A+ E)x = b

3. Total least squares. A and b are both corrupted by errors E and r (resp.).We want to determine:

x ∈ arg minx∈Rn‖E‖2F + ‖r‖22 : (A+ E)x+ r = b

4. Minimum norm least squares. Given any A and b, we want x such that:

x ∈ arg min‖z‖22 : z ∈ arg min ‖Az − b‖22 = arg min‖z‖22 : ATAz = AT b

5. Robust Regression. The linear regression problem with a different normfor the error r.

6. Regularized Least Squares. Given a matrix Γ, we want to find:

x ∈ arg min‖Ax− b‖22 + ‖Γx‖22

7. Linear Programming. x ∈ arg mincTx : Ax ≤ b

8. Quadratic Programming. x ∈ arg min0.5xTAx+ cTx : Bx = d

1.2 Norms

1. Norm.

Definition 1.1. A norm is a real-valued function defined over a vectorspace, denoted by ‖·‖ such that:

(a) ‖x‖ ≥ 0

(b) ‖x‖ = 0 if and only if x = 0

(c) ‖x+ y‖ ≤ ‖x‖+ ‖y‖(d) ‖αx‖ = |α| ‖x‖ for any scalar α

2. Vector Norms

(a) p-norms

i. For p ≥ 1 the p-norm of x ∈ V is ‖x‖p = (∑ni=1 |xi|p)

1/p

3

ii. For p =∞, the∞-norm or Chebyshev norm is ‖x‖∞ = maxi=1,...,n |xi|iii. The Chebyshev norm is the limit of p-norms

Lemma 1.1. Let x ∈ V then limp→∞ ‖x‖p = ‖x‖∞.

Proof. Let ‖x‖∞ = 1. Then ∃j ∈ 1, . . . , n such that xi = 1.Then:

limp→∞

(n∑i=1

|xi|p)1/p

≤ limp→∞

n1/p = 1

Secondly,

limp→∞

(n∑i=1

|xi|p)1/p

≥ limp→∞

|xj | = 1

iv. Weighted p-norms: add a non-negative weight term to each com-ponent in the sum.

(b) Mahalanobis norm. Let A be a symmetric matrix.

‖x‖A =√x∗Ax

3. Matrix Norms

(a) Compatible. Submultiplicative/Consistent.

Definition 1.2. Let ‖·‖M be a matrix norm and ‖·‖V be a vectornorm.

i. A matrix norm is compatible with a vector norm if:

‖Ax‖V ≤ ‖A‖M ‖x‖V

ii. A matrix norm is consistent or submultiplicative if:

‖AB‖M ≤ ‖A‖m ‖B‖M

(b) Holder Norms

i. The Holder p-norm of A is ‖A‖H,p =(∑m

j=1

∑ni=1 |aij |p

)1/pii. The Holder 2-norm is called the Frobenius norm

iii. The Holder ∞-norm is ‖A‖H,p = maxi,j |aij |(c) Induced Norms

i. Induced Norms. Spectral Norm

Definition 1.3. Let ‖·‖α , ‖·‖β be vector norms. The matrixnorm ‖·‖α,β is the induced norm defined by:

‖A‖α,β = maxx 6=0

‖Ax‖α‖x‖β

When α = β = 2, the induced norm is called the spectral norm.

ii. Equivalent Definitions

4

Lemma 1.2. The following are equivalent definitions for an in-duced norm:

A. ‖A‖α,β = sup‖Ax‖α : ‖x‖β = 1B. ‖A‖α,β = sup‖Ax‖α : ‖x‖β ≤ 1

Proof. Let x = v‖v‖β

. Using this in the definition, we see that

the definition and first characterization are equivalent. For thesecond characterization, already have that the norm from thesecond characterization is necessarily less than or equal to theone from the definition. Assume that it is strictly less than theone from the definition. From the first characterization, we knowthat the norm is achieved and so let this point be x′. At x′/2 thenorm is still maximized, so the definitions are equivalent.

iii. Compatibility

Lemma 1.3. Letting ‖·‖ = ‖·‖α,β:

‖Ax‖ ≤ ‖A‖ ‖x‖

Proof. For any x 6= 0:

‖A‖ =≥ ‖Ax‖‖x‖

When x = 0, the result holds simply by plugging in values.

iv. Consistency

Lemma 1.4. Letting ‖·‖ = ‖·‖α,β:

‖AB‖ ≤ ‖A‖ ‖B‖

Proof.

‖AB‖ = max‖A(Bx)‖‖x‖

≤ ‖A‖max‖Bx‖x

≤ ‖A‖ ‖B‖

v. Computing ‖·‖1,1 norm. A ∈ Fm×n.

Lemma 1.5.

‖A‖1,1 = maxj=1,...,n

m∑i=1

|aij |

5

Proof. There exists an x such that ‖x‖1 = 1 and ‖Ax‖1 = ‖A‖.Therefore:

‖A‖ =

m∑i=1

n∑j=1

|aij ||xj |

=

n∑j=1

|xj |m∑i=1

|aij |

≤

(maxj

m∑i=1

|aij |

)n∑j=1

|xj |

≤ maxj

m∑i=1

|aij |

For the other direction, suppose the maximum occurs at the kthcolumn. Then ‖Aek‖1 ≤ ‖A‖ but the left hand side is right handside of the array of equations above.

vi. Computing ‖·‖∞.

Lemma 1.6.

‖A‖∞ = maxi=1,...,m

n∑j=1

|aij |

Proof. There is an x such that ‖x‖∞ = 1 and ‖Ax‖∞ = ‖A‖.Therefore:

‖A‖ = ‖Ax‖∞

= maxi=1,...,m

n∑j=1

|aijxj |

≤ maxi=1,...,m

n∑j=1

|aij |

For the other direction, let k be the index of the maximizing row.Let x be a vector such that xi = sgn(aki). Then ‖x‖ = 1 and‖A‖ ≥ ‖Ax‖∞ =

∑nj=1 |akj |.

1.3 Error Analysis

1. Types of Error given for true value x and computed value x

(a) ‖x− x‖ is the absolute error, but it depends on units

(b) ‖x− x‖ / ‖x‖ is the relative error, and it does not depend on units

(c) Pointwise error: compute ‖y‖ where yi = xi−xixi

1 [xi 6= 0].

2. Backwards Error Analysis

(a) Notation

6

i. Suppose we want to solve Ax = b and we denote ∆(A, b) = x thealgorithm which produces the estimate.

ii. The condition number of a matrix κ(A) = ‖A‖∥∥A−1∥∥.

iii. Let ρ = ‖δA‖ ‖A‖ for some small perturbation matrix δA.

(b) Idea: View x as the solution to a nearby system (A+ δA)x = b+ δb.

(c) Error bound

Lemma 1.7. Suppose A is an invertible matrix and we have a com-

patible norm. If ‖δA‖‖A‖ ≤ ε and ‖δb‖‖b‖ ≤ ε then

‖x− x‖‖x‖

≤ 2ε

1− ρκ(A)

Proof. Note that:

(I +A−1δA)(x− x) = A−1 (δb− δAx)

Then:(1− ρ) ‖x− x‖ ≤

∥∥A−1∥∥ (‖δb‖+ ‖δA‖ ‖x‖)

Dividing both sides by ‖x‖ and multiplying the right hand side by1 = ‖A‖ / ‖A‖:

‖x− x‖‖x‖

≤ κ(A)

(‖δb‖‖A‖ ‖x‖

+‖δA‖‖A‖

)Noting that ‖b‖ ≤ ‖A‖ ‖x‖, the result follows.

1.4 Floating Point Numbers

1. Motivation: computers do not have infinite memory and so they can onlystore numbers up to a certain precision

2. Floating Point Numbers, sign, mentissa, exponent, base

Definition 1.4. Floating Point Numbers are F ⊂ Q, that have the fol-lowing representation:

±a1a2 . . . ak × be1...el

(a) ± is called the sign

(b) a1 . . . ak is called the mentissa and ai are values in some finite field

(c) e1 . . . el is called the exponent and ek are values in some finite field

(d) b is the base

3. Floating Point Representation Standards

(a) Floating Point Representation. Machine Precision.

7

Definition 1.5. A floating point representation is a function fl :R → F which is characterized by the machine precision denoted εmand defined as:

εm = infx ∈ R : x > 0, f l(1 + x) 6= 1

(b) Standard 1: ∀x ∈ R, ∃x′ ∈ F such that |x− x′| ≤ εm|x|(c) Standard 2: ∀x, y ∈ R and there is an |ε1| ≤ εm:

fl(x± y) = (x± y)(1 + ε1)

(d) Standard 3: ∀x, y ∈ R and there is an |ε2| ≤ εm:

fl(xy) = (xy)(1 + ε2)

(e) Standard 4: ∀x, y ∈ R such that y 6= 0, there is an |ε3| ≤ εm:

fl(x/y) = (x/y)(1 + ε3)

4. Floating Point in Computers

(a) Fields: b = 2 and ai, ej ∈ 0, 1(b) Storage

i. A floating point number requires 1 + l + k bits of storage usingthe following storage:

±|e1|e2| · · · |el|a1|a2| · · · |ak

ii. 32-bit computers: 1 bit for sign, 8 bits for exponent, and 23 bitsfor the mentissa

iii. 64-bit computers: 1 bit for sign, 11 bits for exponent, and 52bits for the mentissa

(c) Errors and (typical) Handling

i. Round-off Error is when the number is more precise than thementissa allows, and is handled by cutting off lower priority val-ues.

ii. Overflow Error is when the exponent is too large, and is handledby returning representations of the largest value allowed by thesystem (e.g. 1e− 99 or −∞, etc.)

iii. Underflow Error is when the exponent is too negative, resultingin 0.

5. Examples

Example 1.1. Computing l2 norm. Suppose

x =[

10−49 10−50 10−50 · · · 10−50]∈ R101

This can be stored exactly, but for i = 2, . . . , 101, fl(x2i ) = fl(10−100) = 0.Hence, using a naive algorithm would compute ‖x‖l2 = 10−49, which hasa ∼ 12% (check) absolute error. An improved algorithm uses the naivealgorithm on x = x/ ‖x‖∞. This does not produce underflow errors.

8

Example 1.2. Sample Variance. There are two methods of computingsample variance.

(a) First compute x and then compute s2 = 1n−1

∑ni=1(xi − x)2

(b) Compute s2 = 1n−1

[∑ni=1 x

2i − 1

n

(∑nj=1 xj

)2]The first method is

more accurate but requires two passes over the data while the secondonly requires one.

9

2 Eigenvalue Decomposition

2.1 Eigenvalues and Eigenvectors

1. Eigenvalue. Eigenvector.

Definition 2.1. An eigenvector, x, and eigenvalue, λ of a matrix A satisfyAx = λx.

2. Basic Properties

Lemma 2.1. Let A ∈ Cn×n.

(a) Eigenvectors are scale invariant

(b) The eigenspace of an eigenvalue λ, Vλ := x ∈ Cn : Ax = λx, is avector space.

(c) Any n× n matrix has n eigenvalues counted with multiplicity.

3. Eigenvalue Decompositions

(a) Eigenvalue Decomposition

Definition 2.2. A matrix A ∈ Cn×n admits an eigenvalue decom-position if there exists an invertible matrix X and diagonal matrix Λof eigenvalues of A such that:

A = XΛX−1

(b) Equivalent conditions

Proposition 2.1. The following three statements are equivalent forA ∈ Cn×n:

i. A is diagonalizable

ii. A has an eigenvalue decomposition

iii. A has n linearly independent eigenvectors

Proof. We only prove equivalence between the last two. The firstrequires more machinery:

X =[x1 x2 · · ·xn

]⇔ Axi = λxi

X is invertible if and only if it has n linearly independent columns.

2.2 Jordan Canonical Form

1. Every matrix A ∈ Cn×n has a Jordan Canonical form A = XJX−1 where:

(a) J is a bidiagonal matrix where the main diagonal of J contains theeigenvalues of A with their algebraic multiplicity, and the super-diagonal of J is either 0 or 1. The remaining entries of J are 0.

(b) The matrix X contains the eigenvectors of A and some other vectors

10

(c) Note that if the superdiagonal is all 0s then J is diagonalizable.

2. Jordan Blocks

(a) The matrix J can be written in block form with matrices J1, . . . , Jk:J1 O · · · O

O J2. . .

......

. . .. . . O

O · · · O Jk

(b) The Ji are the Jordan blocks. Let λ ∈ Λ(A) then a Jordan Block is

of the form:

λ 1 0 · · · 0

0 λ 1. . .

......

. . .. . .

. . . 0...

. . .. . . 1

0 · · · · · · 0 λ

(c) The number of Jordan blocks corresponding to a specific λi ∈ Λ(A)

is equal to its geometric multiplicity (that is, the dimension of theeigenspace)

(d) The sum of the dimensions of the Jordan blocks for any λi is equalto the algebraic multiplicity of the eigenvalue

3. Basic Properties

(a) Fundamental Lemma

Lemma 2.2. Let Jr be a Jordan Block of a Jordan Canonical Formwith eigenvalue λr. Let nr be its dimension. Then for k = 1, . . . , nr−1 (Jr − λrI)kek = 0

Proof. Let diagk(A) be the vector of values from A of the form Ai,i+k.Then, we note the following:

diag1(Jr − λrI) = 1nr−1

while all other elements in the matrix are 0. And in general:

diagk((Jr − λrI)k) = 1nr−k

while all other elements of the matrix are 0. So when we multiply byek we will get 0.

(b) Powers of Jordan Canonical Form:

Lemma 2.3.Ak = XJkX−1

4. Drawback: J cannot be computed.

11

2.3 Spectra

1. Spectra. Principal Eigenvalues.

Definition 2.3. Let A ∈ Cn×n. The set of eigenvalues of A is called thespectra of A and is denoted λ(A). The maximum element of λ(A) is calledthe Principal Eigenvalue of A.

2. Spectral Theorems

(a) Hermitian Adjoint. Normal Matrix. Hermitian Matrix.

Definition 2.4. The hermitian adjoint of a matrix A is the transposeof A with its elements conjugated. A matrix A is a normal matrix ifA∗A = AA∗. A is a hermitian matrix if A = A∗.

(b) Spectral Theorem for Normal Matrices

Theorem 2.1. Let A ∈ Cn×n. Then the following are equivalent:

i. A is unitarily diagonalizable

ii. A has an orthonormal eigenbasis

iii. A is a normal matrix

iv. A has an EVD of the form A = V ΛV ∗ where V is unitary (i.e.V V ∗ = V ∗V = I)

(c) Spectral Theorem for Hermitian Matrices

Theorem 2.2. Let A ∈ Cn×n. Then the results of the SpectralTheorem for Normal Matrices applies and the eigenvalues of A arereal.

2.4 Spectral Radius

1. Spectral Radius

Definition 2.5. The spectral radius ρ(A) of a matrix A ∈ Cn×n is thelargest absolute eigenvalue:

ρ(A) = max|λ| : λ ∈ Λ(A)

Note 2.1. The spectral radius is not a norm. Consider J ∈ R2×2 whereall entries are zero except J12 = 1. Then ρ(J) = 0 but J 6= 0, hence, ρcannot be a norm.

2. Minimality of ρ and norms

Lemma 2.4. If A ∈ Cn×n and ‖· · ·‖ is a compatible norm then ρ(A) ≤‖A‖

Proof. Let λ be an eigenvalue of A and x be its corresponding eigenvector.Then:

λ ‖x‖ = ‖Ax‖ ≤ ‖A‖ ‖x‖

Since λ is arbitrary in Λ(A) the result follows.s

12

3. Approximating Spectral Radius

Theorem 2.3. Let A ∈ Cn×n and ε > 0. There exists an operator norm‖·‖α depending on A and ε such that ‖A‖α ≤ ρ(A) + ε.

4. A limiting property

Lemma 2.5. Let A ∈ Cn×n. Am → O as m→∞ if and only if ρ(A) < 1.

Proof. Let λ be the largest eigenvalue and x be an eigenvector. Then,Amx = λmx. If Am → O as m → ∞ then taking the limit on both sidesimplies that λm → 0 as m→∞ and so λ < 1. For the other direction, letε > 0 such that ρ(A) + 2ε = 1. Then, we can find an α such that:

‖Am‖α ≤ ‖A‖mα ≤ (ρ(A) + ε)m < 1

Hence, Am → 0.

2.5 Diagonal Dominance and Gerschgorin’s Disk Theorem

1. Diagonally Dominant

Definition 2.6. A matrix A ∈ Cn×n is strictly diagonally dominant if|aii| >

∑j 6=i |aij | for all i ∈ 1, . . . , n. It is diagonally dominant if the

inequality is not strict.

2. Diagonal dominance and non-singularity

Lemma 2.6. A strictly diagonally dominant matrix is nonsingular.

Proof. Suppose A is a strictly diagonally dominant matrix and it is singu-lar. That is ∃x 6= 0 such that Ax = 0. Let k be the index of the absolutelargest element of the vector x. Then:

0 =

n∑i=1

akixi =⇒ −akkxk =∑i 6=k

akixi

Then:|akk||xk| ≤

∑i6=k

|aki||xi| ≤ |xk|∑i 6=k

|aki| < |xk||akk|

Hence, A cannot be singular.

3. Gerschgorin’s Disk Theorem

(a) Gerschgorin’s Disks

Definition 2.7. For A ∈ Cn×n, define for i = 1, . . . , n the disksGi = z ∈ C : |z− aii| ≤ ri| where ri =

∑j 6=i |aij |. The Gi are called

Gerschgorin’s Disks.

(b) Gerschgorin’s Disk Theorem

13

Theorem 2.4. Let A ∈ Cn×n and Gi be its disks and Λ(A) be itsspectra.

i. Λ(A) ⊂⋃ni=1Gi

ii. The number of eigenvalues (with multiplicity) in each connectedcomponent of

⋃ni=1Gi is the number of Gi constituting that com-

ponent.

Proof. Suppose there is a λ ∈ Λ(A) such that λ 6∈ Gi for any i. Then∀i, |λ − aii| > ri. Hence, A − λI is a strictly diagonally dominantmatrix which implies that det(A−Iλ) 6= 0, but this is a contradictionsince λ is an eigenvalue. For the second part, we use a bit of a trick.Let t ∈ [0, 1]. And let A(t) be the matrix whose off diagonals of Amultiplied by t. Note that Gi(t) ⊂ Gi. Moreover, since eigenvaluesare a continuous function of t, A(0) and A(1) have the same numberof eigenvalues in each connected component of

⋃iGi.

14

3 Singular Value Decomposition

3.1 Theory

1. Singular Value Decomposition (SVD)

Definition 3.1. Let A ∈ Cm×n. The singular value decomposition (SVD)of A is UΣV ∗, where:

(a) U ∈ Cm×m and V ∈ Cn×n are unitary matrices

(b) The columns of U are the left singular vectors of A

(c) The columns of V are the right singular vectors of A

(d) Σ ∈ Rm×n≥0 is a diagonal matrix with values σ1 ≥ σ2 ≥ · · · ≥ σr onthe diagonal where r is the rank of A

(e) A = UΣV ∗

2. Properties of Singular Values and Singular Vectors

Lemma 3.1. The left singular vectors of A are the eigenvectors of AA∗.The right singular vectors of A are the eigenvectors of A∗A. The squareof singular values of A are the eigenvalues of AA∗ and A∗A.

Proof. From Ay = σx and A∗x = σy. Then:

(a) A∗Ay = σA∗x = σ2y

(b) AA∗x = σAy = σ2x

3. Other Forms of SVD

(a) Compact/Reduced SVD

Definition 3.2. The compact or reduced SVD of a matrix A ∈ Cm×ncan be written as A = UΣV ∗ where, if r = rank(A):

i. U ∈ Cm×r, whose columns are the left singular vectors of Acorresponding to non-zero singular values, and U∗U = Ir

ii. V ∈ Cn×r, whose columns are the right singular vectors of Acorresponding to non-zero singular values, and V ∗V = Ir

iii. Σ ∈ Rr×r≥0 is a diagonal matrix whose diagonal values are σ1 ≥σ2 ≥ · · · ≥ σr.

(b) Rank-1 SVD

Definition 3.3. Letting u1, . . . , ur and v1, . . . , vr be the left andright singular vectors, and σ1, . . . , σr be the singular values. then,the Rank-1 SVD is:

A = σ1u1v∗1 + · · ·+ σrurv

∗r

4. Existence of SVD

15

Theorem 3.1. Every matrix has a condensed SVD

Proof. There are three steps:

(a) Constructing the Wielandt Matrix and Characterizing its Eigenvalues

W =

[O AA∗ O

]= W ∗

Since W is Hermitian, by the spectral theorem, it has an eigen-value decomposition W = ZΛZ∗ with real eigenvalues. SupposeA has rank r, then W has rank 2r, so it has 2r eigenvalues (sinceit is Hermitian). Let zT = [ x y ] be the transpose of a col-umn of Z and σ be the corresponding eigenvalue. Then, Wz = σzwhich implies Ay = σx and A∗x = σy. Moreover, we see that[ x −y ] is also an eigenvector of W with eigenvalue −σ. Hence,Λ = diag(σ1, σ2, . . . , σr,−σr, . . . , σ1, 0, . . . , 0).

(b) Normalizing columns of Z and showing A = Y ΣrX∗. Let Σr =

diag(σ1, . . . , σr).And normalize the eigenvectors of W so that z∗z =2. Then, x∗x+ y∗y = 2. Also, since the eigenvectors are orthogonaland we have from the first part that x∗x − y∗y = 0. This impliesthat x∗x = y∗y = 1. Rewriting Z in block notation with X,Y , and−Y blocks, we have that A = Y ΣrX

∗.

(c) Orthonormality of X and Y . We now show that the columns of X areorthonormal to themselves and similarly with Y . This follows fromthe orthonormality of z’s and specifically considering [ x −y ].

3.2 Applications

1. Solving Linear Systems

Example 3.1. Suppose we want to solve Ax = b, and the system isconsistent (that is, b ∈ =(A)). If A = UΣV ∗ is the full SVD of A thenΣV ∗x = U∗b. Letting y = V ∗x and c = U∗b, Σy = c can be solved usingback-solve and yr+1, . . . , yn are free parameters. Then V y = x.

2. Inverting Non-singular Matrices

Example 3.2. If A is non-singular, then Σ has no zeros on its diagonal.Therefore:

A−1 = (UΣV ∗)−1 = (V ∗)−1Σ−1U−1 = V Σ−1U∗

Moreover, if Σ = diag(σ1, . . . , σn) then Σ−1 = diag(σ−11 , . . . , σ−1n )

3. Computing the 2-norm

(a) SVD and EVD of Hermitian, psd matrices

16

Lemma 3.2. If M is Hermitian, positive semi-definite then its EVDand SVD coincide.

Proof. By the spectral theorem M = XΛX∗ be the EVD of M whereΛ = diag(λ1, . . . , λn). Since M is positive semi-definite, λi ≥ 0.Therefore, the EVD is an SVD.

(b) The 2-norm is unitary invariant

Lemma 3.3. Let U, V be unitary matrices. Then ‖UAV ‖2 = ‖A‖2

Proof. First:

‖UAV ‖22 = max‖x‖2=1

‖UAV x‖22 = max‖x‖=1

x∗V ∗A∗AV x = max‖x‖=1

‖AV x‖22

Second: let x be any vector and vi be the columns of V . Then,‖V x‖22 =

∑ni=1 xiv

∗i vix =

∑ni=1 |xi|2. And since V is invertible:

‖AV ‖22 = max‖V x‖22=1

‖AV x‖22 = max‖y‖=1

‖Ay‖22 = ‖A‖22

(c) 2-norm and Singular Values

Corollary 3.1. Let the SVD of A be UΣV ∗. Then ‖A‖2 = σ1.

Proof. ‖A‖22 = ‖Σ‖22. Let x be such that x1 = 1 and all other indices

are zero, then this maximizes ‖Σx‖22 = σ21 .

4. Computing the Frobenius Norm

(a) Frobenius Norm is unitary invariant

Lemma 3.4. Let U, V be unitary matrices. Then ‖UAV ‖F = ‖A‖F .

Proof.

‖UAV ‖2F = tr(V ∗A∗U∗UAV )

= tr(V ∗A∗AV )

= tr(V V ∗A∗A)

= tr(A∗A)

= ‖A‖2F

(b) Frobenius Norm and Singular Values

Corollary 3.2. Let the SVD of A be UΣV ∗. Then

‖A‖2F =

rank(A)∑i=1

σ2i

17

Proof. ‖A‖2F = ‖Σ‖2F =∑rank(A)i=1 σ2

i

5. Schatten and KyFan Norms

(a) Schatten p-norm

Definition 3.4. For p ∈ [1,∞), the Schatten p-norm is

‖A‖pσ,p =∑i

σi(A)p

When p =∞, ‖A‖σ,∞ s = maxi σi(A).

(b) Examples of Schatten p-norm

Example 3.3. From the definition:

i. ‖A‖σ,1 =∑i |σi(A)|. This is also denoted ‖A‖∗ and called the

nuclear norm.

ii. ‖A‖σ,2 = ‖A‖Fiii. ‖A‖σ,∞ = ‖A‖2

(c) Ky Fan (p, k)-norm

Definition 3.5. The Ky Fan (p, k)-norm of A is ‖A‖pσ,p,k =∑ki=1 σi(A)p.

6. Computing the Magnitude of the Determinant

(a) The Eigenvalues of Unitary Matrices are 1.

Lemma 3.5. The eigenvalues of unitary matrices are all 1.

Proof. Let U be unitary and x be an eigenvector with eigenvalue λ.Then Ux = λx. Then:

‖x‖2 = ‖Ux‖2 = ‖λx‖2 = ‖x‖2 |λ|

(b) Let A be a matrix with SVD UΣV ∗ then:

|det(A)| = |det(U) det(Σ) det(V )| = |det(Σ)| =n∏i=1

σi(A)

7. Existence and Computing of Pseudo Inverses

Theorem 3.2. For any A ∈ Cm×n, ∃A† ∈ Cn×m such that:

(a) Symmetries: (AA†)∗ = AA† and (A†A)∗ = A†A

(b) “Identity”: AA†A = A and A†AA† = A†

Proof. Let Σ = diag(σ1, σ2, . . . , σr, 0, . . . , 0) ∈ Cm×n. Then

Σ′ = diag(σ−11 , . . . , σ−1r , 0, . . . , 0)

satisfy these conditions. So we denote Σ† = Σ′. Now let A have SVDUΣV ∗. Letting A′ = V Σ†U∗:

18

(a) Symmetries: (AA†)∗ = (UΣΣ†U∗)∗ = UΣΣ†U∗ = AA†. The secondone follows similarly.

(b) “Identity”: AA†A = UΣV ∗V Σ†U∗UΣV ∗ = UΣΣ†ΣV ∗ = A. Thesecond one follows similarly.

Hence, A† exists and A† = V Σ†U∗.

8. Fredholm Alternative

(a) General Linear Group, Kernels and Images

Lemma 3.6. If A ∈ Cm×n and S ∈ GL(n) and T ∈ GL(m) thenker(TA) = ker(A) and im(AS) = im(A).

Proof. If y ∈ im(A) then ∃x such that Ax = y. Since S is invertible,∃z such that Sz = x. In the other direction, if ∃z such that ASz = ythen letting x = Sz, Ax = y. For the kernels, if x ∈ ker(A) thenAx = 0 so TAx = 0, which implies x ∈ ker(TA). If x ∈ ker(TA),then TAx = 0 and since T is invertible, Ax = 0.

(b) Co-kernel. Co-image.

Definition 3.6. Let A ∈ Cm×n. ker(A∗) is the co-kernel of A andim(A∗) is its co-image.

(c) SVD and Spans of Kernel, Co-Kernel, Image and Co-Image.

Proposition 3.1. Let A be a complex valued m × n matrix. Letu1, · · · , um and v1, . . . , vn be the left and right singular vectors of A.Then:

i. ker(A) = spanvr+1, . . . , vnii. ker(A∗) = spanur+1, . . . , um

iii. im(A) = spanu1, . . . , uriv. im(A∗) = spanv1, . . . , vr

Proof. Let UΣV ∗ be the SVD of A and let r = rank(A).

ker(A) = ker(UΣV ∗) = ker(ΣV ∗)

= V ker(Σ)

= V spaner+1, . . . , en= spanvr+1, . . . , vn

Similar reasoning gives ker(A∗) = spanur+1, . . . , um. For the im-age of A:

im(A) = im(UΣV ∗) = im(UΣ)

= Uim(Σ)

= Uspane1, . . . , er= u1, . . . , ur

Similar reasoning gives im(A∗) = spanv1, . . . , vr.

19

(d) Fredholm Alternative

Corollary 3.3. Let A ∈ Cm×n. Then:

i. ker(A) ⊥ im(A∗)

ii. ker(A∗) ⊥ im(A)

iii. ker(A)⊕ im(A∗) = Cn

iv. ker(A∗)⊕ im(A) = Cm

9. Projections

(a) Projection. Orthogonal Projection.

Definition 3.7. Let P ∈ Cn×n.

i. P is a Projection if it is idempotent (i.e. P 2 = P ).

ii. P is an Orthogonal Projection if it is idempotent and Hermitian(i.e. P ∗ = P ).

(b) Orthogonal Projections onto Kernel, Co-Kernel, Image and Co-Image

Lemma 3.7. The following are orthogonal projections onto the re-spective subspaces:

i. Pim(A) = AA†

ii. Pim(A∗) = A†A

iii. Pker(A∗) = (I −AA†)iv. Pker(A) = (I −A†A)

Proof. First we check that the mappings have the correct targetspaces. Let UΣV ∗ be the SVD of A. Then AA† = UΣΣ†U∗. Wenow note two facts:

i. Since Σ is a diagonal matrix:

ΣΣ† =

[Ir OO O

]ii. Secondly:

im(AA†) = im(UΣΣ†U∗) = im(UΣΣ†) = spanu1, . . . , ur

Similarly, im(A†A) = spanv1, . . . , vr. By the Fredholm alterna-tive, the other two manipulations map to the kernel and co-kernel.Idempotence follows from the identity property of the Moore-PenroseInverse. Orthogonality follows from the symmetric property of theMoore Penrose Pseudo inverse.

10. Least Square Problem

(a) Problem: Find x which minimizes ‖b−Ax‖22.

20

Solution. Let UΣV ∗ be the SVD of A. Letting y = V ∗x and c = U∗b,we can restate the problem as finding y which minimizes:

‖b− UΣy‖22 = ‖U∗b− Σy‖22= ‖c− Σy‖22

=

rank(A)∑i=1

(ci − σiyi)2 +∑

i=rank(A)+1

snc2i

This is minimized when yi = ci/σi for i = 1, . . . , rank(A) and theother yi are free to be whatever they choose. We can recover anysolution of x = V y.

(b) It is clear that unless A is non-singular (i.e. has full rank), that theminimizers are not unique.

11. Minimum Length Least Squares Problem

(a) Problem: Find x ∈ arg min‖x‖2 : ‖b−Ax‖22 ≤ ‖b−Ay‖22 ∀y.

Solution. Again, using the fact that ‖x‖ = ‖V y‖ = ‖y‖, we see thatyi = ci/σi for i = 1, . . . , rank(A) and yi = 0 for all other i recoversthe minimum ‖x‖2.

(b) Pseudo Inverse and Minimum Length Least Square Problem

Lemma 3.8. The minimum length least squares solution z = A†b

Proof. Using the Fredholm Alternative, ∃b1, b2 such that b = b1 + b2and b1 ∈ im(A) and b2 ∈ ker(A). Therefore ‖b−Ax‖22 = ‖b1 −Ax‖22+

‖b2‖22. Moreover, ∃x such that b1 = Ax. Using the projections,AA†b = b1 and so AA†b = Ax. Letting z ∈ ker(A) that is z =(I − A†A)y, we guess the solution x = A†b + (I − A†A)y. Pluggingthis in, we see that we do indeed have all of the solutions. Finally,we want to minimize ‖x‖22:

‖x‖22 =∥∥A†b∥∥2

2+ 2〈A†b, (I −A†A)y〉+

∥∥(I −A†A)y∥∥22

=∥∥A†b∥∥2

2+ 2〈(I −A†A)A†b, y〉+

∥∥(I −A†A)y∥∥22

=∥∥A†b∥∥2

2+∥∥(I −A†A)y

∥∥22

This is minimized when y = 0. Therefore, A†b = x is the minimumlength least squares solution.

12. Rank and Numerical rank

(a) Because of floating point errors, matrices are almost always of fullrank on computers, even if they are not analytically

(b) We can use the decay rate of singular values to approximate thenumerical rank. If the decay slows too much then we have likelyreached the true rank of the matrix.

21

(c) Numerical Ranks

Definition 3.8. Let A ∈ Cm×n and τ > 0 be a tolerance. Numericalranks are:

i. ρ− rank(A) = minr ∈ N : σr+1 ≤ τσrii. µ− rank(A) = minr ∈ N :

∑i≥r+1 σ

2i ≤ τ

∑i≥r σ

2i

iii. ν − rank(A) = ‖A‖2F / ‖A‖22

13. Finding Closest Unitary/Orthonormal Matrices

(a) Let U(n) ⊂ GL(n) be all n × n unitary matrices, and O(n) ⊂ U(n)be all n× n orthogonal matrices

(b) Closest Unitary Approximation

Lemma 3.9. Let A ∈ Cn×n, then minX∈U(n) ‖A−X‖F can be X =UV ∗ where UΣV ∗ is the SVD of A.

Proof. By unitary invariance, let Z = U∗XV . So we want to find:

minZ∈U(n)

‖Σ− Z‖2F = n+ min|zi|=1

r∑i=1

(σ2i − 2σi(Re(zi) + iIm(zi)))

since, Z must be diagonal to minimize the problem and the moduliof its elements must be 1. Using Lagrange multipliers, we have theLagrangian:

σ2i − 2σiRe(zi)− iσiIm(zi) + λ(Re(zi)

2 + Im(zi)2 − 1)

Taking derivatives with respect to the real and imaginary parts ofzi and λ, we conclude that Im(zi) = 0 and Re(zi) ∈ −1, 1. Theminimum occurs when zi = 1 since σi > 0. So z1 = · · · = zr = 1 andzr+1, . . . , zn are complex valued numbers with modulus 1. If they arerequired to be real, then the solution is unique. Hence, UZV ∗ = Xand in the real case UV ∗ = X.

(c) Procrustes Problem (?)

Lemma 3.10. X = VBU∗BUAV

∗A is a minimizer of minX∈U(n) ‖A−BX‖

2F

given A and B.

Note 3.1. This does not look to be true.

14. Best r-rank approximation

(a) Problem: find arg minX:rank(X)≤r ‖A−X‖2 given A

(b) Eckart-Young Theorem:

Theorem 3.3. Let the SVD of A be∑rank(A)i=1 σiuiv

∗i . Then for any r

the solution to the problem is X =∑ri=1 σiuiv

∗i and min ‖A−X‖2 =

σr+1

Proof. Suppose ∃B ∈ Cm×n such that rank(B) ≤ r and ‖A−B‖2 <σr+1.

22

i. By the rank-nullity theorem, the nullity(B) ≥ n − r. Let w ∈ker(B). Then Bw = 0. So:

‖Aw‖2 = ‖(A−B)w‖2 ≤ ‖A−B‖2 ‖w‖2 < σr+1 ‖w‖2

ii. Let v ∈ P := span(v1, . . . , vr+1) where UΣV ∗ is the SVD of A,and vi are the columns of V . Then, ∃z such that v = Vr+1z.

‖Av‖22 = ‖UΣV ∗Vr+1z‖22 =

r+1∑i=1

σ2i |αi|2 ≥ σ2

r+1 ‖v‖22

iii. Since dim(P ) = r + 1 and nullity(B) ≥ n − r, P ∩ ker(B) 6=∅. Hence, ∃av ∈ P ∩ ker(B) such that σr+1 ‖v‖2 ≤ ‖Av‖

22 <

σr+1 ‖v‖2 which is a contradiction. Thus ‖A−X‖2 ≥ σr+1, and‖A−X‖2 = σr+1 when X is given as in the theorem.

15. Least Squares with Quadratic Constraints

(a) Problem: For A ∈ Rm×n, b ∈ Rm, α <∥∥A†b∥∥

2, find

arg min‖b−Ax‖2 : ‖x‖2 = α

Solution. Let UΣV ∗ be the SVD of A. Let U∗b = c and V ∗x = z.Then we can restate the problem as:

arg min‖c− Σz‖ : ‖z‖2 = α

The Lagrangian for this problem is then:

L(z, µ) = ‖c− Σz‖22 + µ(‖z‖22 − α2)

Taking derivatives with respect to z and µ, we have the followingsystem:

−2Σ(c− Σz) + 2µz = 0

‖z‖22 = α2

The first equation gives that (Σ2 + µI)−1Σc = z. More explicitly:

rank(A)∑j=1

σjcjσ2j + µ

ej = z

As long as µ > 0 this matrix is invertible (since it is diagonalized).Now we can use the second equation to solve for µ.

α2 =

rank(A)∑i=1

(σjcjσ2j + µ

)2

Using some numerical method, we can solve for µ. And use this tocompute z.

23

(b) Newton-Raphson is one option for solving for µ.

16. Generalized Condition Number

(a) Generalized Condition Number

Definition 3.9. Given A ∈ Cm×n, its Moore-Penrose Inverse A†,and a matrix norm ‖·‖, the Generalized Condition Number of A isκ(A) = ‖A‖

∥∥A†∥∥(b) If ‖·‖ is the 2-norm or Frobenius Norm, we can compute these values

using the SVD. For example:

κ2(A) = ‖A‖2∥∥A†∥∥

2=

σ1σrank(A)

17. Solving Total Least Squares Problems Problem: Find arg min‖E‖2F +

‖r‖22 : (A+ E)x = b+ r.

Solution. Let C = [A b], F = [E r] and zT = [xT − 1]. Then

arg min‖F‖2F : (C + F )z = 0

Since z 6= 0, we want to find F so that rank(C + F ) < n and ‖F‖F is

minimized. If the SVD of C =∑n+1i=1 σiuiv

∗i then we can reduce the rank

of C+F by removing one of the singular values, but we choose the smallestsingular values to minimize the norm of F . So F = −σn+1un+1v

∗n+1. Now

we want to find z such that:(n∑i=1

σiuiv∗i

)z = 0

A simple choice is vn+1, but we need the last element, vn+1,n+1 = −1, sowe let

z =−1

vn+1,n+1vn+1

24

4 Rank Retaining Factorization

4.1 Theory

1. Rank Retaining Factorization

Definition 4.1. Let A ∈ Cm×n have rank r. A rank retaining factoriza-tion (RRF) of A = GH where

(a) G ∈ Cm×r and H ∈ Cr×n

(b) rank(G) = rank(H) = r

2. Properties

(a) Non-Singularity

Lemma 4.1. If an RRF of A is GH and rank(A) = r then G∗Gand HH∗ are non-singular.

Proof. Let UΣV ∗ = G be its SVD. Note that Σ∗ =[

Σr O]

whereΣr is an r × r diagonal matrix of full rank. Then G∗G = V Σ2

rV∗,

which is the SVD of G∗G. Hence rank(G∗G) = r.

(b) Kernel. Co-kernel. Image. Co-image.

Lemma 4.2. Let the RRF of A = GH and rank(A) = r. Then:

i. Im(A) = Im(G)

ii. ker(A) = ker(H)

iii. Im(A∗) = Im(H∗)

iv. ker(A∗) = ker(G∗)

Proof. If A = GH is a RRF for A then A∗ = H∗G∗ is a RRF forA∗. So we need only prove the first two points. If y ∈ Im(A), ∃x s.t.Ax = y. Let z = Hx. Then Gz = y. So y ∈ Im(G). If y ∈ Im(G)then ∃z such that Gz = y. H : Cn → Cr is onto since it has rankr. Therefore, ∃x such that Hx = z. So Im(A) = Im(G). For thekernel, we note that nullity(G) = 0 by the rank nullity theorem.Therefore, only G0 = 0. Hence, ker(A) = ker(H).

4.2 Applications

1. Suppose we want to solve Ax = b where the system is consistent andA ∈ Cm×n has full column rank n.

Solution. By the Fredholm alternative, Cn = ker(A)⊕im(A∗). Therefore,x = x0 + x1 where x0 ∈ ker(A) and x1 ∈ im(A∗). Therefore:

b = Ax = GH(x0 + x1) = GHx1

since ker(A) = ker(H). Since x1 ∈ H∗ there is a z such that H∗z = x1.So:

b = GHH∗z

25

Finally, G ∈ Cn×n and has full rank, and so

(HH∗)−1G−1b = z

Multiplying through then:

H∗(HH∗)−1G−1b = x1

2. Suppose now we have that A has full column rank and we want to find:

arg min‖x‖22 : ‖Ax− b‖2 ≤ ‖Az − b‖22 ∀z

Solution. Since b ∈ Cm, by Fredholm’s alternative, b = b0 + b1 whereb0 ∈ ker(A∗) and b1 ∈ im(A). Since ker(A∗) = ker(G∗) and since Ax = b1is consistent, G∗Ax = G∗b1 = G∗b is consistent. Moreover, this leavesthat Hx = (G∗G)−1G∗b. Now we can split x as we did above to see that

x = H∗(HH∗)−1(G∗G)−1G∗b

26

5 QR & Complete Orthogonal Factorization

5.1 Theory

1. Gram-Schmidt Orthogonalization

Lemma 5.1. Let A ∈ Cm×n. Then there ∃Q ∈ U(m) and R ∈ Cm×nsuch that R is upper triangular.

Proof. Suppose rank(A) = s. Let qi be the columns of a matrix Q whichwe will construct, and rij be the entries of a matrix R which we willconstruct. Let R = rank(A). First, let a1 be the first column of A andlet r11 = ‖a1‖, and

q1 =1

r11a1

Let P2 be a permutation matrix such that AP2 has first column a1 andsecond column a2 that is linearly independent of the first column. Such apermutation exists as long as s ≥ 2. Let j ≤ s and let Pj be a permuta-tion matrix such that AP2P3 · · ·Pj ’s first j − 1 columns are the same asAP2 · · ·Pj−1 and its jth column is linearly independent of the first j − 1columns. We can continue this process up to AP2 · · ·Ps. Let the columnsof this matrix be a1, . . . , an. Then for the first s columns:

qj =aj −

∑j−1i=1 rijqirjj

whererij = 〈aj , qi〉

rjj =

∥∥∥∥∥aj −j−1∑i=1

rijqi

∥∥∥∥∥Note that as+1, . . . , an ∈ spanq1, . . . , qs. Hence, we can finish popula-tion R by finding the coefficients for these columns in a matrix S. Un-fortunately, we have only computed AΠ = Q′M ′ where Q′ ∈ Cm×s and[R S] = M ′ ∈ Cs×n (where Π is a product of permutation matrices so itsinverse is ΠT ). To get the remaining columns, we can complete the basiswith the vectors in Q′ of Rm and make them orthogonal by this process.To complete M ′ we need only add rows of zeros until the right dimensionis achieved. Hence:

AΠ =[Q′ Qs+1,··· ,m

] [ R′ SO O

]

2. Versions of QR Factorization

(a) Full QR Decomposition. This is the version stated in the Lemma.Note when rank(A) < m ∧ n then Q is not unique.

27

(b) Reduced QR Factorization. This version is simplyQ′[R′ S] calculatedin the proof. It not unique given that the permutations can occur inseveral ways.

(c) Complete Orthogonal Factorization. Consider [R′ S]T , which has fullcolumn rank. Then it has a QR decomposition

Z

[UO

]. The complete orthogonal factorization is then:

A = Q

[R′ SO O

]ΠT =

[UT OO O

]ZTΠT = Q

[L OO O

]ΩT

where L is lower triangular.

5.2 Applications

1. Full Rank Least Squares. Suppose A ∈ Cm×n has full column rankand n ≤ m. Find

arg min‖Ax− b‖2

Solution 1. The solution is unique since x = (A∗A)−1AT b is the analyticsolution. We use the full QR decomposition. So we have that:

‖Ax− b‖22 =

∥∥∥∥[ RO

]x−Q∗b

∥∥∥∥We can partition Q∗b into c and d so that:

‖Ax− b‖22 = ‖Rx− c‖22 + ‖d‖22

Therefore, x = R−1c which we can computed by back solve (back substi-tution).

Solution 2. Now we solve the normal equations A∗Ax = A∗b, which comesfrom taking the derivative of ‖Ax− b‖22. In this case we can do QR onA∗A and since A∗A is of full rank, Rx = Q∗A ∗ b can be solved by back-solve. Note that, κ2(A∗A) = κ2(A)2 and so it is less numerically stable,but also A∗A ∈ Cn×n. So this method may be beneficial if n m.

2. Least Squares with Linear Contraints. Find arg min ‖Ax− b‖22 :CTx = d.

Solution 1. Note that the Lagrangian is:

L(λ, x) = ‖b−Ax‖22 + 2λT (CTx− d)

Differentiating returns the system:−2AT (b−Ax) + 2Cλ = 0

CTx = d

28

The first equation can be re-written as ATAx + Cλ = AT b. Writing thisas an augmented system:[

ATA CCT O

] [xλ

]=

[AT bd

]If there is sparsity in A and C, this system can be solved with sparsemethods.

Solution 2. Let QR = C. Let y = QTx and partition y into u and v.Then we can solve Ru = d1, and solve for u. Using this, we can do a

normal minimization of ‖b−AQy‖ =∥∥∥b− A1u− A2v

∥∥∥, where only v is

unknown.

5.3 Givens Rotations

1. A Givens Rotation, G(i,j) is a unitary matrix of the form:

G(i,j)lk =

1 l = k 6∈ i, jλ l = k ∈ i, jσ l = i, k = j

−σ l = j, i = k

0 otherwise

where σ2 + λ2 = 1

2. If σ and λ are selected correctly for a vector v, (G(i,j)v)i or (G(i,j)v)j canbe set to 0.

Example 5.1. Suppose i < j. We can find λ and σ as follows if we wantto set (G(i,j)v)j = 0. Note that:

(G(i,j)v)k =

vk k 6∈ i, jλvi + σvj k = i

−σvi + λvj k = j

Hence, we need to solve for σ and λ which satisfy:0 = λvj − σvi1 = λ2 + σ2

Taking the positive options:

λ =vi√

v2i + v2j

σ =vj√v2i + v2j

3. For a matrix A, we can easily find G1, . . . , Gn such that Gn · · ·G1A = Rwhere R is upper triangular. And, letting QT = Gn · · ·G1, we have theQR decomposition of A.

4. Givens Rotations are beneficial when A is sparse.

5. Pivoting (Partial or Complete) can be implemented to ensure that theelement on the diagonal of A is the largest in the row and column

29

5.4 Householder Reflections

1. A Householder reflection, H, is a matrix of the form I + τvvT where‖v‖ = 1 and τ ∈ C. It reflects any vector over the line tv : t ∈ R.

2. For use in QR decomposition, we require that HHT = 1 and HTH = 1.Since HT = H, we need to check only one:

HTH = I + 2τvvT + τ2 ‖v‖2 vvT

= I + τvvT (2 + τ ‖v‖2)

= I + τvvT (2 + τ)

This implies τ = −2

3. Moreover, given a vector v, we want Hz = αe1. Since H causes a reflec-tion, we choose

v =z + αe1‖z + αe1‖

Substituting this in, we have that:

(I − 2vvT )z = z − 2(z + αe1)(‖z‖2 + αw1)

‖z + αe1‖2= αe1

which holds if 2(‖z‖+αw1) = ‖z‖2 + 2αw1 +α2. This implies ±‖w‖ = α.

4. Taking the negative case, we have that the appropriate Householder Re-flection is:

I − 2(z − ‖z‖ e1)(z − ‖z‖ e1)T

‖z − ‖z‖ e1‖2

5. Letting Ap =[ap · · · amp

]Tof a matrix A ∈ Cm×n and permutation

matrices Π:

(a) Let A(1) ∈ Cm×n be the matrix we want to QR factorize

(b) Let H1 be the householder matrix such that H1A(1)1 =

∥∥∥A(1)1

∥∥∥ e1 ∈Cn. Let A(2) = H1A

(1).

(c) Let H2 be the householder matrix such that H2A(2)2 =

∥∥∥A(1)2

∥∥∥ e1 ∈Cn−1. Let

H2 =

[1 0

0 H2

]and A(3) = H2A

(2)

(d) Then, Hn · · ·H1A(1) = R an upper right triangular matrix. Letting

QT = Hn · · ·H1 gives us Q.

(e) If necessary, we can pivot the rows to achieve non zeros along thediagonals.

30

6 LU, LDU, Cholesky and LDL Decompositions

1. Both LU and LDU Factorization are based on Gaussian Elimination.

2. Given a system of equation Ax = b, we add multiples of the first row tothe subsequent rows to set them equal to zero.

Example 6.1. Let v = [ v1 · · · vn ]T . To set all vi = 0 for i 6= 1, wemultiply by:

L =

1 0 · · · 0 0−v2/v1 1 · · · 0 0

.... . .

......

. . ....

−vn/v1 0 · · · 0 1

3. This requires that the diagonal elements of a matrix are non-zero. Hence,

a sequence of pivots (partial or complete) resulting in:

MnΠnMn−1Πn−1 · · ·M1Π1A = U MnΠnMn−1Πn−1 · · ·M1Π1AΠ0 = U

where U is upper triangular.

4. To recover A or at least AΠ0 (in the partial case at least):

A = (MnΠnMn−1Πn−1 · · ·M1Π1)−1U

= ΠT1M

−11 ΠT

2 · · ·ΠTnM

−1n U

= ΠT1 ΠT

2 (Π2M−11 ΠT

2 ) · · ·ΠTnM

−1n U

= Π1ΠT2 · · ·ΠT

nL1 · · ·LnU

5. If A has non-singular principal submatrices, then LDU can be recoveredby performing LU decomposition on UT = (U ′)TD

6. If A is symmetric, LDLT is recovered.

7. It is difficult to check if A has non-singular principal submatrices (unlessA is positive definite)

8. Suppose A is symmetric, positive definite. Consider L to be lower trian-gular, we want to find A = LLT , and so we can compute the terms of Ldirectly from this relationship.

9. Similarly, we can find the algorithm form A = LDLT if A is symmetricpositive definite.

31

7 Iterative Methods

7.1 Overview

1. Iterative methods can be used to compute solutions to linear systems,least squares, eigenvalue problems, and singular value problems

2. Suppose we want to solve Ax = b. Iterative methods compute a sequencexk where xk → x = A−1b, and we can control the accuracy to stop theprocess

3. Classes of Iterative Methods

(a) Splitting/One-Step Stationary Methods

i. Split A = M −N where Mx = c is easy to solve for some c.

ii. Solve Mxk = Nxk−1 + b

(b) Semi-Iterative Methods

i. Generate, for a choice of B, yk = Byk−1 + c

ii. Then xk =∑kj=0 ajkyj

(c) Krylov Subspace Methods

i. Find iterates xk ∈ b, Ab,A2b, · · · , Ak−1bii. Uses the fact that eventually, Kr = b, Ab,A2b, · · · , Ar−1b will

be linearly dependent as r increases.

7.2 Splitting Methods

1. Overview

(a) Strategy: Suppose A is invertible in Ax = b. We find M such thatMx = c for some c is easy to solve and let N = M − A. We thenhave the following iteration scheme:

Mxk = Nxk−1 + b

(b) General Convergence of Errors

Proposition 7.1. Let ek = x−xk where x solves Ax = b. ‖ek‖ → 0for any initial x0 if and only if ρ(M−1N) < 1.

Proof. Note that:

xk = M−1Nxk−1 +M−1b

Since x = M−1Nx+M−1b (since (M −N)x = Ax = b):

ek = M−1Nek−1 =: Bek−1

Therefore, ek = Bke0. By Lemma 2.5, both directions follow.

2. Jacobi Method

32

(a) M = diag(A). As long as the diagonal elements of A are non-zero,then iterates can be explicitly computed as:

xik =1

aii

bi −∑j 6=i

aijxjk−1

(b) Jacobi Convergence of Errors

Corollary 7.1. A is strictly diagonally dominant then ek → 0.

Proof. We need only show that ρ(M−1N) < 1. This matrix is:

M−1N =

0 −a12/a11 · · · −a1n/a11

−a21/a22 0. . .

......

. . .. . .

...−an1/ann · · · −an,n−1/ann 0

Suppose A is strictly diagonally dominant, then for any row:∑

j 6=i

|aij | < |aii|

Using the fact that ρ(A) ≤ ‖A‖∞, we then have that

ρ(M−1N) ≤∥∥M−1N∥∥∞ < 1

Applying Proposition 7.1, the result follows.

3. Gauss-Seidel Method

(a) Notice that in the Jacobi Method, we can compute x1k, . . . , xi−1k be-

fore we compute xik. Gauss-Seidel uses these updated values to com-pute xk. This yields:

xik =1

aii

bi −∑j<i

aijxjk −

∑j>i

aijxjk−1

(b) Gauss-Seidel Convergence of Errors

Corollary 7.2. If A is strictly diagonally dominant then ek → 0.

Proof. Let L be the sub-diagonal entries of A and U be the superdiagonal entries of A. Then:

Mxk = b− Lxk − Uxk−1

Therefore, xk = −(M +L)−1Uxk−1 + (M +L)−1b. Hence, the errorsare:

ek = −(M + L)−1Uek−1

33

or explicitly:

eik = − 1

aii

∑j<i

aijejk +

∑j>i

aijejk−1

Let ri =

∑j 6=i

|aij ||aii| . By diagonal dominance, maxi ri =: r < 1. We

now proceed inductively to show that |eik| ≤ r ‖ek−1‖∞. When i = 1:

|e1k| ≤ r ‖ek−1‖∞Suppose this holds up to i− 1. Then:

|eik| ≤∑j<i

|aij ||aii||ejk|+

∑j>i

|aij ||aii||ejk−1|

≤ r ‖ek−1‖∞∑j<i

|aij ||aii|

+ ‖ek−1‖∞∑j>1

|aij ||aii|

≤ r ‖ek−1‖∞Therefore, ‖ek‖∞ ≤ rk ‖e0‖∞ → 0 as k →∞.

(c) Number of iterations

i. From the proof of the corollary, we have that ‖ek‖ ≤ rk ‖e0‖.ii. Given a tolerance ε > 0, and x0 = 0, we need only

k =

⌈log ε

log r

⌉to bound the relative error by ε.

4. Successive Over Relaxation (SOR)

(a) SOR is a generalization of Gauss-Seidel. Let A = D−L−U where Dis diagonal, L has only non-zero subdiagonal and U has only non-zerosuperdiagonals. The iteration is derived as follows:

ωDx = ωb− ωLx− ωUxDx = ωb− ωLx− ωUx+ (1− ω)Dx

Then, using the most recent estimates to compute the iteration as inGauss-Seidel:

Dxk = ωb− ωLxk − ωUxk−1 + (1− ω)Dxk−1

and, explicitly:

xik = (1− ω)xik−1 +ω

aii

bi −∑j<i

aijxjk −

∑j>i

aijxjk

(b) When ω = 1 this is the Gauss-Seidel method, and when ω > 1 this

is Successive Over Relaxation

(c) Ostrowki’s Lemma:

Lemma 7.1. Suppose A is symmetric positive definite. ω ∈ (0, 2) ifand only if ek → 0.

34

7.3 Semi-Iterative Methods

1. Richardson’s Method

(a) Richardson’s Method is an unstable numerical method with a param-eter α which updates iterates using:

xk = (I − αA)xk−1 + αb

Note 7.1. Effectively, Richardson’s Method is a line search methodfor minimizing 1

2xTAx − xT b. Given a starting point xk−1, then

the direction of steepest descent is b−Axk−1 (by taking derivatives).Therefore,

xk = xk−1 + α(b−Axk−1)

where α is the step length parameter.

(b) Richardson Convergence of Errors

Corollary 7.3. Suppose A is symmetric positive definite. ek → 0 ifand only if 0 < α < 1

µminwhere µmin is the smallest eigenvalue of A.

Moreover, ek → 0 optimally if 2µmin+µmax

.

Proof. We have that the errors are:

ek = (I − αA)ek−1

Hence, from Proposition 7.1, ek → 0 if and only if ρ(I − αA) < 1.Since A is symmetric positive definite, its eigenvalues are positive.Letting µ be a vector of the eigenvalues and µmax be the largest andµmin be the smallest. Letting A = Xdiag(µ)X∗ be the EVD of A,we have that the EVD of I − αA is:

X(I − αdiag(µ))X∗

Hence, 0 < ρ(I − αA) = 1− αµmin < 1. This implies the first result.

For optimality, note that the SVD and EVD of A coincide. Hence,‖I − αA‖2 = max |1−αµi| = ‖1− αµ‖∞. We want to minimize thisover α, which gives us the result.

2. Steepest Descent

(a) This method is akin to Richardson’s method, except at every step,we optimize αk so that we minimize the norm of b− Axk+1 (i.e. webring the slope closer to zero and hence closer to the stationary pointof 0.5xTAx+ bTx).

(b) Optimal Choice of αk

Lemma 7.2. Letting rk = b − Axk, the optimal choice of αk tominimize rk+1 is:

αk =rTk rkrTk Ark

35

Proof. Note that norms are equivalent in finite dimensional space.Hence, we can minimize the ‖·‖2, which, in the computation, willrequire determining A2, or we can minimize ‖·‖A−1 which overcomesthis cost. First:

rk+1 = b−Axk+1 = b−A(xk + αkrk) = rk − αkArk

Second:

rTk+1A−1rk+1 = rTk rk − 2αrTk rk + α2rTk Ark

Taking derivatives and noting that the quadratic coefficient is posi-tive, we have the result.

3. Chebyshev’s Method

(a) Notice that in the Steepest Descent Method:

ek = (I − αkA)(I − αk−1A) · · · (I − α0)e0 =: Pk(A)e0

Instead of optimizing over αk stepwise, Chebyshev’s method tries tooptimize over all α0, . . . , αk at each k.

(b) Since ‖ek‖ ≤ ‖Pk(A)‖ ‖e0‖ we want to minimize ‖Pk(A)‖(c) This is solved using Chebyshev’s Polynomials.

7.4 Krylov Space Methods

1. Overview

(a) Suppose we want to solve Ax = b and A is invertible. Hence, x ∈im(A−1). Computing the inverse is expensive, but we can more easilycompute Akc for some c implicitly.

(b) Krylov Subspace

Lemma 7.3. There is an r such that the solution to Ax = b, whenA is invertible, is in Kr(A).

Proof. From the Cayley-Hamilton theorem, there is a minimal poly-nomial of degree r,

Pr(x) =

r∑i=0

αixi

such that Pr(A) = 0. Hence:

A−1 =1

α0

r∑i=1

αiAi−1

Therefore, A−1b ∈ spanb, Ab,Ab2, . . . , Abr−1.

2. Conjugate Gradients

36

(a) Our goal is to minimize 12x

TAx− bTx, or, equivalently, solve Ax = bwhen A ∈ Rn×n is symmetric, positive definite.

(b) Conjugacy.

i. Conjugated Vectors.

Definition 7.1. Let 〈u, v〉A = uTAv and ‖v‖2A = 〈v, v〉A. A setof vectors p1, . . . , pr ⊂ Rn are Conjugated with respect to A iffor all i 6= j:

〈pi, pj〉 = 0

ii. Conjugated Vectors form a basis.

Lemma 7.4. Suppose A ∈ Rn×n is symmetric positive definiteand p1, . . . , pr are conjugated with respect to A. Then p1, . . . , prare linearly independent.

Proof. Suppose this is not true. Then there are α1, . . . , αr notall zero such that:

α1p1 + · · ·+ αrpr = 0

Then:

0 = (α1p1 + · · ·+ αrpr)TA(α1p1 + · · ·+ αrpr)

= α21pT1 Ap1 + · · ·+ α2

rpTr Apr

Since α2i ≥ 0 and pTi Api > 0, we have a contradiction.

(c) Conjugated Gradients

i. Let x0, . . . , xk be a sequence of iterates

ii. Their steepest descent directions are r0 = b − Ax0, . . . , rk =b−Axk

iii. We then create vectors p0, . . . , pk which are conjugated with re-spect to A from the descent directions r0, . . . , rk using a Gram-Schmidt type approach:

pk = rk −∑j<k

〈rk, pj〉A‖pj‖2A

pj

iv. We update xk+1 = xk + αkpk where αk is the

arg min1

2xTk+1Axk+1 − bTxk+1

(d) Relation to Krylov Spaces: The new axes system is an orthonormalbasis for the Krylov Subspace

(e) Geometric Interpretation: we rotate/dilate the axes system with re-spect to A and b, and each iterate minimizes along each “coordinate”of this new axis system. Since there are n coordinates, this requiresat most n iterates.

37