ch 1. applied linear algebraweb.yonsei.ac.kr/nipi/lecturenote/basic-linear algebra.pdfch 1. applied...

Ch 1. Applied Linear Algebra

Yoon Mo Jung

Computational Science and Engineering

Yonsei University

Ch 1. Applied Linear Algebra 2

§0 Prelude: Structure of Computational Science andEngineering

• Overview of Computational Science and Engineering

laws

rules

constraints

−−−−→modeling

Mathematical

problem:

equations

−−−−−−−−−−−−→Numerical procedures

Linear

equations

Ax = bin engineering and physics

Remark 1. Designing and validating numerical procedures is called “Nu-

merical Analysis”.

2. How to solve Ax = b is called Numerical Linear Algebra, which is the

heart of the Scientific Computing. Strang said “Its importance is

now recognized.”

• Four Simplifications

1. Nonlinear becomes linear.

ex) Bending of beam

u′′

(1 + (u′)2)3/2≈ u′′ if u′ is small.

2. Continuous becomes discrete.

ex) f ′(x) = limh→0

f(x + h)− f(x)h

≈f(xn+1)− f(xn)

∆x,

if ∆x is small.

3. Multidimensional becomes one-dimensional.

ex) ut = uxx (heat equation)

Computational Science & Engineering (CSE) Yoon Mo Jung


Let u(t, x) = T (t)X(x): separation of variables

4. Variable coefficients becomes constant coefficients.

ex) inhomogeneous heat equation: ut = (c(x)ux)x

with conductivity c(x).

If c(x) ≈ c, one may use Fourier transform or FFT.



§1 Applied Linear Algebra

§1.1 Four Special Matrices

• Two important problems1. Solving linear systems: Ax = b

x: cause or input, b: result or output.

ex) x: displacements, pressures, voltages, concentrations, ......

2. Eigenvalue problem: Ax = λx

eigen means prime.

• Four special matrices: Kn, Cn, Tn, Bn

K2 =

[2 −1−1 2

],K3 =

2 −1 0−1 2 −10 −1 2

,K4 =

2 −1 0 0−1 2 −1 00 −1 2 −10 0 −1 2

Remark (Good) Properties

1. Symmetric (Kij = Kji or K = KT ).

2. Sparse (lots of zeros if n≫ 1).

3. Tridiagonal, banded.

4. Constant diagonals: called Toeplitz matrix.

- Something is not changing when we move in space or time. Shift-

invariant or time-invariant.

5. Invertible (∃K−1 s.t. KK−1 = K−1K = I).-K−1 also symmetric but full.

- Important: We don’t want or need K−1 to find u = K−1f . All we

compute is the solution x.



6. Positive definite (xTAx > 0 if x ̸= 0).

Circulant matrix C4 =

2 −1 0 −1−1 2 −1 00 −1 2 −1−1 0 −1 2

Remark 1. Singular, not invertible (C4[1, 1, 1, 1]T = 0).

2. Positive semidefinite (xTAx ≥ 0).

T2 =

[1 −1−1 2

], T3 =

1 −1 0−1 2 −10 −1 2

Gaussian elimination:

T =

1 −1 0−1 2 −10 −1 2

−−−→Step 1 1 −1 00 1 −10 −1 2

−−−→Step 2 1 −1 00 1 −10 0 1

= U

U−1 =

1 −1 00 1 −10 0 1

−1

=

1 1 10 1 10 0 1

: The inverse of “difference matrix” is a “sum matrix”. 1 −1 00 1 −1

0 0 1

u1u2u3

= u1 − u2u2 − u3

u3 − 0

1 1 10 1 10 0 1

u1 − u2u2 − u3

u3 − 0

= u1u2u3



Remark: The inverse of triangular matrix is also triangular.

B2 =

[1 −1−1 1

], B3 =

1 −1 0−1 2 −10 −1 1

It is positive semidefinite.

B =

1 −1 0−1 2 −10 −1 1

−−−→Step 1 1 −1 00 1 −10 −1 1

−−−→Step 2 1 −1 00 1 −10 0 0

= U 1 −1 00 1 −10 0 0

111

= 000

and 1 −1 0−1 2 −1

0 −1 1

111

= 000

• Summary

1. Kn and Tn are invertible and positive definite.

2. Cn and Bn are singular and positive semidefinite.

The nullspace(kernel) is the constant vector u = [c, c, · · · , c].

Remark: Bu = f is solvable when f is perpendicular to e = [1, 1, · · · , 1].

f = Bu =

1 −1 0−1 2 −10 −1 1

u1u2u3u4

=

u1 − u2−u12u2 − u3−u2 + 2u3 − u4−u3 + u4

f1 + f2 + f3 + f4 = f

Te = f · e = ⟨f, e⟩ = 0.



Figure 1: Finite Differences

§1.2 Differences, Derivatives, Boundary Conditions

Observation: -1, 2, -1 produces a second difference.

Kn, Cn, Tn, Bn are all involved in approximating the equation

−d2u

dx2= f(x)

with boundary conditions at x = 0 and x = 1.

Part I: Finite Differences

-want to approximate dudx.

dudx≈ △u△x if △ x is small.

ex) Choose test function u(x) = x2.

Forward difference △+f(x) = u(x+h)−u(x)hex) (x+h)

2−x2h

= 2x + h

Backward difference △−f(x) = u(x)−u(x−h)hex) x

2−(x−h)2h

= 2x− hCentered difference △0f(x) = u(x+h)−u(x−h)2hex) (x+h)

2−(x−h)2h

= 2x

Taylor series: series in h



u(x + h) = u(x) + hu′(x) + h2/2u′′(x) + h3/3!u′′′(x) + · · ·u(x− h) = u(x)− hu′(x) + h2/2u′′(x)− h3/3!u′′′(x) + · · ·

u(x + h)− u(x)h

= u′(x) + h/2u′′(x) + · · ·

: It is first order accurate.

u(x + h)− u(x− h)2h

= u′(x) + h2/3!u′′′(x) + · · ·

: Centered is second order.

Centered difference matrix:

△0 =

. . .

−1 0 1−1 0 1

. . .

ui−1uiui+1ui+2

=

...

ui+1 − ui−1ui+2 − ui

...

Rmk: △T0 = −△0: antisymmetric (skew symmetric).The centered difference is the average of forward and backward.

• Second Differences from First Differences

△2ui =△−△+ui = 1/h[(

ui+1 − uih

)−

(ui − ui−1

h

)]=ui+1 − 2ui + ui−1

h2

△2u(x) = u(x+h)−2u(x)+u(x−h)h2

= u′′(x) + 2h2/4!u(4)(x) + · · ·: second order accuracy

• The Important Multiplications1. For constant and linear vectors, the second difference are zero:



△2(constant)

. . .

1 −2 11 −2 1

. . .

1

1

1

1

=

...

0

0...

△2(linear)

. . .

1 −2 11 −2 1

. . .

1

2

3

4

=

...

0

0...

For squares, the second difference is constant:

△2(squares)

. . .

1 −2 11 −2 1

. . .

1

4

9

16

=

...

2

2...

2. Second difference of the ramp vector produce the delta vector:

△2(ramp)

. . .

1 −2 11 −2 1

. . .

0

0

1

2

=0

1

0

0

= delta

3. Second difference of the sine and cosine and exponential produce 2 cos t−2 times those vectors.

△2(sines)

. . .

1 −2 11 −2 1

. . .

sin t

sin 2t

sin 3t

sin 4t

= (2cost− 2)

sin t

sin 2t

sin 3t

sin 4t

△2(cosines)

. . .

1 −2 11 −2 1

. . .

cos t

cos 2t

cos 3t

cos 4t

= (2 cos t−2)

cos t

cos 2t

cos 3t

cos 4t



△2(exponentials)

. . .

1 −2 11 −2 1

. . .

eit

e2it

e3it

e4it

= (2 cos t−2)

eit

e2it

e3it

e4it

Remark: sines or cosines or exponentials are eigenvectors of

K,T,B,C with the right boundary conditions.

Part II: Finite Difference Equations

−d2udx2

= f with boundary conditions u(0) = 0 and u(1) = 0.

Divide the interval [0, 1] into equal pieces of length h = △x.

unknown u =

u(h)

u(2h)...

u(nh)

=

u1u2...

un

where h = 1n+1.

Finite difference equation: −ui+1 − 2ui + ui−1

h2= fi

The first and last (i = 1, i = n) involves u0, un+1.

Ex) Solve −d2udx2

= 1 with u(0) = 0 and u(1) = 0,

−ui+1−2ui+ui−1h2

= 1, u0 = 0 and un+1 = 0.

Sol) Complete solution: ucomplete = uparticular + unullspace

Particular solution: −d2udx2

= 1 is solved by uparticular = −12x2.

Nullspace solution: −d2udx2

= 0 is solved by unullspace = Cx + D.

u(x) = −12x2 + Cx + D.

u(0) = 0: D = 0.



u(1) = 0: −12+ C = 0⇒ C = 1

2.

u(x) = +12x− 1

2x2.

This is special: the differential and difference equation have the same so-

lutions!

u(x) = −12x2 + 1

2x and ui =

12(ih− i2h2).

With h = 1/4,

Ku = f leads 16

2 −1 0−1 2 −10 −1 2

u1u2u3

= 111

⇒ 3/324/323/32

• A Different Boundary Condition

Ex) Solve −d2udx2

= 1 with dudx(0) = 0 (free end) and u(1) = 0.

−ui+1−2ui+ui−1h2

= 1, u1−u0h

= 0 and un+1 = 0.

Sol) u(x) = 12(1− x2).

We expect a O(h) error because of the forward difference u1−u0h

.

For n = 3, h = 1/4,

1/h2

1 −1 0−1 2 −10 −1 2

u1u2u3

= 111

gives h2 653

To have O(h2) accuracy, see the worked examples 1.2.A in the book.

99% of the difficulties with DE’s occurs at the boundary.



§1.3 elimination leads to K = LDLT

Two theme of the book:

1. How to understand equations.

2. how to solve them. - This section’s topic.

Ku = f :

u = K−1f theorectically, not computationally:

u = inv(K) ∗ f in MATLAB.

Solved by Gaussian elimination - LU decomposition.

If symmetric, K = LDLT (related to Cholesky factorization).

u = K\f in MATLAB. For multiple f ’s, [L, U] = lu(K).

Ex) Ku = f

2 −1 0−1 2 −10 −1 2

u1u2u3

= f1f2f3

2 −1 00 3/2 −10 −1 2

u1u2u3

= f1f2 + 1/2f1f3

by 2nd row +1/2× 1st row,

2 −1 00 3/2 −10 0 4/3

u1u2u3

= f1f2 + 1/2f1f3 + 2/3f2 + 1/3f1

by 3rd row + 1

3/22st row.

2u1 − u2 =f1,3/2u2 −u3 =f2 + 1/2f1,

4/3u3 =f3 + 2/3f2 + 1/3f1.Solution by backsubstitution.



Matrix-vector multiplication Ku as a combination of the columns of K: 2 −1 0−1 2 −10 −1 2

u1u2u3

= u1 2−1

0

+ u2 −12−1

+ u3 0−1

2

Solving a system Ku = f is exactly the same as finding a combination of

the column of K that produces the vector f .

Multiplier lij =entry to eliminate

pivot(in row i)(in row j)

The convention: Subtract lij times the pivot row j from row i. Then the

i, j entry is 0.

• Elimination Produces K = LU

K = LU

2 −1 0−1 2 −10 −1 2

= 1 0 0−1/2 1 0

0 −3/2 1

2 −1 00 3/2 −10 0 4/3

L reverses the elimination steps.

Suppose the forward elimination uses the multipliers in L to change the rows

of K in to the rows of U (Upper triangular). Then K is factored into L

times U .

Ku = f : LUu = f ⇒ u = U−1L−1fLc = f ⇒ c = L−1f (forward substitution)Uu = c⇒ u = U−1c (back substitution)

• Singular Systems



Ex) Circulant C =

2 −1 −1−1 2 −1−1 −1 2

2 −1 −10 3/2 −3/20 −3/2 3/2

2 −1 −10 3/2 −3/20 0 0

= U : The rows are linearly dependentAn invertible matrix has a full set of pivots.

No row exchange to get n pivots: A is invertible and A = LU .

Row exchange by P to get n pivots: A is invertible and PA = LU .

No way to find n pivots: A is singular, there is no inverse matrix A−1.

Pivoting matrix P : i row! j rowComposition of In−2×n−2 except i, j rows and columns and

[1

1

]for

i, j rows and columns.

Ex) 2 row! 3 row:

1

1

1

1

• Symmetry Converts K = LU to K = LDLT

K︸︷︷︸symmetric

= L︸︷︷︸lower triangular

U︸︷︷︸upper triangular

: not symmetric

K =

2 −1 0−1 2 −10 −1 2

= LU = 1 0 0−1/2 1 0

0 −3/2 1

2 −1 00 3/2 −10 0 4/3



=

1 0 0−1/2 1 00 −3/2 1

2 0 00 3/2 00 0 4/3

1 −1/2 00 1 −3/20 0 1

= LDLT : symmetric factorization

Remark 1. ATCA is symmetric if C is symmetric.

2. DA: rowwise multiplication by the diagonal entry in D.

3. AD: columnwise multiplication by the diagonal entry in D.

4. For any rectangular matrix A, the product ATA is square and symmet-

ric.

5. LDLT = L√D√DLT = (L

√D)(L

√D)T := L̃L̃T is called

Cholesky factorization.

• The Determinant Kn

detK =by definition of determinant ∼ n! : computationally useless!=detLU = detL︸︷︷︸

=1

detU

=product of diagonal entries of U

=2 · 3/2 · 4/3 · · · (n + 1)/n = n + 1

The LU decomposition is also a quick way to compute determinant.

Remark 1. If a matrix is tridiagonal, then L and U are bidiagonal.

2. If a row/column of K starts with p/q zeros (no elimination needed

there), then that low of L/ column of U also starts with p/q zeros.

3. Zeros inside the band can unfortunately be “filled in” by elimination -

It leads to fundamental problem of reordering the rows and columns to

make the p’s and q’s are as large as possible.



Figure 2: Left: The sparsity pattern of K for 2 dim. Right: The sparsity pattern of the Choleskyfactor of K.

• Positive Pivots and Positive Determinant

If is positive definite (xTAx > 0 if x ̸= 0) if all pivots are positive.[a b

b c

]=

[1

b/a 1

] [a

(ac− b2)/a

] [1 b/a

1

][a b

b c

]is positive definite iff a > 0 and ac− b2 > 0.

• Operation Counts

LU decomposition ∼ 2/3n3 in general, 1/3n3 if symmetric.

Operation Count Full Banded Tridiagonal

Factor: Find L and U ≈ 2/3n3 2w2n + wn 3nSolve: Forward and back on f 2n2 4wn + n 5n



§1.4 Inverse and Delta Functions

: Want to solve for f = point load.

Ku = δj =

0...

1...

0

(j th entry) = j th column of I.−u′′ = δ(x− a): Green’s function u.

Delta function δ(x):

δ(x) = 0 if x ̸= 0,∫∞−∞ δ(x)dx = 1 .

: not “true function”. “spike”, “point load”, “impulsive”concentrated at

x = 0, “infinitely tall and infinitely thin”

A sequence of functions generating or approximating Dirac delta:

fk(x) =

1

2k, if − k ≤ x ≤ k,

0, otherwise.∫∞−∞ fk(x) dx = 1 and ‘fk −−→k→0 δ ’.

Note that

K[u1|u2| · · · |un] = [δ1|δ2| · · · |δn]⇐⇒ KK−1 = I.

uj = column j of K−1. We are solving KK−1 = I column by column.

If we know the Green’s function for all point load δ(x − a), we can solve−u′′ = f for any load f(x).So, it is the “discrete Green’s function”.

K−1ij = the solution at point i from a load at point j .



0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1

Figure 3: Green’s function for fixed-fixed case

• Concentrated load

Ex) −d2udx2

= δ(x− a) with{fixed u(0) = 0 and fixed u(1) = 0

free u′(0) = 0 and fixed u(1) = 0.

Sol)∫ rightleft −

d2udx2

dx =∫ rightleft δ(x− a) dx

=⇒ −(dudx

)right

+(dudx

)left

= 1: The slope drops by 1.

Since u′′ = 0 except x = a,

u =

{Ax + B if x < a,

Cx + D if x > a.

Boundary Conditions Jump/No Jump Conditions at x = a

fixed u(0) = 0 : B = 0 No jump in u : Aa + B = Ca + D

fixed u(1) = 0: C + D = 0 Drop by 1 in u′ : A = C + 1

=⇒ u(x)︸︷︷︸u(x;a)

=

{(1− a)x if x < a,a(1− x) if x > a.

Remark u(x; a) is symmetric w.r.t. x and a. Note that its discrete version

K−1 is also symmetric (since K is symmetric).

• Delta Function and Green’ function

Delta ft δ(x)

δ(x)=dSdx←−−−−−−−−−−→∫ x−∞ δ(y)dy

Step ft S(x)

S(x)=dRdx←−−−−−−−−−−−→∫ x−∞ S(y)dy

Ramp ft R(x)



where S(x) =

{1 if x ≥ 00 if x < 0

and R(x) =

{x if x ≥ 00 if x < 0

.

The first derivative of the ramp function R(x) jumps by 1 at 0 and the

second derivative is a delta function.

Complete solution −d2u

dx2= δ(x− a) is solved by

u(x) = −R(x− a)︸︷︷︸particular solution

+ Cx + D︸︷︷︸null space u′′=0

0 = u(0) = −R(0− a) + C · 0 + D =⇒ D = 0.0 = u(1) = −R(1− a) + C +D = a− 1 + C =⇒ C = 1− a.

u(x)︸︷︷︸u(x,a)

= −R(x− a) + (1− a)x ={(1− a)x if x ≤ a,(1− x)a if x ≥ a.

The response at x to a load at a equals the response at a to a load at x:

symmetric.

cf) (K−1)ij = (K−1)ji

Free-Fixed: u′(0) = 0, u(1) = 1

u(x)︸︷︷︸u(x,a)

= −R(x− a) + (1− a)x ={1− a if x ≤ a,1− x if x ≥ a.

• Discrete Vectors: Load and Step and Lamp

The delta vector δ: δ = (· · · , 0, 0, 1, 0, 0, · · · )The step vector S: S = (· · · , 0, 0, 1, 1, 1, · · · )The Lamp vector R: R = (· · · , 0, 0, 0, 1, 2, · · · )Note that △−S = δ but △+R = S△2 = △−△+ so,

△2R = △−△+R = △−S = δ



△2(ramp)

. . .

1 −2 11 −2 1

. . .

0

0

1

2

=0

1

0

0

= deltaThe solution to △2u = 0 are “linear vectors” with ui = Ci + D.The complete solution to △2u = δ isui = Ri︸︷︷︸

uparticular

+Ci + D︸︷︷︸unullspace

cf) u(x) = Rx + Cx + D

Sampling the ramp u(x) at equally space points without any error.

• The Discrete Equations Ku = δj and Tu = δj

−△2u = δj: ui = −Ri−j + Ci + Du0 = −R0−j + C · 0 + D = 0 =⇒ D = 0un+1 = −Rn+1−j + C(n + 1) + 0 = 0

=⇒ C = n+1−jn+1

= 1−j

n + 1︸︷︷︸1−a

Fixed ends: ui = −Ri−j + Ci =

(n + 1− jn + 1

)i if i ≤ j,(

n + 1− in + 1

)j if i ≥ j.

Note: K−1n is symmetric. cf) u(x) =

{1− a if x ≤ a,1− x if x ≥ a.

Free-Fixed: ui = −Ri−j + (n+1− j) ={n + 1− j if i ≤ j,n + 1− i if i ≥ j.

cf) u(x) =

{1− a if x ≤ a,1− x if x ≥ a.



Green’s Function and Inverse Matrix

f =

f1f2f3

= f1 100

+ f2 010

+ f3 001

:A combination of n point loads

⇒

K−1f = f1 (column 1 of K−1)︸︷︷︸

K−1δ1

+ f2 (column 2 of K−1)︸︷︷︸

K−1δ2

+ f3 (column 3 of K−1)︸︷︷︸

K−1δ3

The load f(x) is an integral of point load f(a)δ(x− a).

−u′′ = f(x) =∫ 10

f(a)δ(x− a)da =⇒ u(x) =∫ 10

f(a)u(x, a)da

The Green’s function u(x, a) corresponds to “row x and column a of

continuous K−1.



§1.5 Eigenvalues Eigenvectors

Part I: Ax = λx and Akx = λkx and Diagonalizing A

• Matrix as a Linear TransformationA matrix A is considered as a linear transformation .

A :Rn →Rm

x 7→Ax

Definition of linear : A(rx + sy) = rAx + sAy

’Superposition Principle’

Example 1 A =

[1 0

0 1

]: identity, A =

[2 0

0 1

]: dilation, A =[

cos θ − sin θsin θ cos θ

]: rotation, A =

[1 0

0 −1

]: reflection.

• We may regard linear transformation as a composition of those dilations,rotations, and reflections, etc.

• Eigenvalues and Eigenvectors

Definition 2 λ: eigenvalue and x: eigenvector if

Ax = λx, x ̸= 0 .

Geometrically, along the eigen-direction, there is only scaling

or dilation.

More specifically, the special vector x lies along the same line as Ax. The

eigenvalue λ tells whether the vector x is stretched or shrunk or reversed or

left unchanged.

eigen: prime in German

Why are we interested in eigenvalues and eigenvectors?



Figure 4: Geometric interpretation of eigenvalue and eigenvector

- It reveals the ‘innate or invariant structure’ of A.

Especially, eigenvalues are invariant under change of basis.

We can understand the matrix A easily by observing the eigenstructure of

A, called spectrum.

The ‘easiest’ matrix:

I =

1 . . .1

: leave every entry unchanged.The ‘second easiest’ matrix:

D =

d1 . . .dn

D

x1...xn

= d1x1...dnxn

: cooridinatewise multiplication.Dx = b can be easily solved by division.

D is positive definite⇔ di > 0 for all i.xTDx =

∑ni=1 dix

2i = d1x

21 + · · ·+ dnx2n.

A ∼ D : A is ‘similar’ to D?Can we treat or understand A as a diagonal matrix?



Definition 3 A is similar to B if there exists a nonsingular X such that

X−1AX = B.

Remark 1. If A is similar to a diagonal matrix D, then X−1AX = D,

called diagonalization.

2. Since AX = XD, X is the eigenvectors of A with eigenvalues D.

3. X is nonsingular means the columns of X consists of a basis.

4. Diagonalization is equivalent to finding n eigenvalues and eigenvectors.

• Diagonalizing a matrixA is a n by n matrix with n independent eigenvectors x1, · · ·xn witheigenvalues λ1, · · ·λn, respectively.

AX =A[x1| · · · |xn] = [Ax1| · · · |Axn] = [λ1x1| · · · |λnxn]

=[x1| · · · |xn]

λ1 . . .λn

= XΛNow X−1AX = Λ. then A = XΛX−1: Consider Ay = XΛX−1y.

X−1: express v w.r.t the eigenbasis X.

Λ: dilation along each eigendirection.

X: send back to the original basis.

Thus, the role of X is change of basis.

When square matrices are diagonalizable?

1. In general, it is not guaranteed because there may not be n independent

eigenvectors.

Other decompositions: Jordan canonical form, Schur’s canonical form,

Singular Value decomposition (SVD).

2. If A is symmetric (for real-valued) or Hermite (for complex-valued), OK.



3. The weakest condition is possibly AAT = ATA, called normal matrix.

• Symmetric Matrices and Orthonormal EigenvectorsIf A is symmetric, there exist n independent eigenvectors x1, · · · , xn withn real-valued eigenvalues λ1, · · · , λn. Furthermore, they are orthogonal:

AU = UΛ and UTU = I

A = UΛUT or UTAU = Λ

Definition 4 The vectors ui, · · ·un are orthonormal if

⟨ui, uj⟩ = uTi uj = δij ={1 if i = j (normality),

0 if i ̸= j (orthogonality).

Definition 5 The square matrix U is orthogonal if UTU = I. i.e.

UTU =

uT1uT2...

uTn

u1 u2 · · · un

= 1 . . .

1

= I• Finding eigenvalues and eigenvectorsFinding x and λ satisfying Ax = λx is n equations with n+1 unknown.

We first try to find λ and next find x:

λ : Ax = λx, x ̸= 0⇔(A− λI)x = 0, x ̸= 0⇔A− λI is singluar⇔ det(A− λI) = 0: charateristic equation

det(A− λI) =cnλn + cn−1λn−1 · · ·+ c0=cn(λ− λ1)(λ− λ2) · · · (λ− λn)

by the Fundamental Theorem of Algebra.

Remark :

1. det(A− λI) ∼ O(n!), not useful computationally.



2. Furthermore, it is sensitive perturbation and rounding error. Also finding

roots is not easy.

3. Numerically power method or some other numerical algorithms are used.

Ex) eig(A) in MATLAB

4. One great success of numerical linear algebra is the development of fast

and stable algorithm to compute eigenvalues, especially for the symmet-

ric case.

Example 6 Consider the symmetric case K =

[2 −1−1 2

].

Sol)

det(K − λI) =∣∣∣∣ 2− λ −1−1 2− λ

∣∣∣∣ = (2− λ)2 − 1=λ2 − 4λ + 3 = (λ− 3)(λ− 1)

For λ = 1 : K − I =[

1 −1−1 1

](K − I)x = (K − I)[u; v] = 0⇒ u− v = 0

x1 =

[1

1

]or 1√

2

[1

1

].

For λ = 3 : K − 3I =[−1 −1−1 −1

](K − 3I)x = (K − 3I)[u; v] = 0⇒ u + v = 0

x2 =

[1

−1

]or 1√

2

[1

−1

].

Now

K =

[2 −1−1 2

]=

1√2

[1 1

1 −1

] [1 0

0 3

]1√2

[1 1

1 −1

]T

Note that K is a discrete approximation of

−d2udx2

with u(0) = 0 and u(1) = 0.



−d2u

dx2= λu⇔ K̃u = λu with K̃u = −

d2u

dx2

and

K̃ is linear . −d2u

dx2= λu,

u(0) = 0 and u(1) = 0.

Eigenfunctions: sin kπx, k = 1, 2, · · · .With h = 1/3, x0 = 0, x1 = 1/3, x2 = 2/3, x3 = 1,

sinπx ∼ [sinπx1; sinπx2] = [sin 1/3π; sin 2/3π] = 1√2[1; 1]and

sin 2πx ∼ [sin 2πx1; sin 2πx2] = [sin 2/3π; sin 4/3π] = 1√2[1;−1].

K = UΛUT , U =1√2

[1 1

1 −1

], Λ =

[1 0

0 3

].

K2 = (UΛUT )(UΛUT ) = UΛ2UT .

Kn = UΛnUT = U

[1 0

0 3n

]UT : Kn grows like 3n.

The product of n eigenvalues equals the determinant of A.

detA =n∏

i=1

λi

Proof) ‘determinant of products is product of determinants’.detA = det(UΛU−1) = detU detΛ detU−1

= det(UU−1) det Λ = detΛ.

Remark It is the constant term w.r.t. λ in det(A− λI).The sum of the n eigenvalues equal the sum of the n diagonal entries.

tr A =n∑

i=1

λi



Remark It is the coefficient of (−λ)n−1 in det(A− λI).

If A = XΛX−1 with no nonzero λi, then A is invertible and

A−1 = XΛ−1X−1 = X

1/λ1 . . .1/λn

X−1 ,which is an eigenvalue decomposition of A−1.

• The Power of a Matrix-Eigenvalues have their greatest importance in dynamic problems.

Example 7 Population problem: u(t + 1) = Au(t) each year where[u1(t + 1)

u2(t + 1)

]=

[0.8 0.3

0.2 0.7

] [u1(t)

u2(t)

].

The column sums are always 1, which means nobody is created or destroyed.

Furthermore, populations stay positive because has no negative entries.

This type of matrices are called Markov matrix which expresses probability

transition matrix.

Let u(0) =

[1000

0

]. The matrixA has eigenvectors

[600

400

],

[400

−400

]with eigenvalues 1, 1/2 respectively.

If Ax = λx, A2x = λ2x, · · · , Akx = λkx.Similarly, Ak(α1x1 + · · ·+ αnxn) = α1λk1x1 + · · ·+ αnλknxn.

u(t) = At[600

400

]=At

([600

400

]+

[400

−400

])=1t

[600

400

]︸︷︷︸steady state

+

(1

2

)t [400

−400

]︸︷︷︸

transient

−−−−→t→∞

[600

400

].

• Three steps to find uk = Aku0 from eigenvalues and eigen-vectors



Step 1 : Write u0 as a combination of the eigenvectors:

u0 = α1x1 + · · ·+ αnxnStep 2 : Multiply each number αj by (λj)

k.

Step 3 : Recombine the eigenvectors into

uk = α1λk1x1 + · · ·+ αnλknxn

In matrix,

Step 1 : u0 = [x1| · · · |xn]

α1...αn

= Xa, α = X−1u0.

Step 2 : Multiply

λk1 . . .λkn

α1...αn

= Λkα, ΛkX−1u0.

Step 3 : Recombine uk = [x1| · · · |xn]

λk1α1...λknαn

= XΛkαuk = XΛ

kX−1u0.

Remark

Ay = XΛX−1y

X−1y expresses y w.r.t. the basis X = [x1| · · · |xn].Λ multiplies eigenvalues.

X recombines.

• Application to Vector Differential equationsExample 8

dy

dt= ay general solution y(t) = Ceat.

y(0) = y0 It determines C.



The solution y(t) = y0eat decays if a < 0: stability.

The solution y(t) = y0eat grows if a > 0: instability.

When a is a complex number, its real part determines the growth or decay,

the imaginary part gives oscillatory factor since

eiwt = coswt + i sinwt (Euler formula)

Vectorial case or system of equations

dy

du= Au, u(0) = u0

Example 9

dy

dt=2y − z

dz

dt=− y + 2z

, ddt

[y

z

]=

[2 −1−1 2

]︸︷︷︸

K2

[y

z

]

Sol) Assume u(t) = eλtx = eλt[y

z

].

dudt

= λeλtx = LHS = RHS = Ku = eλtKx

Kx = λx : eigenvalue problem

x =

[1

1

],

[1

−1

]with λ = 1, 3 respectively.

General solution:

u(t) =c1eλ1tx1 + c2e

λ2tx2

=c1et

[1

1

]+ c2e

3t

[1

−1

]

u0 = u(0) = c1

[1

1

]+ c2

[1

1

]=

[1 1

1 −1

]︸︷︷︸

X

[c1c2

]Three steps for powers apply here too:

Expand u0 = Xα: α = X−1u0.



Multiply each αj by eλjt:

eλ1t . . .eλnt

a = eΛtX−1u0.Recombine into u(t) = XeΛtX−1u0.

Part II: Eigenvectors for Derivatives and Differences

−d2u

dx2= λu is solved by y = cosωx, y = sinωx with λ = ω2.

Analog to Kn: fixed-fixed case y(0) = 0, y(1) = 0.

y(x) = sin kπx with λ = k2π2, k = 1, 2, · · ·Sol) y = a cosωx + b sinωx

0 = y(0) = a

0 = y(1) = b sinω ⇒ ω = kπ: determined by boundary condition!

Analog to Bn: free-free case y′(0) = 0, y′(1) = 0.

y(x) = cos kπx with λ = k2π2, k = 0, 1, 2, · · ·

Analog to Cn: periodic case y(0) = y(1), y′(0) = y′(1).

y(x) = cos 2kπx, sin 2k̃πx with λ = 4k2π2, k = 0, 1, 2, · · · ,k̃ = 1, 2, · · ·

Analog to Tn: free-fixed case y′(0) = 0, y(1) = 0.

y(x) = cos(k + 1/2)πx with λ = k2π2, k = 0, 1, 2, · · ·

• Eigenvectors of Kn: Discrete Sines

−[sin(j − 1)θcos(j − 1)θ

]+2

[sin jθ

cos jθ

]−

[sin(j + 1)θ

cos(j + 1)θ

]=(2− 2 cos θ)

[sin jθ

cos jθ

]Computational Science & Engineering (CSE) Yoon Mo Jung


These are the imaginary and real parts of

−ei(j−1)θ + 2eijθ − ei(j+1)θ = (2− e−iθ − eiθ)eijθ .

The boundary rows decide θ everything!

For sin jθ,

From the first row,

2 sin θ − sin 2θ = (2− 2 cos θ) sin θ, it is true for any θ.From the last row,

− sin(n− 1)θ − 2 sinnθ = (2− cos θ) sinnθ,−(sinnθ cos θ−cosnθ sin θ)+2 sinnθ = 2 sinnθ−2 sinnθ cos θsinnθ cos θ + cos θ sinnθ = sinn(θ + 1) = 0

⇒ θ = kn+1

π, k = 1, 2, · · ·For cos jθ,

From the first row,

2 cos θ − cos 2θ = (2− 2 cos θ) cos θcos2 θ − 1 = 2 cos2 θ⇒ −1 = 0: There is no such θ.

The first eigenvector y1 will sample the first eigenfunction y(x) = sinπx

at n meshpoint with h = 1n+1

:

First eigenvector = dicrete sine y1 = (sinπh, sin 2πh, · · · , sinnπh)First eigenvalue of Kn:

λ1 = 2− 2 cosπh = 2− 2(1− π2h2

2+ · · · ) ≈ π2h2

To match differences with derivatives, divide K by h2 = (△x)2.

eigenvectors = discrete sines yk = (sin kπh, · · · , sinnkπh)eigenvalues of Kn : 2− 2 cos kπh, k = 1, · · ·n.



Discrete sine transform

DST =

sin π4 sin 2π4 sin 3π4sin 2π4 sin 4π4 sin 6π4sin 3π

4sin 6π

4sin 9π

4

=

1√2

1 1√2

1 0 −11√2−1 1√

2

Q = 1√

2DST is orthogonal, i.e. QTQ = I, Q−1 = QT .

Remark∫ 10 sinnπx sinmπxdx = 0 if n ̸= m.

• Eigenvectors of Bn: Discrete Cosines

eigenvalues of Bn : 2− 2 coskπ

n, k = 0, · · ·n− 1

eigenvectors: yk =

(cos

1

2

kπ

n, cos

3

2,kπ

n· · · , cos

(n−

1

2

)kπ

n

)Eigenvalues of B sample cos kπx at the n midpoints x = (j − 1

2)/n.

y′(0) = 0 ∼ y(x1)− y( x0︸︷︷︸ghost grid

) = 0

y′(1) = 0 ∼ y( xn+1︸︷︷︸ghost grid

)− y(xn) = 0

Since the cosine is even, those vectors have zero slope at the ends:

cos−12kπn

= cos 12kπn

and cos(n− 1

2

)kπn

= cos(n + 1

2

)kπn

: The reason for choosing midpoints as gridpoints.

Note that k = 0 gives the all-ones eigenvector y0 = (1, 1, · · · , 1) withλ = 0: DC vector with zero frequency.

Discrete cosine transform

DCT =

cos 0 cos 12π3 cos 12 2π3cos 0 cos 32π3 cos 32 2π3cos 0 cos 5

2π3

cos 522π3

= 1 12

√3 1

2

1 0 −11 −1

2

√3 1

2



• Eigenvectors of Cn: Powers of ω = e2πin

Eigenvectors of Cn: Both sine and cosine

(Euler formula) eiθ = cos θ + i sin θ

Circulant matrix (periodic) C4 =

2 −1 0 −1−1 2 −1 00 −1 2 −1−1 0 −1 2

It has constant diagonals with wrap-around.

The k th eigenvector of Cn comes from sampling yk(x) = ei2πkx at the

n meshpoints x = j./n, j = 0, · · · , n− 1.j th component of yk: e

i2πk(j/n) = ωjk where ω = ei2π/n = n th root

of 1.

eigenvalues of Cn : 2− ωk − ω−k = 2− 2 coskπ

n, k = 0, · · ·n− 1

eigenvectors: yk = (1, ωk, ω2k · · · , ω(n−1)k)

• The Fourier MatrixDiscrete Fourier transform (DFT)

F4 =

1 1 1 1

1 i i2 i3

1 i2 i4 i6

1 i3 i6 i9

, (Fn)jk = ωjk = ei2πjk/n

The columns are orthogonal in C: ⟨x, y⟩ = x∗y = x̄Ty.F̄4

TF4 = 4I so that F

−14 =

14F̄4

T.



In general,

F̄nTFn = nI and F

−1n =

1

nF̄n

T=

1

nF ∗n

.

Un =1√nFn: The normalized Fourier matrix is unitary.

Columns are ’orthonormal’ in C: ŪnTUn =

1√nF̄n

T 1√nFn = I.

Unitary matrix (Q∗Q = Q̄TQ = I) is the complex analog of orthogonal

matrix (ATA = I).



Figure 5: Quadratic function in 1-d example: x2 − x− 2

§1.6 Positive Definite Matrix

What is ‘positive definite’?

3 basic facts

1. K = ATA is symmetric and positive definite (or at least semidefinite).

xTATAx = (Ax)TAx

2. If K1 and K2 are positive definite, then so is K1 + K2.

3. All pivots and all eigenvalues of a positive definite matrix is positive.

Why do we want to consider positive definite matrices?

- It is closely related to the concept of energy as the quadratic form 12uTKu

and we are interested in its minimum.

Example 10 1 dimensional example (See Fig. 5)

f(x) = 12ax2 − bx + c︸︷︷︸

=0

= 12a(x− b

a)2 − 1

2b2

a.

Optimization: For its minimum,

The first necessary condition: f ′(x) = ax− b.The second sufficient condition: f ′′(x) = a > 0.

If K is positive definite,

the minimum of P (u) =1

2uTKu− uTf is Pmin = −

1

2fK−1f

when Ku = f.



Example 11 2 dimensional examples (See Fig. 6)

A =

[1 0

0 2

], B =

[1 0

0 0

], C =

[1 0

0 −2

], D =

[−1 00 −2

].

xTAx = [x y]

[1 0

0 2

] [x

y

]= x2 + 2y2 : positive definite (elliptic).

Similarly,

xTBx = x2: semipositive definite (parabolic),

xTCx = x2 − 2y2: indefinite (hyperbolic),xTDx = −x2 − 2y2 = −(x2 + 2y2): negative definite.

• Examples and Energy-based Definition

Quadratic function:

uTSu = [u1, u2]

[u1u2

]= au21 + 2bu1u2 + cu

2.

Example 12 Sum of squares examples (See also Fig. 6)Positive definite Semipositive definite Indefinite

K =

[2 −1−1 2

]B =

[1 −1−1 1

]M =

[1 −3−3 1

]2u21 − 2u1u2 + 2u22 u21 − 2u1u2 + u22 2u21 − 6u1u2 + 2u22

Always positive Positive or zero Positive or negative

2u21 − 2u1u2 + 2u22

=u21 + (u1 − u2)2 + u22 : A

TA

=2(u1 −1

2u2)

2 +3

2u22 : LDL

T .

K =

[1 −1 00 1 −1

] 1 0−1 10 1

= ATA=

[1 0

−12

1

] [2 0

0 32

] [1 −1

2

0 1

]= LDLT .



Figure 6: Positive definite, Indefinite, Semidefinite functions in 2 dim

u21 − 2u1u2 + u22 = (u1 − u2)

2.

2u21 − 6u1u2 + 2u22 = (u1 − 3u2)

2 − 8u22.

• Positive definiteness from ATA,ATCA,LDLT , QΛQT

1. K = ATA is symmetric positive definite iffA has independent columns:

uTKu = uTATAu = (Au)TAu = ∥Au∥ > 0 for x ̸= 0 if Ahas full rank.

2. K = ATCA is symmetric positive definite iff A has independent

columns and C is symmetric positive definite:

uTKu = uTATCAu = (Au)TCAu > 0 for x ̸= 0.

3. If symmetric K has a full set of positive pivots, it is positive definite:

K = LDLT , the diagonal pivot matrix D is positive definite and LT

has independent columns.

4. If a symmetric K has all positive eigenvalues in Λ, it is positive definite:

K = QΛQT , Q−1 = QT .

• Minimum Problem in n Dimensions

Very often, 12uTKu is the “internal energy” in the system.



P (u) =1

2uTKu− uTf : total energy

∇P = Ku− f = 0: the first necessary conditionH = K > 0, positive definite: the second sufficient conditon

P (K−1f) = 12(K−1f)TK(K−1f)− (K−1f)Tf = −1

2fTK−1f

P (u)− P (K−1f) =1

2uTKu− uTf − (−

1

2fTK−1f)

=1

2(u−K−1f)TK(u−K−1f) ≥ 0.

• Test for a minimum: Positive Definite Second DerivativesTest for 1 dimensional function:

f(x) = f(a)︸︷︷︸const

+ f ′(a)︸︷︷︸slope = 0

(x− a) +1

2f ′′(a)︸︷︷︸

concavity >0

(x− a)2 + · · ·

by Taylor series.

Test for n dimensional function:

P (u) = P (u∗)+(u−u∗)T∇P (u∗)+1

2(u−u∗)TH(u∗)(u−u∗)+· · ·

again by Taylor series.

For minimum,

(1st derivative vector: slope) ∇P (u∗) =

∂P∂u1...∂P∂un

= 0and

(2nd derivative matrix: concavity) Hij =∂2P

∂ui∂uj=

∂2P

∂uj∂ui= Hji > 0

i.e. positive definite.

• Newton method



Approximation by by quadratic form:

P (u) ≈P (u∗)︸︷︷︸=0

+(u− u∗)T︸︷︷︸uT

∇P (u∗)︸︷︷︸−f

+1

2(u− u∗)T︸︷︷︸

uT

H(u∗)︸︷︷︸K

(u− u∗)︸︷︷︸u

:=1

2uTKu− uTf

⇒ Ku = f.

H(u∗)(u− u∗) = −∇P (u∗).

Newton’s method: H(ui)(ui+1 − ui) = −∇P (ui)

If ui hits exactly a minimum u∗ (not too likely)∇P (u∗) = 0, so ui+1−ui = 0, no more steps.

It is an iterative method to solve a minimum problem.



§1.7 Numerical Linear Algebra: LU, QR, SVD

Ex) Ku = f or Kx = λx or Mu′′ + Ku = 0

Crucial properties of K: symmetric? banded? sparse? well-conditioned?

• Three Essential Factorization

1. A = LU = lower triangle matrix × upper triangle matrixby Gaussian elimination.

2. A = QR = Orthogonal matrix × upper triangle matrixby Gram-Schmidt orthogonalization or Householder transformation.

3. A = UΣV T = orthonormal columns × singular values × orthonor-mal lows

by singular value decomposition.

It is a ‘generalized eigenvalue decomposition’. cf) QΛQT .

• Orthogonal Matrices

⟨qi, qj⟩ = qTi qj = 0 if i ̸= j (orthogonality)⟨qi, qi⟩ = qTi qi = 1 (normalization to unit vector)Let Q = [q1|q2| · · · |qn].

QTQ =

qT1qT2...

qTn

q1 q2 · · · qn

= 1 . . .

1

= IThe inverse is its transpose: Q−1 = QT .

Length preserving (also angle preserving): ∥Qx∥ = ∥x∥



Ex) permutations, rotations, reflections.

Example 13 Permutation: the same rows as I, in a different order. 0 1 00 0 11 0 0

xyz

= yzx

P TP = IExample 14 Rotation

Rotation matrix in the 1− 3 plane:

cos θ 0 − sin θ0 1 0sin θ 0 cos θ

Example 15 Reflection: The reflection takes v to Hv on the other side

of a plane mirror. The unit vector u perpendicular to the mirror is reversed

into Hu = −u.Reflection matrix u = (cos θ, 0, sin θ):

H =I − 2uuT =

1 0 00 1 00 0 1

− 2 cos θ0

sin θ

[cos θ, 0, sin θ]=

1− cos2 θ 0 −2 sin θ cos θ0 1 0−2 sin θ cos θ 0 1− sin2 θ

= − cos 2θ 0 − sin 2θ0 1 0− sin 2θ 0 cos 2θ

detH = −1, Hu = (I − 2uuT )u = u− 2u = −u.It is a popular method for QR decomposition.

• Orthogonalization A = QR

1. Gram-Schmidt algorithm

Am×n =

a1 · · · an



Figure 7: Householder transformation

Figure 8: Gram-Schmidt orthogonalization

rank n, independent n vectors, consists of a basis for the column space of

A

q1 :=a1∥a1∥

, a1 = r11q1 with r11 =: ∥a1∥.B = a2 − (qT1 a2)q1 is orthogonal to q1.qT1 a2: projection in the q1 direction := r12.

q2 =B∥B∥ = r22.

Gram-Schmidt a1 an =

q1 q2 [ r11 r12

0 r22

]i.e.

a1 = r11q1

a2 = r12q1 + r22q2

2. Householder algorithm : I − 2uuT- used in MATLAB and popular numerical linear algebra package.

The great virtue of Q is its stability.

Qx = b is perfectly conditioned since ∥x∥ = ∥b∥ and an error △bproduces an error △x of the same size:Q(x +△x) = b +△b gives Q(△x) = △b and ∥△x∥ = ∥△b∥.



Singular Value Decomposition

Motivation: If Am×n is symmetric positive definite, QΛQT .

If a full-rank matrix A is not symmetric, furthermore, not square i.e. general

m× n, what can we do?

A︸︷︷︸m×n

= U︸︷︷︸m×m

Σ︸︷︷︸m×n

V T︸︷︷︸n×n

with UTU = I and V TV = I.

Note that ATA is symmetric positive definite.ATA︸︷︷︸n×n

=(UΣV T )T (UΣV T ) = V ΣTUTUΣV T

= V︸︷︷︸n×n

ΣTΣ︸︷︷︸n×n

V T︸︷︷︸n×n

= V

σ21 . . .σ2n

V T

:=QΛQT = Q

λ . . .λ

QT with Q = V, λi = σ2.It is an eigenvalue decomposition of ATA.

From AV = UΣ Avi = σiui ⇒ ui = Avi/σi.Since AAT = (UΣV T )(UΣV T )T = UΣΣTUT ,

ui are orthonormal eigenvectors of AAT .

Reduced SVD :

(rank r case)

Am×n = Um×r︸︷︷︸left singular vector

Σr×r UTr×n︸︷︷︸

right singular vector

=

u1 · · · ur σ1 . . .

σr

vT1...

vTr

with singular values σ1 ≥ σ2 ≥ · · ·σr > 0.MATLAB command: svd(A, 0).

To complete v’s, add any orthogonal basis vr+1, · · · , vn for nullspace ofA.



Figure 9: Reduced SVD of the full-rank A

To complete u’s, add any orthogonal basis ur+1, · · · , um for nullspace ofAT .

To complete Σ to an m by n matrix, add zeros.

Full SVD :

Am×n = Um×mΣm×nUTn×n

=

u1 · · · ur ur+1 · · · umσ1

. . .

σr

vT1...

vTrvTr+1...

vTn

MATLAB command: svd(A).

Figure 10: Full SVD of the full-rank A

A = u1σ1v1 + u2σ2v2 + · · ·urσrvr

Avj =

{σjuj for j ≤ r,0 for j > r.

ATuj =

{σjvj for j ≤ r,0 for j > r.



Figure 11: The transformation A in terms of SVD: ‘Fundamental Theorem of Linear Algebra’ byStrang.

Example 16 Find the SVD for A =

[1 1

7 7

].

Sol) ATA =

[1 7

1 7

] [1 1

7 7

]=

[50 50

50 50

].

det(ATA− λI) =∣∣∣∣ 50− λ 5050 50− λ

∣∣∣∣ = (50− λ)2 − 502=λ2 − 100λ = λ(λ− 100).

λ = 100, 0 with eigenvectors [v1, v2] =1√2

[1 1

1 −1

].

Now σ = 10, 0:

u1 = Av1/σ1 =110

[1 7

1 7

]1√2

[1

1

]= 1

10√2

[2

14

]= 1

5√2

[1

7

].

u2 =1

5√2

[−71

]since uT2u1 = 0 and ∥u2∥ = 1.

A = UΣV T =1

5√2

[1 −77 1

] [10 0

0 0

]1√2

[1 1

1 −1

](full)

=1

5√2

[1

7

]10

1√2

[1 1

](reduced)

.



Example 17 SVD of the n + 1 by n backward difference matrix △−.vk and uk are DST and DCT matrices. i.e. △− = (DCT)Σ(DST)T .Thus △−(DST)T = (DCT)Σ.cf)(sinnπx)′ = nπ cosnπx.

• The Pseudoinverse

If A = QΛQT = Q

λ1 . . .λn

QT with full lank, AQ = QΛ.A−1 = QΛ−1QT = Q

1/λ1 . . .1/λn

QT , A−1Q = QΛ−1.If Aqi = λqi, A

−1qi = 1/λiqi.

Similarly, Avi = σiui, then A−1ui = 1/σivi.

For a square and invertible A,

if A = UΣV T then A−1 = V Σ−1UT .

Now, if A is nonsquare or singular?

Pseudoinverse A+ = V Σ+UT , A+ui =

vi

σifor i ≤ r

0 for i > r

Example 18 Find the pseudoinverse A+ of A =

[1 1

7 7

].

Sol)



A+ = V Σ+UT =1√2

[1 −11 1

] [1/10 0

0 0

]1

5√2

[1 7

−7 1

](full)

=1√2

[1

1

]1/10

1

5√2

[1 7

](reduced)

=1/100

[1 7

1 7

] .

• The Condition Number and Norm

The condition number c(K) = λmaxλmin

for symmetric positive definite K.

It measures the “sensitivity” of the linear system Ku = f .

△f : measurement error, roundoff etc.Ku = f K(u +△u) = f +△f.The error equation: K△u = △f ⇒△u = K−1△f

∥△u∥ ≤ λmax(K−1)∥△f∥ =1

λmin(K)∥△f∥,

since it is maximized when △f = qmin. i.e. K−1qmin = 1λmin(K)qmin.The λmin indicates how close K to a singular matrix.

With c >> 1, λmin(cK) = cλmin(K), it is far away from singular.

But if we multiply K by 1000 for examle, then u and△u should be dividedby 1000. That rescaling to make K less singular and λmin larger cannot

change the reality of the problem.

The relative error ∥△u∥∥u∥ stays the same.

∥△u∥ ≤∥△f∥λmin(K)

.

∥f∥ = ∥Ku∥ ≤ λmax(K)∥u∥ ⇒1

∥u∥≤

λmax(K)

∥f∥.



∥△u∥∥u∥

≤λmax(K)

λmin(K)

∥△f∥∥f∥

.

Condition number for

symmetric positive definite Kc(K) =

λmax(K)

λmin(K)

Definition 19 Matrix norm (induced)

∥A∥ = maxx̸=0

∥Ax∥∥x∥

= max∥x∥=1

∥Ax∥

Remark The norm of a matrix measures the maximum stretching the matrix

does to any vector.

Figure 12: Matrix norm

Definition 20 Condition number

c(A) = ∥A∥∥A−1∥

∥Ax∥∥x∥

≤ ∥A∥ for all x ̸= 0⇒ ∥Ax∥ ≤ ∥A∥∥x∥.

∥AB∥ ≤ ∥A∥∥B∥ and ∥A + B∥ ≤ ∥A∥+ ∥B∥.



∥A∥2 = maxx ̸=0

∥Ax∥2

∥x∥2= max

x̸=0

xTATAx

xTx︸︷︷︸Raleigh quotient

= λmax(ATA) = σ2max.

c(A) = ∥A∥∥A−1∥ =σmax

σmin.

1 = ∥I∥ = ∥AA−1∥ ≤ ∥A∥∥A−1∥.λmax(A) ≤ σmax(A).

Example 21 A =

[0 2

0 0

].

|A− λI| =∣∣∣∣ −λ 20 −λ

∣∣∣∣ = λ2 = 0, λmax = 0.ATA =

[0 0

2 0

] [0 2

0 0

]=

[0 0

0 4

], σmax = 2 = ∥A∥.


ch 1. applied linear algebraweb.yonsei.ac.kr/nipi/lecturenote/basic-linear algebra.pdfch 1. applied...

Documents