matrix computation - purdue university

Chapter 3

Matrix Computation

The main objective of this chapter is to solve the linear system

Ax = f ,

where A is a square n× n matrix, and x and f are vectors of order n.

3.1 BasicsMatrix.

A =

a11 a12 · · · a1n

a21 a22 · · · a2n...

......

an1 an2 · · · ann

= [aij ] i, j = 1, 2, ..., n.

Column Vectors.

x =

x1

x2...

xn

, f =

f1

f2...

fn

Row Vectors.

xT = (x1, x2, . . . , xn)

fT = (f1, f2, . . . , fn)

Inner Products.

aT b = (a1, a2, . . . , an)

b1

b2...

bn

=

n∑

i=1

aibi ← scalar

Operation count: n multiplications, n− 1 additions $ 2n operations.

29

Chap. 3. Matrix Computation CS414 Class Notes 30

Outer Products.

abT =

a1

a2...

an

(b1, b2, . . . , bn) =

a1b1 a1b2 · · · a1bn

a2b1 a2b2 · · · a2bn...

......

anb1 anb2 · · · anbn

← matrix

Operation count: n2 multiplications = n2 operations.

3.1.1 Setting up matrix problems in MATLAB

Hilbert matrix. Suppose we want to construct a matrix A s.t.

Aij =1

i + j − 1

MATLAB

A = zeros(n,n);for i = 1:n,

for j = 1:n,A(i,j) = 1/(i+j-1);

end;end;

This is known as a Hilbert matrix, and can also be created in MATLAB by A = hilb(n). For n = 5, thismatrix is of the form

A =

1 1/2 1/3 1/4 1/51/2 1/3 1/4 1/5 1/61/3 1/4 1/5 1/6 1/71/4 1/5 1/6 1/7 1/81/5 1/6 1/7 1/8 1/9

Observing that the Hilbert matrix is symmetric, the above MATLAB code fragment could be modified totake advantage of this symmetry:

MATLAB

A = zeros(n,n);for i = 1:n,

for j = i:n,A(i,j) = 1/(i+j-1);A(j,i) = A(i,j);

end;end;

Another matrix. As another example of matrix set-up, consider the following MATLAB code:

MATLAB

P = zeros(n,n);P(:,1) = ones(n,1); % P(:,1) = 1st column of Pfor i = 2:n,

for j = 2:i,P(i,j) = P(i-1,j-1)+P(i-1,j);

end;end;


For n = 5, the above code generates the following matrix

P =

1 0 0 0 01 1 0 0 01 2 1 0 01 3 3 1 01 4 6 4 1

Next, we present two special matrices that can be created given a vector of n components, i.e., theelements of the matrix depend only on n parameters.

Vandermonde matrix. This a special type of matrix that can be created from a vector of n components.Given a vector

x =

x1

x2...

xn

we can create a matrix V as follows:

MATLAB

n = length(x);V(:,1) = ones(n,1);for j = 2:n,

V(:,j) = x .* V(:,j-1);end;

Over here, the jth column of V is obtained by element-wise multiplication of the vector x with the (j− 1)thcolumn. For n = 4, V is of the form

V =

1 x1 x21 x3

1

1 x2 x22 x3

2

1 x3 x23 x3

3

1 x4 x24 x3

4

Circulant matrix. Another matrix that can be created from a vector of size n is the circulant matrix.Given a vector

a =

a1

a2...

an

we can create a circulant matrix C as follows:

MATLAB

function C = circulant(a)n = length(a,1);C(1,:) = a’;for j = 2:n,

C(i,:)=[C(i-1,n) C(i-1,1:n-1)];end;

Over here, the ith row of C is obtained by permuting (i − 1)th row; the last element of (i − 1)th row ispermuted to the start of the ith row. For n = 4, C is of the form

C =

a1 a2 a3 a4

a4 a1 a2 a3

a3 a4 a1 a2

a2 a3 a4 a1


3.1.2 Structure of Matrices

The nonzero elements of a matrix determine its structure. Following are commonly occurring matrices.

! 0 0 0 00 ! 0 0 00 0 ! 0 00 0 0 ! 00 0 0 0 !

Diagonal

! ! 0 0 0! ! ! 0 00 ! ! ! 00 0 ! ! !0 0 0 ! !

Tridiagonal

! 0 0 0 0! ! 0 0 0! ! ! 0 0! ! ! ! 0! ! ! ! !

Lower triangular

! ! ! ! !0 ! ! ! !0 0 ! ! !0 0 0 ! !0 0 0 0 !

Upper triangular

! ! ! ! !! ! ! ! !0 ! ! ! !0 0 ! ! !0 0 0 ! !

Upper Hessenberg

A diagonal matrix can be constructed from a vector d, where

d = [d1 d2 d3 · · · dn];

by the MATLAB statement D = diag(d). This constructs the diagonal matrix D, s.t.,

D =

d1

d2

. . .dn

MATLAB also provides a way to specify diagonal entries other than the main diagonal. For example, considerthe following code:

MATLAB

m = 3; k = 2;v = [10 20 30];A = diag(v,k);

This constructs a matrix A with the vector v forming the kth diagonal of A:

A =

0 0 10 0 00 0 0 20 00 0 0 0 300 0 0 0 00 0 0 0 0

Also, the statement v = diag(A,k) extracts the kth diagonal in the form of a vector. A negative k refersto diagonals below the main diagonal while k = 0 refers to the main diagonal.

Block structures. Matrices with block structure can also be constructed in MATLAB. A block structuredmatrix consists of elements that are in turn smaller matrices. For example, a block structured matrix of theform

A =

A11 A12

(3× 3) (3× 2)

A21 A22

(2× 3) (2× 2)

can be constructed from the following code

MATLAB


A11 = rand(3,3);A12 = rand(3,2);A21 = [1 2 3 ; 4 5 6];A22 = [7 8 ; 9 10];A = [A11 A12 ; A21 A22];

3.2 Matrix Operations

Matrix-Vector Multiplication. Given an m× n matrix A and a vector x of size n, we wish to computethe vector y of size m s.t.

y = Ax

where

A =

a11 a12 · · · a1n

a21 a22 · · · a2n...

...am1 am2 · · · amn

x =

x1

x2...

xn

gives the vector y with elements

yi =n∑

k=1

aikxk i = 1, 2, . . . , m

Operation count: mn multiplications, m(n− 1) additions $ 2mn operations.

MATLAB

y = zeros(m,1);for i = 1:m,

for j = 1:n,y(i) = y(i) + A(i,j) * x(j)

end;end;

This can be computed in a number of ways.Alternative 1: (row-oriented)

A =

aT1

aT2...

aTm

, x =

x1

x2...

xn

⇒ y =

aT1 x

aT2 x...

aTmx

The corresponding MATLAB program uses m inner products:

MATLAB

for i = 1:m,y(i) = A(i,:) * x;

end;

Alternative 2: (column-oriented)

A = (b1,b2, . . . ,bn), x =

x1

x2...

xn

⇒ y =

n∑

j=1

xjbj

The corresponding MATLAB program uses summation of n vectors, i.e., “scalar * vector + vector”:

MATLAB


for j = 1:n,y = y + x(j) * A(:,j);

end;

Matrix-Matrix Multiplication. Given an m× r matrix A and an r × n matrix B we wish to computethe product C of size m× n s.t.

C = A ∗ B↓ ↓ ↓

(m× n) (m× r) (r × n)

The matrix C can be represented as

C = [cij ],i = 1, 2, . . . , mj = 1, 2, . . . , n

where

cij =r∑

k=1

aikbkj

Operation count: mnr multiplications, mn(r − 1) additions $ 2mnr operations.

MATLAB

C = zeros(m,n);for j = 1:n,

for i = 1:m,for k = 1:r,C(i,j) = C(i,j) + A(i,k)*B(k,j);

end;end;

end;

Alternative 1: (inner products)

A =

aT1

aT2...

aTm

, B = (b1,b2, . . . ,bn) ⇒ cij = aT

i bj

MATLAB

for j = 1:n,for i = 1:m,

C(i,j) = A(i,:)*B(:,j);end;

end;

Alternative 2: (outer products)

A = (g1,g2, . . . ,gr) B =

hT1

hT2...

hTr

, ⇒ C =

r∑

k=1

gkhTk ← m× n matrix

MATLAB


for k = 1:r,C = C + A(:,k)*B(k,:);

end;

Alternative 3: (matrix-vector products)

B = (b1,b2, . . . ,bn) ⇒ C = (Ab1, Ab2, . . . , Abn)

MATLAB

for j = 1:n,C(:,j) = A*B(:,j);

end;

3.2.1 Matrix Norm, Inverse, and Sensitivity of Linear Systems

Inverse of a Matrix. If A is a square matrix of order n, then A is nonsingular if there exists a uniquen× n matrix X such that

AX = XA = I =

11

. . .1

The inverse, X , is also denoted by A−1. The following property holds for inverse of a product of two matrices:

(A ∗B)−1 = B−1 ∗A−1.

Note also that(A ∗B)T = BT ∗AT .

The identity matrix I is often expressed as

I = (e1, e2, . . . , en), eTi = (0, 0, . . . , 0, 1, 0, . . . , 0),

i.e., ei has a one in the ith position and zero everywhere else.

Vector and matrix norms. Suppose that x′ and x′′ are two approximations of x. Let

x =

111

, x′ =

1.011.011.01

, x′′ =

1

1.021

.

Based on the errors in the approximation, can we decide which one is a better approximation of x?

∆x′ = x′ − x =

0.010.010.01

, ∆x′′ = x′′ − x =

0.00.020.0

.

A norm is a mapping that assigns to each vector a nonnegative real number, i.e., it is a generalizationof the scalar absolute value. We define the maximum norm or the infinity norm of a vector as the largestabsolute value of all the elements, i.e.,

‖x‖ = max1≤i≤n

|xi| = ‖x‖∞

In MATLAB this is computed by norm(x,inf).In our example,

‖∆x′‖ = 0.01 ‖∆x′′‖ = 0.02.


Thus, using the max-norm, we conclude that x′ is a better approximation of x. (The outcome could bedifferent if we use a different norm.)

Norms can be defined for matrices in a similar way. The matrix norm corresponding to the vectormax-norm is given by,

‖A‖ = max1≤i≤n

n∑

j=1

|aij | = ‖A‖∞

Clearly, this equals the maximum row sum of the absolute values of the matrix. In MATLAB this is computedby norm(A,inf).

Example 3.1

A =(

1 −20 1

)⇒ ‖A‖ = max{3, 1} = 3

!

A number of norms can be defined. The most common of these are presented below:

1. ‖x‖1 =n∑

i=1

|xi|.

2. ‖x‖2 = (xT x) 12 .

3. ‖A‖1 = max1≤j≤n

n∑

i=1

|aij | (max. col. sum).

4. ‖A‖F =

∑

j

∑

i

|aij |2

12

(Frobenius).

5. ‖A‖2 = max‖x‖2=1

‖Ax‖2 (spectral).

Properties of norms.

1. ‖x‖ > 0; moreover, ‖x‖ = 0 if and only if x = 0.

2. ‖αx‖ = |α|‖x‖.

3. ‖x + y‖ ≤ ‖x‖+ ‖y‖.

4. ‖A‖ > 0; moreover, ‖A‖ = 0 if and only if A = 0.

5. ‖αA‖ = |α|‖A‖.

6. ‖Ax‖ ≤ ‖A‖‖x‖.

7. ‖A + B‖ ≤ ‖A‖+ ‖B‖.

Theorem 3.1 If A is the stored version of the matrix A on a computer with ε unit roundoff, then A = A+Ewhere ‖E‖ ≤ ε ∗ ‖A‖.

Proof We know thatA = [aij ], aij = fl(aij) = aij(1 + εij)


with |εij | ≤ ε (unit roundoff). Now,

‖E‖ = ‖A−A‖

= max1≤i≤n

n∑

j=1

|aij − aij |

≤ max1≤i≤n

n∑

j=1

|aijεij |

≤ ε ∗ max1≤i≤n

n∑

j=1

|aij |

= ε ∗ ‖A‖.

!

Residual and Error Vectors. Consider the linear system Ax = f . Suppose that the computed solutionis x. Then we define the following:

• residual: r = f −Ax

• error: δx = x− x

Therefore,r = f −Ax = Ax−Ax = A(x− x) = −Aδx

In some situations one requires a small residual, i.e., ‖r‖ ≤ tolerance, and the size of the error is of noconsequence. In other situations, however, one needs to obtain small errors, i.e., ‖δx‖ ≤ small quantity. Forexample, consider the linear system

(0.780 0.5630.913 0.659

)(x1

x2

)=(

0.2170.254

).

The solution is given by

x =(

1−1

).

Consider an approximate solution xa

xa =(

0.341−0.087

), ra =

(10−6

0

), δxa = xa − x =

(−0.659

0.913

).

The residual and error norms are ‖ra‖ = 10−6 and ‖δxa‖ $ 0.9, respectively. On the other hand, for anothercomputed solution xb,

xb =(

0.999−1.000

), rb =

(0.0007800.000913

), δxb = xb − x =

(−0.001

0

),

Clearly, in the first case, the solution is meaningless in spite of the smaller residual norm!

Problem Sensitivity. In the preceding 2× 2 system, the matrix

A =(

0.780 0.5630.913 0.659

)

has a determinant, det(A) $ 10−6. In fact

A =(

0.780 0.563001095 . . .0.913 0.659

)


is exactly singular; i.e., a perturbation of roughly 10−6 of the element in the (1,2) position renders theproblem insoluble. In other words, the above problem is ill-conditioned.

Consider the hypothetical situation in which there is no roundoff error during the entire solution processexcept when A and f are stored. Thus, we have the system

(A + ∆A)(x + ∆x) = (f + ∆f).

It can be shown that‖∆x‖‖x‖ ≤ β ∗

[‖∆A‖‖A‖ +

‖∆f‖‖f‖

]

where the “magnification factor” β is given by

β =κ(A)

1− κ(A)‖∆A‖‖A‖

in whichκ(A)

‖∆A‖‖A‖ < 1.

Here,κ(A) = ‖A‖‖A−1‖

is called the condition number of A with respect to the max. norm. Note that κ(A) ≥ 1. To see that, recallthat AA−1 = I, and

‖I‖ = 1 = ‖AA−1‖ ≤ ‖A‖‖A−1‖.

Moreover, if‖∆A‖‖A‖ ≤ ε,

‖∆f‖‖f‖ ≤ ε, [ε = unit roundoff]

then‖∆x‖‖x‖ <∼ κ(A) ∗ ε,

i.e., ‖∆x‖‖x‖ is less than a quantity that is proportional to κ(A) ∗ ε. Furthermore, if κ(A) ∗ ε is close to 1 then

A is “numerically” singular.

Example 3.2 Consider the linear system Ax = f in which the representation error of A is given by

‖∆A‖‖A‖ $ 10−16.

If κ(A) $ 1011, give an upper bound on the rel. error in the solution ‖∆x‖/‖x‖.

Solution‖∆x‖‖x‖ ≤

κ(A)1− κ(A)‖∆A‖

‖A‖

[‖∆A‖‖A‖

]$ 1011

1− 10−5(10−16) $ 10−5.

!

Example 3.3 Let

A =(

4.1 2.89.7 6.6

). A−1 =

(−66 28

97 −41

).

Verify that AA−1 = I. Assuming that the relative perturbation in the right-hand side is

‖∆f‖‖f‖ $ 0.000613,

obtain an upper bound for the rel. error in the solution ‖∆x‖/‖x‖.


Solution We have‖A‖ = max{6.9, 16.3} = 16.3‖A−1‖ = max{94, 138} = 138κ(A) = (16.3)(138) $ 2249.4

Now,‖∆x‖‖x‖ ≤ κ(A)

[‖∆f‖‖f‖

]$ 1.38

In fact, if

x =(

11

), x =

(0.062.38

), i.e., ∆x =

(−0.94

1.38

),

then‖∆x‖‖x‖ = 1.38

exactly! !

3.3 Solution of Linear Systems

3.3.1 Gaussian Elimination

Consider the system of equations:

3x1 + 6x2 + 9x3 = 39 Eq.(1)2x2 + 5x2 − 2x3 = 3 Eq.(2)x1 + 3x2 − x3 = 2 Eq.(3)

These equations can be solved by the process of Gaussian Elimination. Gaussian Elimination consists of twostages - forward elimination and back substitution.

Stage I. Forward elimination. The goal of this stage is to use the equations numbered (1)–(k) toeliminate the terms with unknowns x1, x2, . . . , xk from equations numbered (k+1)–(n). This is achieved inn− 1 steps. For our example system, the steps are as follows.

Step 1: Eliminate x1 from Eq.(2) by adding − 23∗ Eq.(1) to Eq.(2), and from Eq.(3) by adding − 1

3∗Eq.(1) to Eq.(3). The linear system becomes:

3x1 + 6x2 + 9x3 = 39 Eq.(1)0 + x2 − 8x3 = −23 Eq.(2)0 + x2 + 4x3 = −11 Eq.(3)

Note that the right-hand side of Eq.(2) and Eq.(3) are also updated similarly.

Step 2: Eliminate x2 from Eq.(3) by subtracting Eq.(2) from Eq.(3). Now, the linear system is

3x1 + 6x2 + 9x3 = 39 Eq.(1)x2 − 8x3 = −23 Eq.(2)

4x3 = 12 Eq.(3)

At the end of forward elimination stage, Eq. (k) has terms with unknowns xk, . . . , xn only, i.e., the linearsystem consists of an upper triangular matrix. The next stage computes the values of unknowns in reverseorder, i.e., xn, xn−1, . . . , x1, by using equations in that same order.


Stage II. Back substitution. The linear system can be rewritten as follows:

3x1 + 6x2 + 9x3 = 39 Eq.(1)x2 − 8x3 = −23 Eq.(2)

4x3 = 12 Eq.(3)

Step 1: Obtain x3 from Eq. (3), i.e., x3 = 3.

Step 2: Obtain x2 from Eq. (2), i.e., x2 = 8x3 − 23 = 1.

Step 3: Substitute x2 and x3 in Eq. (1) to get x1, i.e.,

3x1 = 39− 6x2 − 9x3 = 6 ⇒ x1 = 2.

The solution obtained at the conclusion of the algorithm is

x =

213

.

Gaussian Elimination in Matrix Notation. Our example can be written in matrix form:

3 6 92 5 −21 3 −1

x1

x2

x3

=

3932

,

where

A =

3 6 92 5 −21 3 −1

f =

3932

.

The operations in the forward elimination stage can also be represented by matrices multiplying A. LetAi−1x = fi−1 be the linear system at the start of step i, and let Mi be the matrix that is multiplied to Ai−1

to obtain Ai.

Stage I. Forward elimination.

Step 1: At the start of this step, we have the linear system A0x = f0, where A0 = A and f0 = f . Thetransformation of the linear system at this step is given by M1A0x = M1f0. It is easy to see that

1− 2

3 1− 1

3 1

3 6 92 5 −21 3 −1

x1

x2

x3

=

1− 2

3 1− 1

3 1

3932

,

⇒

3 6 90 1 −80 1 −4

x1

x2

x3

=

39−23−11

and thus,

M1 =

1− 2

3 1− 1

3 1

Step 2: The next transformation is given by M2A1x = f1, where A1 and f1 are obtained in theprevious step. Observe that

1

1−1 1

3 6 9

1 −81 −4

x1

x2

x3

=

1

1−1 1

39−23−11


⇒

3 6 9

1 −84

x1

x2

x3

=

39−23

12

and thus,

M2 =

1

1−1 1

The linear system at the end of this step is A2x = f2. The matrix A2 is an upper triangular matrix,and the corresponding system is solved by back substitution.

Gaussian Elimination for General n. Let us develop the Gaussian Elimination algorithm to solve thelinear system Ax = f where A is a nonsingular matrix of size n× n and x and f are vectors of size n. Theforward elimination stage consists of n− 1 steps to convert A into an upper triangular matrix.

Forward elimination

A0 = A, f0 = ffor k = 1, 2, . . . , n− 1

MkAk−1x = Mkfk−1

[Ak = MkAk−1, fk = Mkfk−1]end;

Back-substitution is used to solve the upper triangular system obtained from the forward elimination stage:An−1x = fn−1.

Let us now compute the matrices Mk, k = 1, 2, . . . , n− 1. Suppose

A0 = A =

a(0)11 a(0)

12 · · · a(0)1n

a(0)21 a(0)

22 · · · a(0)2n

a(0)31 a(0)

32 · · · a(0)3n

......

...a(0)

n1 a(0)n2 · · · a(0)

nn

.

where the symbol a(k)ij is used to denote the value of the element aij at the start of the kth step.

Forward elimination step 1. The matrix M1 that transforms all the elements below a11 in the firstcolumn of A0 to zero is given by

M1 =

1−m21 1−m31 1

.... . .

−mn1 1

where

mi1 =a(0)

i1

a(0)11

, i = 2, . . . , n

The transformed matrix A1 = M1A0 is given by

A1 =

a(0)11 a(0)

12 a0)13 · · · a(0)

1n

0 a(1)22 a(1)

23 · · · a(1)2n

0 a(1)32 a(1)

33 · · · a(1)3n

......

......

0 a(1)n2 a(1)

n3 · · · a(1)nn


wherea(1)

ij = a(0)ij −mi1a

(0)1j ,

i = 2, . . . , nj = 1, . . . , n

It is important to note that the first row remains unchanged.

Forward elimination step 2. At this step, M2 is used to transforms all the elements below a22 in thesecond column of A1 to zero. This is given by

M2 =

11

−m32 1−m42 1

.... . .

−mn2 1

,

where

mi2 =a(1)

i2

a(1)22

, i = 3, . . . , n


A2 =

a(0)11 a(0)

12 a(0)13 · · · a(0)

1n

0 a(1)22 a(1)

23 · · · a(1)2n

0 0 a(2)33 · · · a(2)

3n...

......

...0 0 a(2)

n3 · · · a(2)nn

wherea(2)

ij = a(1)ij −mi2a

(1)2j ,

i = 3, . . . , nj = 2, . . . , n

Observe that at this step, the first and second rows remain unchanged.

Forward elimination step k. At step k, the matrix Ak−1 has the following structure

Ak−1 =

a(0)11 a(0)

12 a(0)13 · · · a(0)

1k · · · a(0)1n

a(1)22 a(1)

23 · · · a(1)2k · · · a(1)

2n

a(2)33 · · · a(2)

3k · · · a(2)3n

. . ....

...0 a(k−1)

kk · · · a(k−1)kn

......

a(k−1)nk · · · a(k−1)

nn

and the matrix Mk used to transform all the elements below akk in the kth column of Ak−1 to zero is givenby

Mk =

1. . .

1−mk+1,k 1−mk+2,k 1

.... . .

−mnk 1

,


where

mik =a(k−1)

ik

a(k−1)kk

, i = k + 1, . . . , n

The elements of the transformed matrix Ak = MkAk−1 are

a(k)ij =

a(k−1)ij −mika(k−1)

kj , i, j = k + 1, . . . , n0, i ≥ k + 1, j ≤ k

a(k−1)ij , i ≤ k

The reader is encouraged to verify that the kth column below diagonal in Ak is zero.At the kth step, the right-hand side is also transformed by Mk, and the resulting vector fk = Mkfk−1 is

given as

f (k)i =

{f (k−1)

i −mikf (k−1)k , i = k + 1, . . . , n

f (k−1)i , i ≤ k

The forward elimination algorithm has the potential to break down when a(k−1)kk = 0 or a(k−1)

kk ≈ 0. Thissituation can be rectified by using pivoting techniques; this will be discussed in later sections.

Back substitution. At this stage, we need to solve the upper triangular system Ux = g, where

U = An−1 =

a(0)11 a(0)

12 · · · a(0)1n

a(1)22 · · · a(1)

2n. . .

...a(n−1)

nn

g = fn−1 =

f (0)1

f (1)2

...f (n−1)

n

Gaussian Elimination in MATLAB. We now present a MATLAB program for the forward eliminationand back substitution stages of Gaussian Elimination.

MATLAB

% FORWARD ELIMINATIONm = zeros(n,1);for k=1:n-1,

% Compute kth column of M{k}m(k+1:n) = A(k+1:n,k)/A(k,k);

% Multiply M{k} with A{k-1}for i=k+1:n,

A(i,k+1:n)=A(i,k+1:n)-m(i)*A(k,k+1:n);end;

% Update f{k}f(k+1:n) = f(k+1:n) - m(k+1:n) * f(k);

end;U = triu(A); % Extract upper triangular part

% BACK SUBSTITUTIONx(n) = f(n)/U(n,n);for k=n-1:-1:1,

f(1:k) = f(1:k)-U(1:k,k+1)*f(k+1);f(k) = f(k)/U(k,k);

end;

The operation count for the forward elimination stage is given below:


Matrix Operations:

Div:n−1∑

k=1

(n− k) =12n(n− 1) (compute Mk)

Mult:n−1∑

k=1

(n− k)2 =16n(n− 1)(2n− 1) (compute Ak)

Add:n−1∑

k=1

(n− k)2 =16n(n− 1)(2n− 1) (compute Ak)

Total: Tu = 23n3 − 1

2n2 − 16n

Right-Hand Side:

Mult:n−1∑

k=1

(n− k) =12n(n− 1)

Add:n−1∑

k=1

(n− k) =12n(n− 1)

Total: Tf = n(n− 1)

The operation count for the back substitution stage is given below:

Div: n

Mult:n−1∑

k=1

(n− k) =12n(n− 1)

Add:n−1∑

k=1

(n− k) =12n(n− 1)

Total: Tb = n2

Recall the formulas for summation:m∑

j=1

j =12m(m + 1),

m∑

j=1

j2 =16m(m + 1)(2m + 1)

Thus, the overall time for the solution of a linear system of order n using Gaussian Elimination is

T ∗ = Tu + Tf + Tb =23n3 +

32n2 − 7

6n

In general, we say that

• Gaussian Elimination takes O(n3) operations.

• Back-substitution takes O(n2) operations.

3.3.2 Triangular Factorization: LU Decomposition

Let us again consider the system Ax = f . The forward elimination process may be written as:

Mn−1Mn−1 · · ·M2M1A = An−1 = U.

Multiplying both sides by the matrix M−11 M−1

2 · · ·M−1n−1, we have

A = M−11 M−1

2 · · ·M−1n−1U

Recall that each Mk is a lower triangular matrix. In fact, Mk is a unit lower triangular matrix which is alower triangular matrix with 1’s on diagonal. Furthermore, each M−1

k is unit lower triangular, and so is theproduct matrix M−1

1 · · ·M−1n−1. Let L be the unit lower triangular matrix defined as

L = M−11 M−1

2 · · ·M−1n−1,


then we haveA = LU.

With this result, we have expressed a general nonsingular matrix A as a product of a unit lower and anupper triangular matrix. The result would be much more useful if we could construct L easily.

We now describe how to obtain L from the forward elimination stage. The following assertions are easyto verify.

• It can be shown that

M−1k =

1. . .

1mk+1,k 1mk+2,k 1

.... . .

mnk 1

,

i.e., M−1k is identical to Mk with the exception that the sign on mik is reversed.

• It can be also be shown that

M−1k M−1

k+1 =

1. . .

1mk+1,k 1

mk+2,k mk+2,k+1. . .

......

mnk mnk+1 1

• Finally, using the previous result, it can be shown that

L =

1m21 1m31 m32 1

......

. . ....

... 1mn1 mn2 mn,n−1 1

.

Solution of Linear Systems Using LU Decomposition. The factorization of a matrix A into thetriangular factors L and U can be used to solve the linear system Ax = f . Since A = LU , we need to solvethe system

LUx = f .

Assuming y = Ux, the solution consists of the following two steps.

1. Solve Ly = f .

2. Solve Ux = y.

The reader can verify that the first step corresponds to the transformation of f during the forward eliminationstage, and that indeed y = fn−1.

The operation count for LU decomposition of a matrix is

Tu <23n3


and for the solution of the triangular systems with matrices L and U is

Ts < 2n2.

The advantage of computing LU decomposition over Gaussian Elimination is evident when solving asystem with multiple right-hand sides. For example, when solving AX = F , where F is an n ×m matrix,the operation count is

T ∗LU = Tu + Ts ≈

23n3 + 2mn2

whereas using Gaussian Elimination for each right-hand side, the operation count would have been

T ∗GE =

23mn3,

which is a factor of m more!

Example 3.4 Compute the LU decomposition of A from our earlier example:

A =

3 6 92 5 −21 3 −1

We know that

M1 =

1− 2

3 1− 1

3 1

, M2 =

1

1−1 1

, U =

3 6 90 1 −80 0 4

Thus,

L = M−11 M−1

2 =

123 113 1

1

11 1

=

123 113 1 1

!

3.3.3 Pivoting for Stability

Gaussian elimination, as described above, will break down if a(k−1)kk = 0 for any k = 1, . . . , n−1. the following

example illustrates this for a 2× 2 linear system.

Example 3.5 Consider the system0 ∗ x1 + 1 ∗ x2 = 11 ∗ x1 + 1 ∗ x2 = 2

or (0 11 1

)(x1

x2

)=(

12

)

with the exact solution:x =

(11

).

Clearly, Gaussian elimination fails in the first step since a(0)11 = 0. If we exchange the rows, however, we get

(1 10 1

)(x1

x2

)=(

21

)

and proceed with back substitution. Row 2, which is now row 1, is called the “pivotal row”. !

An important concept in numerical computations is the following:


If a mathematical process fails when a particular parameter achieves a certain value, then thecorresponding numerical algorithm will most likely be unstable when his parameter is near thecritical value.

A slight perturbation in the linear system from the previous example serves to illustrate this point.

Example 3.6 Consider the linear system Ax = f where

A =(

10−4 11 1

)f =

(12

).

Solve the above linear system with Gaussian elimination, using 3-digit decimal arithmetic with rounding.

Solution The first step in forward elimination gives

m21 = 1/10−4 = 104, M1 =(

1 0−104 1

)

andfl(M1A) =

(10−4 1

0 fl(1− 104)

), f l(M1f) =

(1

fl(2− 104)

).

But,fl(1− 104) = fl[105(−0.100000) + 101(0.100000)]

= fl[105(−0.100000 + .000010)]= fl[105(−.099990)]= −105(0.100) = −104,

f l(2− 104) = −104.

Thus, we have (10−4 1

0 −104

)(x1

x2

)=(

1−104

)

and the approximate solution is

x =(

01

).

The exact solution, however, is given by

x =(

1.0001000100010.999899989998

).

Now, if we exchange rows so that |m21| < 1, i.e., m21 = 10−4, then

fl(A1) =(

1 10 fl(1− 10−4)

)=(

1 10 1

)fl(f1) =

(21

),

and the solution isx =

(11

),

which is a much better computed solution! !

The pivoting strategy adopted in this example is known as partial pivoting. Such a strategyexchanges rows so that all multipliers mik are less than or equal to 1 in magnitude.

Observe that in Example ?? m21 = 104 is very large compared to the sizes of the original elements of thelinear system. As a result, elements of A1 (or Ak, in general) can grow unacceptably large. These, in turn,can result in loss of accuracy in fixed precision arithmetic, giving inaccurate results.


Partial Pivoting Strategy. At the kth step of Gaussian elimination, find row i, i ≥ k, for which

|a(k−1)ik | ≥ |a(k−1)

jk |, j = k, . . . , n

and exchange it with row k. This ensures that |mjk| < 1 at this step. Such a row permutation is appliedat each step to ensure that |mik| < 1 for all k. In most cases this keeps the growth of elements of Ak,k = 1, 2, . . . , n− 1 within acceptable limits.

Triangular Factorization with Partial Pivoting. To demonstrate this strategy, let us consider the LUdecomposition of the matrix

A =

1 0 12 5 −23 6 9

.

We will also use a vector, IPIV , to keep track of the row exchanges made during the factorization. Thesteps in the forward elimination proceed as follows.

Step 1: Row 3 is the pivotal row since

max1≤i≤3

|a(0)i1 | = a(0)

31 = 3;

therefore, we exchange rows 1 and 3, and initialize the first element of IPIV with 3. Now, we have

A0 =

3 6 92 5 −21 0 1

, IPIV =

3••

.


A1 = M1A0 =

1− 2

3 1− 1

3 1

3 6 92 5 −21 0 1

=

3 6 90 1 −80 −2 −2

Storing the multipliers −m21 and −m31 in A1, we get

A1 =

3 6 923 1 −813 −2 −2

Step 2: Again, row 3 is the pivotal row since

max2≤i≤3

|a(1)i2 | = a(1)

31 = 2;

therefore, we exchange rows 2 and 3, and initialize the second element of IPIV with 3. Now, we have

A1 =

3 6 923 −2 −213 1 −8

, IPIV =

33•

.


A2 = M2A1 =

1

112 1

3 6 9−2 −21 −8

=

3 6 9−2 −2

0 −9


Storing the multiplier −m32 in A2, we get

A2 =

3 6 923 −2 −213 − 1

2 −9

Instead of the factorization A = LU , we have computed a factorization of a matrix whose rows have beenpermuted according to the vector IPIV . In other words, we have

PA = L ∗ U,

where

L =

123 113 − 1

2 1

, U =

3 6 9−2 −2

−9

, P =

0 0 11 0 00 1 0

.

Note that the permutation matrix P is obtained by applying the row exchanges stored in IPIV to the identitymatrix I. The exchanges must be applied sequentially in the order specified by IPIV (i), for i = 1, . . . , n−1.

IPIV =

33•

← row to be exchanged with row 1← row to be exchanged with row 2← not used

Now, let us solve the linear system Ax = f , where A is given above, and

f =

2518

.

We know that PAx = P f , or LUx = P f , and so we solve

LUx = P f = f =

1825

.

Forward Sweep: Solve Ly = f

113 123 − 1

2 1

y1

y2

y3

=

1825

⇒y1 = 18y2 + 6 = 2 ⇒ y2 = −4y3 + 2 + 12 = 5 ⇒ y3 = −9

Backward Sweep: Solve Ux = y

3 6 9−2 −2

−9

x1

x2

x3

=

18−4−9

⇒x3 = 1−2x2 − 2 = −4 ⇒ x2 = 13x1 + 6 + 9 = 18 ⇒ x1 = 1

Therefore,

x =

111

.

Next, we give the MATLAB programs for the triangular factorization and solution of triangular systems.


Triangular factorization with partial pivoting.

function [L,U,IPIV] = GErowpiv(A);[n,n] = size(A);IPIV = 1:n;L = eye(n,n);for k=1:n-1

[maxv,r] = max(abs(A(k:n,k))); % max value in columnq = r+k-1; % pivot rowIPIV([k q]) = IPIV([q k]); % Exchange IPIV valuesA([k q],:) = A([q k],:); % Exchange rows of Aif A(k,k) ~= 0

L(k+1:n,k) = A(k+1:n,k)/A(k,k);A(k+1:n,k+1:n) = A(k+1:n,k+1:n)

- L(k+1:n,k)*A(k,k+1:n);end

endU = triu(A);

Solving lower triangular system.

function y = LowerTriSolve(L,f)n = length(f);y = zeros(n,1);for j=1:n-1

y(j) = f(j)/L(j,j);f(j+1:n) = f(j+1:n)-L(j+1:n,j)*y(j);

endy(n) = f(n)/L(n,n);

Solving upper triangular system.

function x = UpperTriSolve(U,y)n = length(y);x = zeros(n,1);for j=n:-1:2,

x(j) = y(j)/U(j,j);y(1:j-1) = y(1:j-1)-x(j)*U(1:j-1,j);

end;x(1) = y(1)/U(1,1);

The MATLAB code that solves a linear system Ax = f using triangular factorization with partial pivotingis given as

MATLAB

[L,U,IPIV] = GErowpiv(A); % IPIV has row permutationy = LowerTriSolve(L,f(IPIV)); % f(IPIV) is permuted rhsx = UpperTriSolve(U,y);

Let us pay a closer look at GErowpiv(A). At stage k, after partial pivoting, let Ak be of the form

Ak =

! ! ! ! ! !o ! ! ! ! !

o o a(k)33 ! ! !

o o a(k)43 ! ! !

o o a(k)53 ! ! !

o o a(k)63 ! ! !

, Mk =

11

1! 1! 1! 1

,


i.e.,

Ak =(

U1 F1

0 Bk

)Mk =

(I2 00 Mk

)

Therefore,

Ak+1 = MkAk =(

U1 F1

0 MkBk

)

Let us focus on Mk ∗Bk:(

1 0m I3

)(a(k)33 bT

c Ek

)=

(a(k)33 bT

a(k)33 m + c Ek + mbT

),

Note that m = −c/a(k)33 ; in fact, −m is computed by the MATLAB statement A(k+1:n,k)=A(k+1:n,k)/A(k,k).

Also, Ek + mbT is computed by the statement A(k+1:n,k+1:n)=A(k+1:n,k+1:n)-A(k+1:n,k)*A(k,k+1:n).

Example 3.7 How should one compute α = cT A−1d ?

Solution SincePA = LU,

taking the inverse of both sides, we get

A−1P−1 = U−1L−1 ⇒ A−1 = U−1L−1P.

Therefore, α = cT U−1L−1Pd. Let g = Pd,

• solve Ly = g,

• solve Ux = y, and

• compute α = cT x.

Alternately, using MATLAB functions developed earlier,

MATLAB

[L,U,IPIV] = GErowpiv(A);g = d(IPIV);y = LowerTriSolve(L,g);x = UpperTriSolve(U,y);alpha = c’*x;

!

Iterative Refinement. An important technique to improve the accuracy of a computed solution relieson repeated solution of linear systems with successive residuals as the right hand side. In other words, onesolves the linear system Ax(2) = f − Ax(1), where x(1) is the initial solution. The procedure is outlinedbelow:

Iterative Refinement

1. Compute PA = LU .2. Solve LUx = P f = f , i.e.,

Ly = f ,Ux = y ⇒ x [computed solution].

3. Let x(1) = x; the residual is r = f −Ax(1) = A(x− x(1)) = −Aδx(1).4. Solve Aδx(1) = −r for error in solution.5. Compute improved solution: x(2) = x(1) − δx(1) (repeat if needed)


Example 3.8 Consider the system Ax = f , i.e.,(

20 2000002 2

)(x1

x2

)=(

2000004

)

The triangular factorization yields

L =(

1 00.1 1

), U =

(20 2000000 −20000

), IPIV =

(1•

)

Thus, the computed solution is

x(1) =(

01.0

).

Using 4-digit decimal arithmetic the residual is computed as

r(1) = f −Ax(1) =(

02.0

).

Solving Aδx(1) = −r(1) for the error δx(1), we get

δx(1) =(−1.00.0001

)⇒ x(2) = x(1) − δx(1) =

(1.00.9999

).

Compute r(2) and show it is much smaller than r(1). !

3.4 Least Squares Problems

Least Squares Fitting. LS fitting problems arise in many applications. Consider a linear model forcomputing the height of a plant as a linear function of four nutrients, i.e.,

h = a1x1 + a2x2 + a3x3 + a4x4,

where h is the plant height, and a1, a2, a3, and a4 are nutrient concentrations in the soil. The unknownparameters x1, x2, x3, and x4 can be determined from m observations of height and concentrations form > 4. Typically, a large number of experiments are conducted to obtain the values for h, a1, a2, a3, anda4. Suppose the observations for the ith experiment are given by hi, ai1, ai2, ai3, ai4. If the model is perfect,then

hi = ai1x1 + ai2x2 + ai3x3 + ai4x4

for i = 1, 2, . . . , m. In matrix notation we would then have h = Ax, or,

h1

h2...

hm

=

a11 a12 a13 a14

a21 a22 a23 a24...

......

...am1 am2 am3 am4

x1

x2

x3

x4

,

where A is an m× 4 matrix.In general, however, the model will not be perfect, making it impossible to find that vector x which

exactly satisfies the above equation. In such situations, the main objective is to obtain an x for which

‖h−Ax‖2

is minimized. Recall that for a vector y, ‖y‖22 = yT y.LS fitting arises also in the approximation of known functions. Suppose we wish to approximate f(x) =√

x on [0.25,1] via a linear approximation of the form

l(y) = α + βy


Thus, given the tablex x1 x2 · · · xm√x f1 f2 · · · fm

in which fi = √xi, determine the best least squares fit parameters α and β. Let,

r = f −Aa =

f1

f2...

fm

−

1 x1

1 x2...

...1 xm

(αβ

),

then we seek to determine a so that ‖r‖22 = ‖f −Aa‖22 is a minimum.

Least Squares Problem. The least squares problem is stated as follows:

minx‖f −Ax‖22

where A is an m × n matrix with m ≥ n, and f , x are vectors of order m and n, respectively. Also, weassume that A has linearly independent columns, i.e., A is of full column-rank.

We seek a factorization of A of the form,

QT A =(

R0

)

where R is an n×n upper triangular matrix, and Q is orthogonal, i.e., QQT = QT Q = Im. Since the vector2-norm is invariant under orthogonal transformations, i.e.,

‖QT r‖22 = (QT r)T (QT r) = rT QQTr = rT r = ‖r‖22

we have,

‖f −Ax‖22 = ‖QT (f −Ax)‖22

=∥∥∥∥

(g1

g2

)−(

R0

)x∥∥∥∥

2

2

=∥∥∥∥

(g1 −Rx

g2

)∥∥∥∥2

2

This quantity is minimized for x = x, for which Rx = g1; then x is called the least squares solution, and‖r‖2 = ‖f −Ax‖2 = ‖g2‖2.

Next, we describe how to construct the orthogonal factorization

QT A =(

R0

).

Consider the 2× 2 orthogonal matrix

G =(

c s−s c

)

in which c = cos θ and s = sin θ. It can be easily verified that

GGT = GT G =(

c2 + s2 00 c2 + s2

)=(

1 00 1

).

In fact, given a vector x of size 2, we can construct G such that the vector y = Gx has zero as the secondelement. To do this, let (

c s−s c

)(x1

x2

)=(

y1

0

),


then−sx1 + cx2 = 0 ⇒ s

c=

x2

x1

Using the identity c2 + s2 = 1, we have

1 +x2

2

x21

=1c2

that further gives the following expression for c and s:

c =x1√

x21 + x2

2

s =x2√

x21 + x2

2

Note thaty1 = cx1 + sx2 =

√x2

1 + x22 = ‖x‖2.

Now, we can construct a 2×2 orthogonal matrix that introduces a zero in the second position of a vectorof size 2. We use this idea to introduce zeros in the lower triangular part of A to obtain the required uppertriangular form by repeated multiplication with suitable 2× 2 orthogonal matrices. The matrix

G(1)m−1,m =

1. . .

1c(1)m−1,m s(1)

m−1,m

−s(1)m−1,m c(1)

m−1,m

is used to introduce a zero in the last position, i.e., mth location in the first column. After that, the matrix

G(1)m−2,m−1 =

1. . .

1c(1)m−2,m−1 s(1)

m−2,m−1

−s(1)m−2,m−1 c(1)

m−2,m−1

1

is used to introduce a zero in the (m− 1)th position of the first column. The process of introducing zeros inthe first column is continued by forming the matrix G(1)

j−1,j , for j = m− 2, m− 3, . . . , 2, that introduce zerosin the jth location. Let us define QT

1 to be the product of these matrices, i.e.,

QT1 = G(1)

1,2 · · ·G(1)m−2,m−1G

(1)m−1,m

The same technique can be used to zero out the elements below the diagonal in the second column,followed by the third column all the way up to the last column. Note that introducing zeros in this mannerdoesn’t effect the zeros introduced earlier. Let G(k)

j−1,j be the matrix used to zero out the element in the(j, k) position. Let us also define

QT2 = G(2)

2,3 · · ·G(2)m−2,m−1G

(2)m−1,m

......

QTn = G(n)

n−1,n · · ·G(n)m−2,m−1G

(n)m−1,m,

then,

QTn · · ·QT

2 QT1 A =

(R0

).


Note that since each G(k)j−1,j is orthogonal, each QT

k is also orthogonal. Finally, defining

QT = QTn · · ·QT

2 QT1 ,

we have the required orthogonal matrix that transforms A into an upper triangular matrix. The sametransformations applied to f give

g = QTn · · ·QT

2 QT1 f .

For example, if A is a 5 × 3 matrix (m=5, n=3), QT1 introduces zeros below the diagonal in the first

column of A, QT2 introduces zeros below the diagonal in the second column, and so on until

QTn · · ·QT

1 A =(

R0

).

In other words, zeros in locations 1, 2, 3, and 4 are introduced by QT1 , zeros in locations 5, 6, and 7 are

introduced by QT2 , and zeros in locations 8 and 9 are introduced by QT

3 .

! ! !

4 ! !

3 7 !

2 6 91 5 8

MATLAB

function [xLS,res] = LSq (A,f)[m,n] = size(A);for j=1:n,

for i=m:-1:j+1x = [A(i-1,j),A(i,j)];[c,s] = x/norm(x);A(i-1:i,j:n) = [c s; -s c]*A(i-1:i,j:n);f(i-1:i) = [c s; -s c]*f(i-1:i);

end;end;xLS = UpperTriSolve(A(1:n,1:n), f(1:n))if m==n,

res = 0;else

res = norm(f(n+1:m));end;

matrix computation - purdue university

Documents