108systems of le5.1 systems of linear equations 5 solving …€¦ · 113systems of le5.3...

108 Systems of LE 5.1 Systems of Linear Equations

5 Solving Systems of Linear Equations

5.1 Systems of Linear Equations

System of linear equations:

a11x1 +a12x2 + ...+a1nxn = b1

a21x1 +a22x2 + ...+a2nxn = b2

. . . . . . . . . . . . . .

am1x1 +am2x2 + ...+amnxn = bm

109 Systems of LE 5.1 Systems of Linear Equations

Matrix form: A- given matrix, vector b - given; vector of unknowns xa11 a12 · · · a1n

a21 a22 · · · a2n...

... . . . ...am1 am2 · · · amn

x1

x2...

xn

=

b1

b2...

bm

or Ax = b (8)

Suppose,

• m > n – overdetermined system; Does this system have a solution?a system with more equations than unknownshas usually no solution

• m < n – underdetermined system; How many solutions does it have?a system with fewer equations than unknownshas usually infinitely many solutions

• m = n – what about the solution in this case?a system with the same number of equations and unknowns has usuallya single unique solution←− we will deal only with this case now

110 Systems of LE 5.2 Classification

5.2 Classification

Two main types of systems of linear equations: systems with

• full matrix

– most of the values are nonzero

– how to store it?storage in a 2D array

• sparse matrix

– most of the matrix values are zero

– How to store such matrices?storing in a full matrix system would be waste of memory

∗ different sparse matrix storage schemes

Quite different strategies for solution of systems with full or sparse matrices


5.2.1 Problem Transformation

• Common strategy is to modify the problem (8) such that

– the solution remains the same

– modified problem – more easy to solve

What kind of transformations do not change the solution?

• Possible to multiply the both sides of the equation (8) with an arbitrary nonsin-gular matrix M without the change in solution.

– To check it, notice that the solution of MAz = Mb is:

z = (MA)−1Mb = A−1M−1Mb = A−1b = x.


– For example, M = D – diagonal matrix,

– or M = P permutation matrix

NB! Although, theoretically the multiplication of (8) with nonsingular matrix M doesnot change the solution, we will see later that it may change the numerical process ofthe solution and the exactness of the solution...

The next question we ask: what type of systems are easy to solve?

113 Systems of LE 5.3 Triangular linear systems

5.3 Triangular linear systemsIf the system matrix A has a row i with a nonzero only on the diagonal, it is easy

to calculate xi = bi/aii; if now there is row j where except the diagnal a j j 6= 0 theonly nonzero is at position a ji, we find that x j = (b j− a jixi)/a j j and again, if thereexist a row k such that akk 6= 0 and akl = 0 if l 6= {i, j}, we can have xk = (bk−akixi−ak jx j)/akk etc.

• Such systems – easy to solve

• called triangular systems.

With rearranging rows and unknowns (columns) it is possible to transform the systemto Lower Triangular form L or Upper Triangular form U :

L =

`11 0 · · · 0`21 `22 · · · 0...

... . . . ...`n1 `n2 · · · `nn

, U =

u11 u12 · · · u1n

0 u22 · · · u2n...

... . . . ...0 0 · · · unn

114 Systems of LE 5.3 Triangular linear systems

L =

`11 0 · · · 0`21 `22 · · · 0...

... . . . ...`n1 `n2 · · · `nn

, U =

u11 u12 · · · u1n

0 u22 · · · u2n...

... . . . ...0 0 · · · unn

Solving system Lx = b is calledForward Substitution :x1 = b1/`11

xi =(

bi−∑i−1j=1 `i jx j

)/`ii

Solving system Ux = b is calledBack Substitution :xn = bn/unn

xi =(

bi−∑nj=i+1 ui jx j

)/uii

But how to transform an arbitrary matrix to a triangular form?

115 Systems of LE 5.4 Elementary Elimination Matrices

5.4 Elementary Elimination Matrices

for a1 6= 0: [1 0

−a2/a1 1

][a1

a2

]=

[a1

0

].

In general case, if a = [a1,a2, ...,an] and ak 6= 0:

Mka =

1 · · · 0 0 · · · 0... . . . ...

... . . . ...0 · · · 1 0 · · · 00 · · · −mk+1 1 · · · 0... . . . ...

... . . . ...0 · · · −mn 0 · · · 1

a1...

ak

ak+1...

an

=

a1...

ak

0...0

,

where mi = ai/ak, i = k+1, ...,n.

116 Systems of LE 5.4 Elementary Elimination Matrices

The divider ak is called pivot (juhtelement)Matrix Mk is called also elementary elimination matrix or Gauss transforma-

tion

1. Mk is nonsingular⇐= Why?being lower triangular and unit diagonal

2. Mk = I−meTk , where m = [0, ...,0,mk+1, ...,mn]

T and ek is column k of unitmatrix

3. Lk =(de f ) M−1

k = I +meTk .

4. If M j, j > k is some other elementary elimination matrix with multiplicationvector t, then

MkM j = I−meTk − teT

j +meTk teT

j = I−meTk − teT

j ,

due to eTk t = 0.

117 Systems of LE 5.5 Gauss Elimination and LU Factorisation

5.5 Gauss Elimination and LU Factorisation

• apply series of Gauss elimination matrices from the left: M1, M2,...,Mn−1, takingM = Mn−1 · · ·M1

– we get the linear system:

MAx = Mn−1 · · ·M1Ax = Mn−1 · · ·M1b = Mb

– upper triangular =⇒

∗ easy to solve.

The process is called Gauss Elimination Method (GEM)


• Denoting U = MA and L = M−1, we get that

L = M−1 = (Mn−1 · · ·M1)−1 = M−1

1 · · ·M−1n−1 = L1 · · ·Ln−1

is lower unit triangular (ones on the diagonal)

• =⇒A = LU.

Expressed in an algorithm:


Algoritm 5.1. LU-factorisation using Gauss elimination method (GEM)� �do k=1,...,n-1 # cycle over matrix columns

if akk==0 then stop # stop in case pivot == 0

do i=k+1,n

mik = aik/akk # coefficient calculation in column kenddo

do i=k+1,n

do j=k+1,n # applying transformations to

ai j = ai j−mikak j # the rest of the matrix

enddo

enddo

enddo�NB! In practical implementation: For storing mik use corresponding elements in

A (will be zeroes anyway)

120 Systems of LE 5.6 Number of operations in GEM

5.6 Number of operations in GEM

Finding operation counts for Alg. 5.1:

• Replace loops with corresponding sums over number of particular operations:

n−1

∑i=1

(n

∑j=i+1

1+n

∑j=i+1

n

∑k=i+1

2

)

=n−1

∑i=1

((n− i)+2(n− i)2) =23

n3 +O(n2)

• used that ∑mi=1 ik =mk+1/(k+1)+O(mk) (which is enough for finding the num-

ber of operations with the highest order)

• How many operations there are for solving a triangular system?Number of operations for forward and backward substitution for L and U isO(n2)

121 Systems of LE 5.7 GEM with row permutations

• =⇒ the whole system solution Ax = b takes 23n3 +O(n2) operations

5.7 GEM with row permutations

• If pivot == 0 GEM won’t work

• Row permutations or partial pivoting may help

• For numerical stability, also the pivot must not be small

Example 5.1

• Consider matrix

A =

[0 11 0

]

– non-singular

– but LU-factorisation impossible without row permutations


• But on contrary, the matrix

A =

[1 11 1

]

– has the LU-factorisation

A =

[1 11 1

]=

[1 01 1

][1 10 0

]= LU

– But, what is wrong with matrix A?with A being actually singular matrix!


Example 5.2. Small pivots

• Consider

A =

[ε 11 1

],

with ε such that 0 < ε < εmach in given floating point system

– (i.e. 1+ ε = 1 in floating point arithmetics)

– Without row permutation we get (in floating-point arithmetics):

M =

[1 0−1/ε 1

]=⇒ L =

[1 0

1/ε 1

],U =

[ε 10 1−1/ε

]=

[ε 10 −1/ε

]

• But then

LU =

[1 0

1/ε 1

][ε 10 −1/ε

]=

[ε 11 0

]6= A


Using row permutation

• the pivot is 1;

• multiplier −ε =⇒

M =

[1 0−ε 1

]=⇒ L =

[1 0ε 1

],U =

[1 10 1− ε

]=

[1 10 1

]

in floating point arithmetics

• =⇒

LU =

[1 0ε 1

][1 10 1

]=

[1 1ε 1

]← OK!


Algoritm 5.2. LU-factorisation with GEM using row permutations� �do k=1,...,n-1 # cycle over matrix columns

Find index p such that: # looking for the best pivot

|apk| ≥ |aik|, k ≤ i≤ n # in given column

if p 6= k then interchange rows k and pif akk = 0 then continue with next k # skip such column

do i=k+1,n

mik = aik/akk # multiplier calculation in column kenddo

do i=k+1,n

do j=k+1,n # transformation application

ai j = ai j−mikak j # to the rest of the matrix

enddo

enddo

enddo�


As a result, MA =U , where U upper-triangular, OK so far?but actually

M = Mn−1Pn−1 · · ·M1P1

• M−1 still lower-triangular?is not lower-triangular any more, although it is still denoted by L

– but is it triangular?we have still triangular L

– knowing the permutations P = Pn−1 · · ·P1 in advance would give

PA = LU,

where L indeed lower triangular matrix


• But, do we really need to actually perform the row exchanges – explicitly?instead of row exchanges we can perform just appropriate mapping ofmatrix (and vector) indexes

– We start with unit index mapping p = [1,2,3,4, ...,n]

– If rows i and j need to be exchanged, we exchange corresponding valuesp[i] and p[ j]

– In the algorithm, take everywhere ap[i] j instead of ai j (and other arrayscorrespondingly)

To solve the system Ax = b (8) How does the whole algorithm look like now?

• Solve the lower triangular system Ly = Pb with forward substitution

• Solve the upper triangular system Ux = y backward substitution

The term partial pivoting comes from the fact that we are seeking for the best pivotonly in the current column (starting from the diagonal and below) of the matrix


• Complete pivoting – the best pivot is chosen from the whole remaining part ofthe matrix

– This means exchanging both rows and columns of the matrix

PAQ = LU,

where P and Q are permutation matrices.

– The system is solved in three stages: Ly = Pb; Uz = y and x = Qz

Although numerical stability is better in case of complete pivoting it is rarely used,because

• more costly

• usually not needed

129 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting

5.8 Reliability of the LU-factorisation with partial pivoting

Introduce the vector norm:

‖x‖∞= maxi|xi|, x ∈ Rn

and corresponding matrix norm:

‖A‖∞ = supx∈Rn‖Ax‖∞

‖x‖∞

, x ∈ Rn.

Looking at the rounding errors in GEM, it can be shown that actually we find L, andU which satisfy the relation:

PA = LU−E,

where error E can be estimated:

‖E‖∞ ≤ nε‖L‖∞‖U‖∞ (9)


where ε is machine epsilonIn practice we replace the system PAx = Pb with the system LU = Pb. =⇒the

system we are solving is actually

(PA+E)x = Pb

for finding the approximate solution x.

• How far is it from the real solution?

From the matrix perturbation theory:Let us solve the system

Ax = b (10)

where A ∈ Rn×n and b ∈ Rn are given and x ∈ Rn is unknown. Suppose, A is givenwith an error A = A+δA. Perturbed solution satisfies the system of linear equations

(A+δA)(x+δx) = b. (11)


Theorem 5.1. Let A be nonsingular and δA be sufficiently small, such that

‖δA‖∞‖A−1‖∞ ≤12. (12)

Then (A+δA) is nonsingular and

‖δx‖∞

‖x‖∞

≤ 2κ(A)‖δA‖∞

‖A‖∞

, (13)

where κ(A) = ‖A‖∞‖A−1‖∞ is the condition number.Proof of the theorem (http://www.ut.ee/~eero/SC/konspekt/

perturb-toest/perturb-proof.pdf)(EST) (http://www.ut.ee/~eero/SC/konspekt/perturb-toest/

perturb-toest.pdf)

http://www.ut.ee/~eero/SC/konspekt/perturb-toest/perturb-proof.pdf

http://www.ut.ee/~eero/SC/konspekt/perturb-toest/perturb-proof.pdf

http://www.ut.ee/~eero/SC/konspekt/perturb-toest/perturb-toest.pdf

http://www.ut.ee/~eero/SC/konspekt/perturb-toest/perturb-toest.pdf


Remarks

1. The result is true for an arbitrary matrix norm derived from the correspondingvector norm

2. Theorem 5.1 says, that in case of small condition number a small relative errorin matrix A can cause small or large?only a small error in the solution x

3. If condition number is big, what can happen?everything can happen

4. It is not simple to calculate condition number but still, it can be estimated

Combining the result (9) with Theorem 5.1, we see that:GEM forward error can be estimated as follows:

‖δx‖∞

‖x‖∞

≤ 2nεκ(A)G,

where the coefficient G =[‖L‖∞‖U‖∞

‖A‖∞

]is called growth factor


Conclusions

• If G is not large, then well-conditioned matrix gives fairly good answer (i.e.O(nε)).

• In case of partial pivoting, the elements of matrix L are ≤ 1.

– Nevertherless, ∃ examples, where with well-conditioned matrix A the ele-ments of U exponentially large in comparison with the elements in A.

– But these examples are more like “academic” – in practice rarely one findssuch (and the method can be used without any fair)

108systems of le5.1 systems of linear equations 5 solving …€¦ · 113systems of le5.3...

Documents