lec05 systems

8/9/2019 Lec05 Systems

1/36

SOLVING LINEAR SYSTEMS OF EQUATIONS

See Chapter 3 of text

Background on linear systems

Gaussian elimination and the Gauss-Jordan algorithms

The LU factorization

Gaussian Elimination with pivoting

5-1 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


2/36

Background: Linear systems

The Problem: Ais an n n matrix, and ba vector ofRn.

Find x such that:

Ax=b

x is theunknownvector, b theright-hand side, and A

is thecoefficient matrix

Example:

2x1 + 4x2 + 4x3 = 6

x1 + 5x2 + 6x3 = 4x1 + 3x2 + x3 = 8

or2 4 41 5 6

1 3 1

x1x2x3

= 648

Solution of above system ?5-2 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


3/36

Standard mathematical solution by Cramers rule:

xi = det(Ai)/ det(A)

Ai = matrix obtained by replacing i-th column by b.

Note: This formula is useless in practice beyond n= 3

orn= 4.

Three situations:

1. The matrix A is nonsingular. There is a unique solutiongiven by x=A1b.

2. The matrix A is singular and b Ran(A). There are

infinitely many solutions.3. The matrix A is singular and b /Ran(A). There are no

solutions.



4/36

Example: (1) Let A = 2 0

0 4 b =

1

8 . A is non-

singular a unique solution x=

0.5

2

.

Example: (2) Case whereAis singular &b Ran(A):

A= 2 0

0 0 , b= 1

0 . infinitely many solutions: x() =

0.5

.

Example: (3) Let A same as above, but b=

1

1

.

No solutions since 2nd equation cannot be satisfied



5/36

Triangular linear systems

Example:

2 4 40 5 20 0 2

x1x2x3

= 214

One equation can be trivially solved: the last one.x3 = 2

x3 is known we can now solve the 2nd equation:

5x2 2x3 = 1 5x2 2 4 = 1 x2 = 1

Finally x1 can be determined similarly:

2x1+ 4x2+ 4x3 = 2 ... x1 =5



6/36

ALGORITHM : 1 Back-Substitution algorithm

Fori =n:1 : 1 do:t :=biForj =i+ 1 :n do

t := t (ai,i+1:n, xi+1:n)= t an inner productt:=t aijxjEndxi =t/aii

End

We must require that each aii = 0

Operation count?

Round-off error (use previous results for (, ))?



7/36

Backward error analysis for the triangular solve

The computed solution xof the triangular systemU x=b computed by the previous algorithm satisfies:

(U+E)x=b

with|E| n u |U| +O(u 2)

Backward error analysis. Computed x solves a slightly

perturbed system.

Backward error not large in general. It is said that

triangular solve is backward stable.



8/36

Column version of back-substitution:

Back-Substitution algorithm. Column versionForj =n :1 : 1 do:

xj =bj/ajj

Fori = 1 :j 1 dobi := bi xj aij

End

End

Justify the above algorithm [Show that it does indeed

compute the solution]

See text for analogous algorithms for lower triangular

systems.



9/36

Linear Systems of Equations: Gaussian Elimination

Back to arbitrary linear systems.

Principle of the method: Since triangular systems are

easy to solve, we will transform a linear system into

one that is triangular. Main operation: combine rows so

that zeros appear in the required locations to make the

system triangular.

Notation: use a Tableau:

2x1 + 4x2 + 4x3 = 2

x1 + 3x2 + 1x3 = 1

x1 + 5x2 + 6x3 = 6

tableau:2 4 4 21 3 1 1

1 5 6 6



10/36

Main operation used: scaling and adding rows.

Example: Replace row2 by: row2 - 1

2*row1:2 4 4 2

1 3 1 1

1 5 6 6

2 4 4 2

0 1 1 0

1 5 6 6 This is equivalent to:

1 0 0

12

1 0

0 0 1

2 4 4 2

1 3 1 1

1 5 6 6

=

2 4 4 2

0 1 1 0

1 5 6 6

The left-hand matrix is of the form

M =I veT1 with v =

012

0



11/36

Linear Systems of Equations: Gaussian Elimination

Go back to original system. Step 1 must transform:

2 4 4 2

1 3 1 1

1 5 6 6

into:

x x x x

0 x x x

0 x x x

row2 :=row2 12

row1: row3 := row3 12

row1:

2 4 4 2

0 1 1 0

1 5 6 6

2 4 4 2

0 1 1 0

0 3 4 7



12/36

Equivalent to

1 0 0

12

1 0

12

0 1

2 4 4 2

1 3 1 1

1 5 6 6

=

2 4 4 2

0 1 1 0

0 3 4 7

[A, b][M1A, M1b]; M1 =I v(1)eT1 ; v

(1) =

012

12

New system A1x=b1. Step 2 must now transform:

2 4 4 20 1 1 0

0 3 4 7

into:x x x x0 x x x

0 0 x x


13/36

row3 := row3 3 row2 :

2 4 4 2

0 1 1 00 0 7 7

Equivalent to

1 0 0

0 1 0

0 3 1

2 4 4 2

0 1 1 0

0 3 4 7

=

2 4 4 2

0 1 1 0

0 0 7 7

Second transformation is as follows:

[A1, b1][M2A1, M2b1] M2 =I v(2)eT2 v(2) =00

3

Triangular system Solve.



14/36

Row k

Pivot

A =k



15/36

ALGORITHM : 2 Gaussian Elimination

1. For k = 1 :n 1 Do:2. For i=k+ 1 :n Do:

3. piv :=aik/akk4. For j :=k+ 1 :n+ 1 Do :

5. aij :=aij piv akj6. End

6. End

7. End

Operation count:

T =n1k=1

ni=k+1

[1+n+1

j=k+1

2] =n1k=1

ni=k+1

(2(n k)+3) =...

Complete the above calculation. Order of the cost?5-15 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


16/36

The LU factorization

Now ignore the right-hand side from the transformations.

Observation: Gaussian elimination is equivalent to n

1 successive Gaussian transformations, i.e., multiplica-

tions with matrices of the formMk =I v(k)eTk , where

the first k components ofv(k) equal zero.

Set A0 A

A M1A0 =A1 M2A1 =A2 M3A2 =A3

Mn1An2 =An1 U

Last Ak U is an upper triangular matrix.



17/36

At each step we have: Ak =M1k+1Ak+1. Therefore:

A0 = M11 A1= M11 M

12 A2

= M11 M12 M

13 A3

= . . .= M11 M

12 M

13 M

1n1An1

L=M11 M12 M

13 M

1n1

Note: L is Lower triangular, An1 is upper triangular

LU decomposition : A=LU



18/36

How to get L?

L=M11 M

12 M

13 M

1n1

Consider only the first 2 matrices in this product.

Note M

1

k = (I v

(k)

e

T

k )

1

= (I+v

(k)

e

T

k ). So:

M11 M12 = (I+ v

(1)eT1 )(I+ v(2)eT2 ) =I+ v

(1)eT1 + v(2)eT2

Generally,

M11 M12 M

1k =I+v

(1)eT1 +v(2)eT2 + v

(k)eTk


19/36

The L factor is a lower triangular matrix with ones on

the diagonal. Column k of L, contains the multipliers

lik used in the k-th step of Gaussian elimination.



20/36

A matrix A has an LU decomposition if

det(A(1 :k, 1 :k))= 0 for k = 1, , n 1.

In this case, the determinant ofA satisfies:

det A = det(U) =

ni=1

uii

If, in addition, A is nonsingular, then the LU factoriza-

tion is unique.



21/36

Show how to obtain L directly from the multipliers

Practical use: Show how to use the LU factorization tosolve linear systems with the same matrix A and different

bs.

LU factorization of the matrixA =2 4 41 5 6

1 3 1

?

Determinant ofA

? True or false: Computing the LU factorization of

matrix A involves more arithmetic operations than solving

a linear system Ax=b by Gaussian elimination.



22/36

Gauss-Jordan Elimination

Principle of the method: We will now transform thesystem into one that is even easier to solve than triangular

systems, namely a diagonal system. The method is very

similar to Gaussian Elimination. It is just a bit moreexpensive.

Back to original system. Step 1 must transform:

2 4 4 2

1 3 1 1

1 5 6 6

into:

x x x x

0 x x x

0 x x x



23/36

row2 :=row20.5row1: row3 :=row30.5row1:

2 4 4 2

0 1 1 0

1 5 6 6

2 4 4 2

0 1 1 0

0 3 4 7

Step 2:2 4 4 20 1 1 0

0 3 4 7

into:x 0 x x0 x x x

0 0 x x

row1 :=row1 4 row2: row3 :=row3 3 row2:

2 0 8 2

0 1 1 0

0 3 4 7

2 0 8 2

0 1 1 0

0 0 7 7



24/36

There is now a third step:

To transform: 2 0 8 20 1 1 0

0 0 7 7

into: x 0 0 x0 x 0 x

0 0 x x

row1 :=row1 87 row3: row2 :=row2 1

7 row3:

2 0 0 10

0 1 1 0

0 0 7 7

2 0 0 10

0 1 0 1

0 0 7 7

Solution: x3 =1; x2 =1; x1 = 5



25/36

ALGORITHM : 3 Gauss-Jordan elimination

1. For k = 1 :n Do:

2. For i= 1 :n and ifi! =k Do :

3. piv :=aik/akk4. For j :=k+ 1 :n+ 1 Do :

5. aij :=aij piv akj6. End

6. End

7. End

Operation count:

T =n

k=1

n1i=1

[1 +n+1

j=k+1

2] =n1k=1

n1i=1

(2(n k) + 3) =

Complete the above calculation. Order of the cost?

How does it compare with Gaussian Elimination?5-25 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


26/36

function x = gaussj (A, b)%---------------------------------------------------% function x = gaussj (A, b)% solves A x = b by Gauss-Jordan elimination%---------------------------------------------------

n = size(A,1) ;A = [A,b];for k=1:n

for i=1:nif (i ~= k)

piv = A(i,k) / A(k,k) ;A(i,k+1:n+1) = A(i,k+1:n+1) - piv*A(k,k+1:n+1);

endend

endx = A(:,n+1) ./ diag(A) ;



27/36

Gaussian Elimination: Partial Pivoting

Consider again Gaussian Elimination for the linear system

2x1 + 2x2 + 4x3 = 2

x1 + x2 + x3 = 1

x1 + 4x2 + 6x3 = 5

Or:

2 2 4 2

1 1 1 1

1 4 6 5row2 :=row2

12

row1: row3 := row3 12

row1:

2 2 4 2

0 0 1 0

1 4 6 5

2 2 4 2

0 0 1 0

0 3 4 6

Pivot a22 is zero. Solution : permute rows 2 and 3:2 2 4 2

0 3 4 6

0 0 1 05-27 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


28/36

Gaussian Elimination with Partial Pivoting

Partial Pivoting

General situation:

Largest a ik

Perm

uterows

akk

Row k

Always permute row k with row l such that|alk|= maxi=k,...,n |aik|

More stable algorithm.



29/36

function x = gaussp (A, b)%---------------------------------------------------

% function x = guassp (A, b)% solves A x = b by Gaussian elimination with% partial pivoting/%---------------------------------------------------

n = size(A,1) ;A = [A,b]for k=1:n-1

[t, ip] = max(abs(A(k:n,k)));ip = ip+k-1 ;

%% swaptemp = A(k,k:n+1) ;A(k,k:n+1) = A(ip,k:n+1);A(ip,k:n+1) = temp;

%%

for i=k+1:npiv = A(i,k) / A(k,k) ;A(i,k+1:n+1) = A(i,k+1:n+1) - piv*A(k,k+1:n+1);

endend

x = backsolv(A,A(:,n+1));5-29 GvL 3.{1,3,5}; Heath 2; TB 7,20-21 Systems


30/36

Pivoting and permutation matrices

A permutation matrix is a matrix obtained from theidentity matrix by permuting its rows

For example for the permutation = {3, 1, 4, 2} we

obtain

P =

0 0 1 0

1 0 0 00 0 0 1

0 1 0 0

Important observation: the matrix P A is obtained from

A by permuting its rows with the permutation

(P A)i,: =A(i),:


Wh i h i P A h


31/36

What is the matrix P A when

P =

0 0 1 0

1 0 0 0

0 0 0 1

0 1 0 0

A=

1 2 3 4

5 6 7 8

9 0 1 2

3 4 5 6

?

Any permutation matrix is the product of interchange

permutations, which only swap two rows ofI.

Notation: Eij = Identity with rows i and j swapped



32/36

Example: To obtain ={3, 1, 4, 2} from ={1, 2, 3, 4}

we need to swap (2) (3) then (3) (4) and

finally (1) (2). Hence:

P =

0 0 1 0

1 0 0 00 0 0 1

0 1 0 0

=E1,2 E3,4 E2,3

In the previous example where

>> A = [ 1 2 3 4; 5 6 7 8; 9 0 -1 2 ; -3 4 -5 6]

Matlab gives det(A) =896. What is det(P A)?


At h t f G E ith ti l i ti


33/36

At each step of G.E. with partial pivoting:

Mk+1Ek+1Ak =Ak+1

whereEk+1encodes a swap of rowk+1with rowl > k+1.

Notes: (1) E1i = Ei and (2) M1

j Ek+1 = Ek+1

Mj1

for k j, where Mj has a permuted Gauss vector:

(I+v(j)eTj)Ek+1 = Ek+1(I+Ek+1v(j)eTj)

Ek+1(I+ v(j)eTj)

Ek+1 Mj

Here we have used the fact that above row k+ 1, the

permutation matrix Ek+1 looks just like an identity matrix.


Res lt


34/36

Result:

A0 = E1M11 A1

= E1M11 E2M

12 A2 =E1E2

M11 M12 A2

= E1E2 M11 M

12 E3M

13 A3

= E1E2E3 M11 M

12 M

13 A3

= . . .

= E1 En1 M11 M

12 M

13 M

1n1 An1

In the end

P A=LU with P =En1 E1



35/36

Error Analysis

If no zero pivots are encountered during Gaussian elimi-nation (no pivoting) then the computed factors L and U

satisfyL

U =A+H

with

|H| 3(n 1) u

|A| + |L| |U|

+O(u 2)

Solution x computed via Ly =b and Ux= y is s. t.(A+E)x =b with

|E| nu 3|A| + 5 |L| |U|+O(u 2)


Backward error estimate


36/36

Backward error estimate.

|L| and |U| are not known in advance they can be

large.

What if partial pivoting is used?

Permutations introduce no errors. Equivalent to stan-dard LU factorization on matrix P A.

|L| is small since lij 1. Therefore, only U is uncer-

tain

In practice partial pivoting is stable i.e., it is highly

unlikely to have a very large U.


lec05 systems

Documents