matrix multiplication chapter iii - the best groupbest.eng.buffalo.edu/research/lecture series...
TRANSCRIPT
![Page 1: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/1.jpg)
Matrix Multiplication
Chapter III – General Linear Systems
By Gokturk Poyrazoglu
The State University of New York at Buffalo – BEST Group – Winter Lecture Series
![Page 2: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/2.jpg)
General Outline
1. Triangular Systems
2. The LU Factorization
3. Pivoting
![Page 3: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/3.jpg)
Triangular Systems Outline
1. Row oriented Forward Substitution
2. Row oriented Back Substitution
3. Column oriented Forward Substitution
4. Column oriented Back Substitution
5. Multiple Right-Hand Sides
6. Nonsquare Triangular Systems Solving
7. Algebra of Triangular Matrices
![Page 4: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/4.jpg)
Forward Substitution
Consider a lower triangular system
The unknowns are:
General Procedure:
![Page 5: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/5.jpg)
Row Oriented Forward Substitution
L is a square lower triangular matrix
b is a vector
Overwrite b with the solution of Lx=b
![Page 6: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/6.jpg)
Row Oriented Back Substitution
U is a square upper triangular matrix
b is a vector
Solution x:
Overwrite b with the solution of Ux=b
![Page 7: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/7.jpg)
Column Oriented Versions
Example:
Solve for x1 (x1=3); remove from the equations by
![Page 8: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/8.jpg)
Column Oriented Forward Substitution
L is a square lower triangular matrix
b is a vector
Overwrite b with the solution of Lx=b
![Page 9: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/9.jpg)
Column Oriented Back Substitution
U is a square upper triangular matrix
b is a vector
Overwrite b with the solution of Ux=b
![Page 10: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/10.jpg)
Multiple Right-Hand Sides
Consider block matrices L, X, and B
Solve L11X1=B1 for X1.
Remove X1 from block equations as follows :
![Page 11: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/11.jpg)
Nonsquare Triangular System Solving
Consider a block matrix L where m > n
Assume L11 is nonsingular and lower triangular.
Solve L11x=b1 for x
Then x should solve the system 𝐿21 𝐿−111𝑏1 = 𝑏2
Otherwise, there is no solution to the overall system.
![Page 12: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/12.jpg)
Triangular Matrix Properties
Unit Triangular Matrix:
Triangular matrix with 1’s on the diagonal.
Other Properties
Inverse of an upper triangular is an upper triangular matrix.
The product of two upper triangular is an upper triangular.
Inverse of a unit upper triangular is a unit upper triangular.
The product of two unit upper triangular is an upper triangular.
![Page 13: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/13.jpg)
The LU Factorization
Outline
Background
Gauss transformations
Application
Upper Triangularizing
Existence of LU
Other versions of LU
Rectangular Matrix
Block LU
![Page 14: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/14.jpg)
Background
Example :
Matrix Notation:
Multiply the 1st equation by 2,
Subtract it from the 2nd equation.
![Page 15: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/15.jpg)
Gauss Transformations
Consider a vector V (a stack of 2 block vectors v1 and v2)
Suppose , and define Gauss Vector
Define Gauss Transformation matrix
![Page 16: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/16.jpg)
Application
Consider a matrix C, and apply an outer product update
Repeat the process;
Algorithm
![Page 17: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/17.jpg)
Upper Triangularizing
Consider a square matrix-A, then
Example:
Note :Diagonal components of A (pivots) should be zero
![Page 18: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/18.jpg)
Existence of LU
If no zero pivots are encountered; then
and
so that
LU factorization does NOT exist if is singular.
That means kth pivot is zero.
![Page 19: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/19.jpg)
Construction of matrix-L
Consider the example
kth column of L is defined by the multipliers from kth step
![Page 20: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/20.jpg)
Outer Product Point of View
Consider a matrix A as;
Gauss Elimination results:
where
Hence;
![Page 21: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/21.jpg)
LU Factorization of a Rectangular Matrix
Such matrices L and U exist if is nonsingular.
Examples:
Algorithm for the 1st example:
Operation: nr2-r3/3 flops
![Page 22: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/22.jpg)
Roundoff Error in Gaussian Elimination
![Page 23: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/23.jpg)
Triangular Solving with Inexact Triangles
If a small pivot is encountered, then we can expect large numbers to be
present in L and U.
Example :
Solution is in contrast to exact solution
![Page 24: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/24.jpg)
Pivoting
Outline
Interchange Permutations
Partial Pivoting
Complete pivoting
Rook Pivoting
![Page 25: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/25.jpg)
Interchange Permutations
Consider a permutation matrix:
If we multiply matrix A from the left,
rows 1 and 4 interchanged
If we multiply matrix A from the right;
columns 1 and 4 interchanged
![Page 26: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/26.jpg)
Partial Pivoting
Motivation: To guarantee that no multiplier is greater
than 1 in absolute value.
Example : Consider matrix-A, get the largest entry in
the first column to a11
![Page 27: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/27.jpg)
Partial Pivoting
Particular row interchange strategy is called partial
pivoting.
In general :
where U is an upper triangular, and no multiplier is greater
than 1 in absolute value as a consequence of partial
pivoting.
![Page 28: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/28.jpg)
Complete Pivoting
Partial Pivoting was scanning the current subcolumn for
maximal element;
Complete Pivoting scans the current submatrix to find the
largest entry to pivot.
Hence;
![Page 29: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/29.jpg)
Complete Pivoting Properties
Gaussian Elimination with Complete Pivoting is STABLE.
No significant reason to choose Complete pivoting over
Partial pivoting.
Only if matrix A is rank deficient.
In principal, when the pivot of current submatrix is ZERO
at the beginning of step r+1; that indicates that the
rank(A) =r;
In practice, encountering an exactly zero pivot is lesslikely.
![Page 30: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/30.jpg)
Rook Pivoting
Computes the factorization :
Choosing Pivot: Search for an element of current submatrix
that is maximal in BOTH its ROW and COLUMN.
Complete Pivoting
Rook Pivoting Candidate
![Page 31: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/31.jpg)
Comparison
Flops:
Partial pivoting : O(n2)
Complete Pivoting O(n3)
Rook Pivoting O(n2)
Rook has same level of reliability as complete pivoting
and represents same O(n2) overhead as partial pivoting.
Complete Pivoting may be used for rank identification in
principal.
All pivoting methods are stable.
![Page 32: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/32.jpg)
Extra Proof Slides
![Page 33: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/33.jpg)
Proof of LU Factorization Theorem (slide 18)
![Page 34: Matrix Multiplication Chapter III - The BEST Groupbest.eng.buffalo.edu/Research/Lecture Series 2013/General Linear... · Complete Pivoting Properties Gaussian Elimination with Complete](https://reader033.vdocuments.site/reader033/viewer/2022060300/5f0805c27e708231d41ff0d5/html5/thumbnails/34.jpg)
Proof of Slide 23