l.vandenberghe ece133a(spring2021) 7.linearequations
TRANSCRIPT
L. Vandenberghe ECE133A (Fall 2021)
7. Linear equations
β’ QR factorization method
β’ factor and solve
β’ LU factorization
7.1
QR factorization and matrix inverse
QR factorization of nonsingular matrix
every nonsingular π΄ β RπΓπ has a QR factorization
π΄ = ππ
β’ π β RπΓπ is orthogonal (πππ = πππ = πΌ)
β’ π β RπΓπ is upper triangular with positive diagonal elements
Inverse from QR factorization: the inverse π΄β1 can be written as
π΄β1 = (ππ )β1 = π β1πβ1 = π β1ππ
Linear equations 7.2
Solving linear equations by QR factorization
Algorithm: to solve π΄π₯ = π with nonsingular π΄ β RπΓπ,
1. factor π΄ as π΄ = ππ
2. compute π¦ = πππ
3. solve π π₯ = π¦ by back substitution
Complexity: 2π3 + 3π2 β 2π3 flops
β’ QR factorization: 2π3
β’ matrix-vector multiplication: 2π2
β’ back substitution: π2
Linear equations 7.3
Multiple right-hand sides
consider π sets of linear equations with the same coefficient matrix π΄:
π΄π₯1 = π1, π΄π₯2 = π2, . . . , π΄π₯π = ππ
β’ equivalently, solve π΄π = π΅ where π΅ is π Γ π matrix with columns π1, . . . , ππ
β’ can be solved in 2π3 + 3ππ2 flops if we reuse the factorization π΄ = ππ
β’ for π οΏ½ π, cost is roughly equal to cost of solving one equation (2π3)
Application: to compute π΄β1, solve the matrix equation π΄π = πΌ
β’ equivalent to π equations
π π₯1 = πππ1, π π₯2 = πππ2, . . . , π π₯π = ππππ
(π₯π is column π of π and ππππ is transpose of πth row of π)
β’ complexity is 2π3 + π3 = 3π3 (here the 2nd term π3 is not negligible)
Linear equations 7.4
Factorβsolve approach
to solve π΄π₯ = π, first write π΄ as a product of βsimpleβ matrices
π΄ = π΄1π΄2 Β· Β· Β· π΄π
then solve (π΄1π΄2 Β· Β· Β· π΄π)π₯ = π by solving π equations
π΄1π§1 = π, π΄2π§2 = π§1, . . . , π΄πβ1π§πβ1 = π§πβ2, π΄ππ₯ = π§πβ1
Examples
β’ QR factorization: π = 2, π΄ = ππ
β’ LU factorization (this lecture)
β’ Cholesky factorization (later)
Linear equations 7.5
Complexity of factorβsolve method
#flops = π + π
β’ π is complexity of factoring π΄ as π΄ = π΄1π΄2 Β· Β· Β· π΄π (factorization step)
β’ π is complexity of solving the π equations for π§1, π§2, . . . π§πβ1, π₯ (solve step)
β’ usually π οΏ½ π
Example: solving linear equations using the QR factorization
π = 2π3, π = 3π2
Linear equations 7.6
LU factorization
LU factorization without pivoting
π΄ = πΏπ
β’ πΏ unit lower triangular, π upper triangular
β’ does not always exist (even if π΄ is nonsingular)
LU factorization (with row pivoting)
π΄ = ππΏπ
β’ π permutation matrix, πΏ unit lower triangular, π upper triangular
β’ exists if and only if π΄ is nonsingular (see later)
Complexity: (2/3)π3 if π΄ is π Γ π
Linear equations 7.7
Solving linear equations by LU factorization
Algorithm: to solve π΄π₯ = π with nonsingular π΄ of size π Γ π
1. factor π΄ as π΄ = ππΏπ ((2/3)π3 flops)
2. solve (ππΏπ)π₯ = π in three steps
(a) permutation: π§1 = πππ (0 flops)(b) forward substitution: solve πΏπ§2 = π§1 (π2 flops)(c) back substitution: solve ππ₯ = π§2 (π2 flops)
Complexity: (2/3)π3 + 2π2 β (2/3)π3 flops
this is the standard method for solving π΄π₯ = π
Linear equations 7.8
Multiple right-hand sides
two equations with the same matrix π΄ (nonsingular and π Γ π):
π΄π₯ = π, π΄π₯ = οΏ½ΜοΏ½
β’ factor π΄ once
β’ forward/back substitution to get π₯
β’ forward/back substitution to get π₯
complexity: (2/3)π3 + 4π2 β (2/3)π3
Exercise: propose an efficient method for solving
π΄π₯ = π, π΄ππ₯ = οΏ½ΜοΏ½
Linear equations 7.9
LU factorization and matrix inverse
suppose π΄ is nonsingular and π Γ π, with LU factorization
π΄ = ππΏπ
β’ inverse from LU factorization
π΄β1 = (ππΏπ)β1 = πβ1πΏβ1ππ
β’ gives interpretation of solve step: we evaluate
π₯ = π΄β1π = πβ1πΏβ1πππ
in three stepsπ§1 = πππ, π§2 = πΏβ1π§1, π₯ = πβ1π§2
Linear equations 7.10
Computing the inverse
solve π΄π = πΌ column by column
β’ one LU factorization of π΄: 2π3/3 flops
β’ π solve steps: 2π3 flops
β’ total: (8/3)π3 flops
slightly faster methods exist that exploit structure in right-hand side πΌ
Conclusion: do not solve π΄π₯ = π by multiplying π΄β1 with π
Linear equations 7.11
LU factorization without pivoting
[π΄11 π΄1,2:ππ΄2:π,1 π΄2:π,2:π
]=
[1 0
πΏ2:π,1 πΏ2:π,2:π
] [π11 π1,2:π0 π2:π,2:π
]=
[π11 π1,2:π
π11πΏ2:π,1 πΏ2:π,1π1,2:π + πΏ2:π,2:ππ2:π,2:π
]Recursive algorithm
β’ determine first row of π and first column of πΏ
π11 = π΄11, π1,2:π = π΄1,2:π, πΏ2:π,1 =1π΄11
π΄2:π,1
β’ factor the (π β 1) Γ (π β 1)-matrix π΄2:π,2:π β πΏ2:π,1π1,2:π as
π΄2:π,2:π β πΏ2:π,1π1,2:π = πΏ2:π,2:ππ2:π,2:π
this is an LU factorization (without pivoting) of size (π β 1) Γ (π β 1)
Linear equations 7.12
Example
LU factorization (without pivoting) of
π΄ =
8 2 94 9 46 7 9
write as π΄ = πΏπ with πΏ unit lower triangular, π upper triangular
π΄ =
8 2 94 9 46 7 9
=
1 0 0πΏ21 1 0πΏ31 πΏ32 1
π11 π12 π130 π22 π230 0 π33
Linear equations 7.13
Example
β’ first row of π, first column of πΏ:8 2 94 9 46 7 9
=
1 0 01/2 1 03/4 πΏ32 1
8 2 90 π22 π230 0 π33
β’ second row of π, second column of πΏ:[
9 47 9
]β[
1/23/4
] [2 9
]=
[1 0πΏ32 1
] [π22 π230 π33
][
8 β1/211/2 9/4
]=
[1 0
11/16 1
] [8 β1/20 π33
]β’ third row of π: π33 = 9/4 + 11/32 = 83/32
Conclusion
π΄ =
8 2 94 9 46 7 9
=
1 0 01/2 1 03/4 11/16 1
8 2 90 8 β1/20 0 83/32
Linear equations 7.14
Not every nonsingular π΄ can be factored as π΄ = πΏπ
π΄ =
1 0 00 0 20 1 β1
=
1 0 0πΏ21 1 0πΏ31 πΏ32 1
π11 π12 π130 π22 π230 0 π33
β’ first row of π, first column of πΏ:
1 0 00 0 20 1 β1
=
1 0 00 1 00 πΏ32 1
1 0 00 π22 π230 0 π33
β’ second row of π, second column of πΏ:[
0 21 β1
]=
[1 0πΏ32 1
] [π22 π230 π33
]π22 = 0, π23 = 2, πΏ32 Β· 0 = 1 ?
Linear equations 7.15
LU factorization (with row pivoting)
if π΄ is π Γ π and nonsingular, then it can be factored as
π΄ = ππΏπ
π is a permutation matrix, πΏ is unit lower triangular, π is upper triangular
β’ not unique; there may be several possible choices for π, πΏ, π
β’ interpretation: permute the rows of π΄ and factor πππ΄ as πππ΄ = πΏπ
β’ also known as Gaussian elimination with partial pivoting (GEPP)
β’ complexity: (2/3)π3 flops
we skip the details of calculating π, πΏ, π
Linear equations 7.16
Example
0 5 52 9 06 8 8
=
0 0 10 1 01 0 0
1 0 01/3 1 00 15/19 1
6 8 80 19/3 β8/30 0 135/19
the factorization is not unique; the same matrix can be factored as
0 5 52 9 06 8 8
=
0 1 01 0 00 0 1
1 0 00 1 03 β19/5 1
2 9 00 5 50 0 27
Linear equations 7.17
Effect of rounding error
[10β5 1
1 1
] [π₯1π₯2
]=
[10
]
solution:π₯1 =
β11 β 10β5 , π₯2 =
11 β 10β5
β’ let us solve using LU factorization for the two possible permutations:
π =
[1 00 1
]or π =
[0 11 0
]β’ we round intermediate results to four significant decimal digits
Linear equations 7.18
First choice: π = πΌ (no pivoting)[10β5 1
1 1
]=
[1 0
105 1
] [10β5 1
0 1 β 105
]β’ πΏ, π rounded to 4 significant decimal digits
πΏ =
[1 0
105 1
], π =
[10β5 1
0 β105
]β’ forward substitution[
1 0105 1
] [π§1π§2
]=
[10
]=β π§1 = 1, π§2 = β105
β’ back substitution[10β5 1
0 β105
] [π₯1π₯2
]=
[1
β105
]=β π₯1 = 0, π₯2 = 1
error in π₯1 is 100%Linear equations 7.19
Second choice: interchange rows[1 1
10β5 1
]=
[1 0
10β5 1
] [1 10 1 β 10β5
]β’ πΏ, π rounded to 4 significant decimal digits
πΏ =
[1 0
10β5 1
], π =
[1 10 1
]β’ forward substitution[
1 010β5 1
] [π§1π§2
]=
[01
]=β π§1 = 0, π§2 = 1
β’ backward substitution[1 10 1
] [π₯1π₯2
]=
[01
]=β π₯1 = β1, π₯2 = 1
error in π₯1, π₯2 is about 10β5
Linear equations 7.20
Conclusion: rounding error and LU factorization
β’ for some choices of π, small errors in the algorithm can cause very large errorsin the solution
β’ this is called numerical instability: for the first choice of π in the example, thealgorithm is unstable; for the second choice of π, it is stable
β’ from numerical analysis: there is a simple rule for selecting a good permutation
(we skip the details, since we skipped the details of the factorization)
Linear equations 7.21
Sparse linear equations
if π΄ is sparse, it is usually factored as
π΄ = π1πΏππ2
π1 and π2 are permutation matrices
β’ interpretation: permute rows and columns of π΄ and factor οΏ½ΜοΏ½ = ππ1 π΄ππ2
οΏ½ΜοΏ½ = πΏπ
β’ choice of π1 and π2 greatly affects the sparsity of πΏ and π: several heuristicmethods exist for selecting good permutations
β’ in practice: #flops οΏ½ (2/3)π3; exact value depends on π, number of nonzeroelements, sparsity pattern
Linear equations 7.22
Conclusion
different levels of detail in understanding how linear equation solvers work
Highest level
β’ x = A \ b costs (2/3)π3
β’ more efficient than x = inv(A) * b
Intermediate level: factorization step π΄ = ππΏπ followed by solve step
Lowest level: details of factorization π΄ = ππΏπ
β’ for most applications, level 1 is sufficient
β’ in some situations (e.g., multiple right-hand sides) level 2 is useful
β’ level 3 is important for experts who write numerical libraries
Linear equations 7.23