simple iteration - boise state universitycalhoun/teaching/math...direct methods 2 we have so far...
TRANSCRIPT
![Page 1: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/1.jpg)
Numerical Analysis
Iterative Methods
1
Simple Iteration
1Tuesday, October 29, 13
![Page 2: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/2.jpg)
Direct Methods
2
We have so far focused on solving Ax = b using directmethods.
• Gaussian Elimination• LU Decomposition• Variants of LU, including Crout and Doolittle• Other decomposition methods including QR
Advantages • Algorithm terminates in a fixed number of
iterations• Can take advantage of sparsity (to some extent)
2Tuesday, October 29, 13
![Page 3: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/3.jpg)
Direct Methods
3
Possible disadvantages?
• We have to store the matrix, or at least know what the entries are
• No obvious way to get a solution that is “close enough” to the exact solution for whatever purpose we have in mind
• No way to reduce the operation count unless we have a special sparse structure.
3Tuesday, October 29, 13
![Page 4: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/4.jpg)
Iterative methods - scalar case
4
Can we apply what we know about root-finding to the problem of finding the solution to a linear system of equations?
Newton’s method, when applied to a scalar linear equation converges in one step:
r = b� ax
f(x) = b� ax
Newton : x1 = x0 �b� ax
�a
=b
a
4Tuesday, October 29, 13
![Page 5: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/5.jpg)
Iterative methods - scalar case
5
f(x) = 5� 3x
g(x) =1
2x+
5
6
We can formulate a fixed point problem with the func-
tion g(x) defined as
g(x) =
1
m
(b� ax) + x =
⇣1� a
m
⌘x+
b
m
where we choose m so that |g0(x)| < 1 (choose m = 2a,
for example).
5Tuesday, October 29, 13
![Page 6: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/6.jpg)
Iterative methods - matrix case
6
We can apply Newton’s Method to the function
F (x) = b�Ax
where x,b 2 Rn and A 2 Rn⇥n.
We get
xk+1 = xk � [F 0(xk)]�1
F (xk)
= xk +A�1 (b�Axk)
This also gives us the answer in exactly one step!
6Tuesday, October 29, 13
![Page 7: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/7.jpg)
Iterative methods - matrix case
7
Appling the fixed point iteration to F (x) = b�Ax, we
get
g(x) = x+M�1(b�Ax)
where now M 2 Rn⇥n.
The fixed point iteration then looks like
xk+1 = xk +M�1 (b�Axk)
The choice of M will affect the convergence of the scheme.
7Tuesday, October 29, 13
![Page 8: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/8.jpg)
Comparison
8
Newton step: xk+1 = xk +A�1 (b�Axk)
Fixed point step: xk+1 = xk +M�1 (b�Axk)
The Newton step is no different than using a direct method, since we have to invert A at each step.
The fixed point approach is promising, if we can understand how different choices for M affect convergence of the scheme.
8Tuesday, October 29, 13
![Page 9: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/9.jpg)
A simple iteration
9
xk+1 = xk +M�1 (b�Axk)
Some classic schemes. Note that for each scheme, M approximates A.
M = D The Jacobi iteration
M = L The Gauss-Seidel iteration
M = !�1D � L The Successive Over Relaxation or SOR Method
where D is the diagonal of A and L is the lower triangular portion of A (including diagonal).
The matrix M for each of these choices is easy to invert!9Tuesday, October 29, 13
![Page 10: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/10.jpg)
A simple iteration scheme
10
For the scalar case, we have that
g(x) =
1
m
(b� ax) + x =
⇣1� a
m
⌘x+
b
m
which will converge to a root of f(x) = b� ax if
|g0(x)| =���1�
a
m
��� < 1
and the closer m is to a, the faster the convergence. Infact, if m = a, we get convergence in one step.
10Tuesday, October 29, 13
![Page 11: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/11.jpg)
A simple iteration scheme
11
Do we also have to have a condition like |g0(x)| < 1?
In the matrix case, we have
g(x) = x+M�1(b�Ax)
=
�I �M�1A
�x+M�1
b
we also want M to be as close to A as is feasible, while
still being easily invertible. If M = A�1, we converge
in one step.
11Tuesday, October 29, 13
![Page 12: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/12.jpg)
Convergence?
12
Let A be an n ⇥ n matrix. The spectral radius of a
matrix is defined as
⇢(A) = max {|�| : � is an eigenvalue of A}
The following are equivalent
• ⇢(A) < 1
• Ak ! 0 as k ! 1, and
• Ax ! 0 as k ! 1 for any vector x.
If one of the above conditions is true, they are all true.
12Tuesday, October 29, 13
![Page 13: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/13.jpg)
Convergence
13
Let ek = A�1b� xk. Then
ek = (I �M�1A)ek�1 = . . . = (I �M�1A)ke0
Then
kekk k(I �M�1)kk ke0k
And the iteration converges for every initial guess if and only if
⇢(I �M�1A) < 1
13Tuesday, October 29, 13
![Page 14: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/14.jpg)
Preconditioned system
14
In a general splitting method, we write
A = P �Q
This leads to the iterative scheme
Pxk+1 = Qxk + b
xk+1 = P�1Qxk + P�1b
or
14Tuesday, October 29, 13
![Page 15: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/15.jpg)
A simple iteration
15
Our method
M�1Ax = M�1b
applied to the preconditioned system
xk+1 =�I �M�1A
�xk +M�1
b
{{M�1A = I � (I �M�1A)
P Q
can be viewed as a simple splitting method with splitting
where M is a preconditioner.15Tuesday, October 29, 13
![Page 16: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/16.jpg)
Fixed point iteration
16
In the scalar case, we could have chosen m directly :
1
m
(b� ax) = 0
to ensure convergence of the fixed point method.
���1�a
m
��� < 1
This led to the requirement that
Of course, there were many other ways that we could have designed the fixed point scheme.
16Tuesday, October 29, 13
![Page 17: Simple Iteration - Boise State Universitycalhoun/teaching/Math...Direct Methods 2 We have so far focused on solving Ax = b using direct methods. • Gaussian Elimination • LU Decomposition](https://reader033.vdocuments.site/reader033/viewer/2022042219/5ec4c3902ba0ab269c51a0ef/html5/thumbnails/17.jpg)
Simple Iteration Algorithm
17
Given an initial guess x0, compute r0 = b� Ax0, and
solve Mz0 = r0.
For k = 1, 2, . . .,Set xk = xk�1 + zk�1.
Compute rk = b�Axk.
Solve Mzk = rk
17Tuesday, October 29, 13