08.03.2016
SOLUTION METHODS for f(x)=0
Single Nonlinear Equation o The Bisection Method of Bolzano o Method of False Position (Regula Falsi Method) o Method of Linear Iteration (Fixed Point Iteration) o Newton's Method o Secant Method
System of nonlinear equations o Jacobi Method o Modified Jacobi Method o Newton's Method
Single Nonlinear Equation
f(x)=0 (1)
Example : ex-x ln(x)=0
Our aim is to find the value of x which satisfies Eq. (1). That value of x is called a root of Eq. (1). When f is linear function of x, f(x)=0 will be a linear equation (LE). When f is nonlinear (NL) function of x, f(x)=0 will be a nonlinear equation (NLE). LE’s or NLE's arise from physical problems.
Example for linear equation (LE):
f(x)=ax+b, f(x)=0 ax+b=0 x=-b/a
where x is the root of the LE. Unique solution!
Nonlinear equation (NLE)
If f(x) is NL function of x, it might have no roots or one root or more than one root.
Example
f(x)=xtan(x)-1 Nonlinear equation (NLE)
f(x)=0 xtan(x)-1=0.
To make a crude plot, xtan(x)=1, then tan(x)=1/x
Then graph is as follows:
From graph, function has many roots.
The Bisection Method of Bolzano A continuous function f(x)=0 is considered. Choose initial interval [a,b] where f(a) and f(b) have different signs.
2
bac
If f(a) and f(c) have opposite signs (i.e. f(a)*f(c)<0), then there is a root in [a,c] If f(c) and f(b) have opposite signs (i.e. f(c)*f(b)<0), then there is a root in [c,b] If f(c)=0, then c is a root.
Start with [a0,b0], find c0
2
bac 00
0
If f(a0) and f(c0) have opposite signs then a1=a0 and b1= c0. If f(c0) and f(b0) have opposite signs then a1=c0 and b1=b0, Therefore the new interval that contains root is 11 b,a
2211
111 b,a 2
bacb,a
3322
222 b,a2
bacb,a
2
bacb,a nn
nnn
When n goes to infinity or for very large value of n nn1n1n c,ab,a or
nn1n1n b,cb,a for all n. And nn b,ar or 1n1n b,ar .
012n1n1nn210 bbbbbraaaaa
Theorem: (Bisection method) Assume that b,aC)x(f and there exists a number b,ar such that f(r)=0. If f(a) and f(b)
have apposite signs and 0nnc represents the sequence midpoints generated by the bisection
process, and 1n
00n 2
abcr
for n=0,1,2,... then the sequence 0nnc converges to the root
r, that is rcLim nn
Proof :
nnn
nn
bca
bra
1nallfor2
abcr nn
n
2
abab 00
11
200
00
1122 2
ab
22
ab
2
abab
and so on, 22
abab
n00
nn
Combining (1) and (2)
22
ab
crn
00
n
, then 1n
00n 2
abcr
for all n.
1n
00n
n 2
abcr Lim
1n
00
nn
n 2
ab Limcr Lim
it is obvious that 02
ab Lim
1n
00
n
then left hand side 0rcLim nn
rcLim nn
or rcn nc is a root.
Therefore, proof of theorem also gives us error expression for Bisection method which is
nn cre or 1n
00n 2
abe
Example f(x)=xsin(x)-1 and f(x)=0 xsin(x)-1=0 sin(x)=1/x
From graph, initial interval can be taken as [0,2] Since f(0)= -1 and f(2)= 0.818595 There is a root in [0,2]
k left end point ak midpoint ck right end point bk f(ck)
0 1 2 3 4 5 6 7 8
0.000000 1.000000 1.000000 1.000000 1.000000 1.062500 1.093750 1.109375 1.109375
1.000000 1.500000 1.250000 1.125000 1.062500 1.093750 1.109375 1.117188 1.113281
2.000000 2.000000 1.500000 1.250000 1.125000 1.125000 1.125000 1.125000 1.117188
-0.158529 0.496224 0.186231 0.015051 -0.071827 -0.028362 -0.006643 0.004208 -0.001216
For two decimal place accuracy, root is 1.11 from table, 8cr . Then error is
1n
00n 2
abe
00390625.0
2
02e
188
for 31 iteration error will be 1013131 10x65.4
2
02e
Method of False Position(Regula Falsi Method)
ac
af0
ab
afbf
ac
af0m
ab
afbfm
afbf
abafac
(*)
cb
0bf
ab
afbf
cb
0bfm
ab
afbfm
afbf
abbfbc
(**)
The three possibilities are:
If f(a) and f(c) have opposite signs, (f(a)*f(c)<0) then root lies in [a,c] If f(c) and f(b) have opposite signs, (f(c)*f(b)<0) then root lies in [c,b] If f(c)=0, then c is the root.
For an iteration, Equation (**) is chosen. (Equation (*) may also be chosen)
nn
nnnnn afbf
abbfbc
Example
f(x)=xsin(x)-1
Take initial interval as [0,2]
ii
iiiii afbf
abbfbc
Starting with i=0
10 00 afa
81859485.02 00 bfb
09975017.1
181859485.0
0281859485.020
c
01010 02001921.0)1( bbcafcf
12124074.1
02001921.081859485.0
09975017.1281859485.021
c
12121 00983461.0 cbaacf
k ak ck bk f(ck)
0
1
2
3
0.0000000
1.09975017
1.09975017
1.09975017
1.09975017
1.12124074
1.11416120
1.11415714
2.0000000
2.0000000
1.12124074
1.11416120
-0.02001921
0.00983461
0.000000563
0.000000000
Method of Linear Iteration (LIM) (or Fixed Point Iteration)
f(x)=0 (1)
Put (1) into the form:
x=g(x) (2)
The representation of Eq. (2) of f(x) is not unique, such as
f(x)=xtan(x)-6
f(x)=0 xtan(x)=6 x=6/tan(x) then g(x)= 6/tan(x)
or
xx
xxxx
6tan
6tan6tan 1 then
xxg
6tan 1
or 6tan xx 06tan xx xxxx 6tan then x6xtanxxg
Derivation of formula for iteration:
Note that if r is a root of f(x)=0, then f(r)=0 and r = g(r).
Choose an initial approximation x0 to a root of f = 0
x=g(x)
x1=g(x0) , x1 is the next improved estimate
x2=g(x1)
x3=g(x2) . . . xi+1=g(xi) i=0,1,2,3... (2)
Claim: If the squence {x0,x1,x2,...} generated by (2) ( xi+1=g(xi) ) converges, let’s say to , then will be one of the roots of f=0
Proof
roottheis
0f
g
xglimxlimxgx
assumedisxlim
ii
1ii
i1i
ii
Therefore is a root.
When does {x0,x1,x2,...} converges ?
In fact it is easy to construct the sequence which will diverge.
f(x)=xtan(x)-1 Nonlinear equation (NLE)
f(x)=0 xtan(x)-1=0. To make a crude plot, xtan(x)=1 tan(x)=1/x
wish to find r1 by LIM. Note that there is a root in the interval 2
r0 1
1xtanxxf
xtan
1x0xf then xtan
1xg
79.04
xChoose 0
.1
4tan
1
xtan
1x
01
64.0.1tan
1
xtan
1x
12
i xi
0 0.79
1 1.
2 0.64
3 1.34
4 0.24
5 4.12
|xi+1-xi|
0.21
0.26
0.7
1.1
3.88
It diverges since relative simple error |xi+1-xi| is increasing while i is increasing.
Converging case:
1xtanxxf x
1xtan0xf then
x
1arctanxg
79.04
xChoose 0
91.0
4
1arctan
x
1arctanx
01
84.091.0
1arctan
x
1arctanx
12
i xi |xi+1-xi|
0 0.79
1 0.91 0.12
2 0.84 0.07
3 0.87 0.03
4 0.86
5 0.86
It converges since error |xi+1-xi| is decreasing while i is increasing.
1r86.0x which is correct to 2 rounded decimals
21 105.086.0r
CONVERGENCE CONDITION FOR LIM
{x0,x1,x2...} will converge if :
a) g & g' are continuous in I
b) 1xg in I
c) x0 is chosen to be in “I”
Theorem : Let x=r be a root. Let “I” be an interval containing this root
i. g(x) and g'(x) are continuous in I, ii. |g'(x)|<1 for all points in I,
iii. initial approximation x0 is chosen in I,
then the iteration converges to root r
x=g(x) equivalent form of f(x)=0 to iterate
r=g(r) (1)
On the other hand,
xi+1=g(xi) (2) recursive formula for LIM
Subtracting (2) from (1)
xi+1-r=g(xi)-g(r)
By mean value theorem, there exists a point i between xi and r. Then
g(xi)-g(r) = g'(ηi)(xi-r)
xi+1-r = g'(ηi)(xi-r)
let i=0
x1-r = g'(η0)(x0-r)
Taking absolute values of both sides
x
a b
I
r x0
f(x)
rxgrx 001 )(
rx 1 error in first iteration and rx 0 error in initial guess.
For convergence rxrx 01 implies that 1g 0
let i=1
x2-r = g'(η1)(x1-r)
Taking absolute values of both sides
rxgrx 112 )(
rx 2 error in 2nd iteration and rx 1 error in first iteration.
For convergence rxrx 12 implies that 11 g
when iteration converges, (ei+1 =|xi+1-r| and ei =|xi-r|)
ii
iii
1ii
erglimeglimelim
Since rx ii when i goes to infinity, in short
ei+1=constant*ei for i>>1
This result implies that; the proportion of the error at any iteration to the error at previous iteration is constant when "i" becomes very large. This is the reason why this iteration or method is called linear and named as “Linear Iteration Method”..
Proof of Convergence:
Start with i=0 in xi+1-r = g'(ηi)(xi-r)
It is assumed that 1g 0 and x1-r =
g'(η0)(x0-r) or rxgrx 001 and
rx0
Then rxrx 01 which means that x1 is in the interval I.
x
a b
I
r x0
f(x)
Next, take i=1,
rxrxgrx 1112
e1 =|x1-r|, e2 =|x2-r| errors in iterations. Then the error expression for LIM is:
ii1i ege for any i (ei+1 =|xi+1-r| and ei =|xi-r|)
Since power of ei is 1, it is said to be the rate of convergence is 1 or linear. Now let us introduce positive quantities k0, k1,…, ki defined by
ii1100 gkgkgk
and K, such that K=max(k0, k1,…, ki,…), since 0, 1, …, i,… are all in interval I . Due to assumption in the theorem k0, k1,…, ki,… are all less than one, hence K<1.
ii1i ege or i1i eKe
which is true for any i. It may be written recursively
1ii eKe , 2i1i eKe , 3i2i eKe
therefore, one gets
0rxlimelim i
ii
i
since K<1 and 0Ki , then xi = r.
Error Expression in General
i1i xgx (I)
rgr (II)
(II)-(I) rgxgrx i1i (III) Taylor expansion of g(xi) about r
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxg 3
i2
iii
0i
iii
2i2
1ii eKeKeKeKe
0xxlimrxlim
rxlimi1i
i1ii
ii
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxg 3
i2
iii (H.O.T stands for Higher Order Terms) And rewriting the Equation,
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxgrx 3
i2
iii1i
T.O.Hrxrg!3
1rxrg
!2
1rxrgrx 3
i2
ii1i
Where rxe 1i1i and rxe ii
Then error expression takes following form in general:
T.O.Herg!3
1erg
!2
1erge 3
i2ii1i
Where ierg is dominating term and rest of the terms may be assumed as higher order terms
and neglected, therefore error expression for LIM is as follows:
i1i erge Note that when i goes to infinity, in short rxx ii1i .
Example 0x3exf x
X f(x) - 0 1 1 -0.28 2 1.39 Wish to find r1 and r2 by LIM
x3lnelnx3e0x3e xxx x3lnxgx3lnx (*)
1x1when1xg
1x
1x1xwhen1xg
x
1xg
x
13
x3
1xg
(*) is suitable for finding r2 provided that x0 is choosen in (1,). (*) is not suitable for finding r1.
For finding r2 choose x0=2 x1=ln(3 x0)=ln(6)=1.79 x1=1.68 x2=1.62 x3=1.58 x8=1.52 x9=1.52 correct to two rounded decimals choose x0=1.50 (observe what is happening) x1=1.50 x2=1.51 The form x=ln(3x) is suitable for r2. To find r1, we should find a different g(x). Example x2xsin4.01xf , Roots of f(x)=?
1x2xsin4.00x2xsin4.010xf Wish to find root r, correct to 2 decimal places, by LIM 0xf
xsin4.01x20x2xsin4.01 xsin2.05.0x
xsin2.05.0xg and xcos2.0xg
xallfor12.0xcos2.0xg then g(x) is suitable for arbitrary x0
Choosing x0=0.00 x1= g(x0) =0.50 x2= g(x1) =0.60 x3=0.61 x4=0.61 then r0.61=x4 and
244 105.0rxe
x
y
-1r
y=2x-1
y=0.4sin(x)
Now we like to find the number of additional iterations which assumes for 4 decimal place accuracy.
4
m
4m4
4
2
4556
44445
ii1i
exge
exgege
exgege
ege
where m is the number of additional iterations. Wish to find m so that 4
m4 105.0e
2m
4m4 105.0xge and 4
m4 105.0e then 42m
4 105.0105.0xg
2m
4
42m
4
10xg
105.0105.0xg
1639.061.0cos2.0xgxcos2.0xg 4
10ln21639.0lnm101639.0 2m 3m54.2m
x4=0.61 x5=g(x4) x5= 0.6146 similarly x6=0.6153
x7=0.6154 x8=0.6154
Newton's Method (NM)
f(x)=0 xi : initial approach hi: correction so that xi+1 =xi+hi
For large values of i, r x 1i and 0 )f(xf(r) 1i f(xi+hi)=0, wish to find h so that the equation is satisfied. Taylor expansion of f(xi+hi) about xi:
T.O.H
i
2
iiiii xf!2
hxfhxfhxf
i
Assuming h is small, f(xi+hi)=0 and neglecting Higher Order Terms (H.O.T)
i
iiiii xf
xfh0xfhxf
xi+1 =xi+hi
i
ii1i
h
i
ii1i xf
xfxx
xf
xfxx
i
where xi+1 is the next approximate value for r. i starts from zero
1
112
0
001
xf
xfxx
xf
xfxx
i
ii1i xf
xfxx
(i=0,1,2...) Recursive formula for the Newton’s Method.
Claim: If {x0,x1,x2,...} generated by recursive formula of Newton's Method converges to , then is a root of f.
Proof:
For enough large values of i, αi
x1i
x
f
f
xf
xfxlimxlim
i
ii
i1i
i
0f0fIf rootais
Since i1i
i
ii1i xgx
xf
xfxx
is the same form as that of LIM. But in NM, g(xi) has
the specific form: i
iii xf
xfxxg
In NM
Axf
xfxxgx
xf
xfxxg
(A) can be derived in the following way too
xf
fx0
f
f0
f
f0f
. By indexing
i
ii1i xf
xfxx
Or geometrically from the graph by writing slope of straight line tangent to the function f(x) at ix
Red line is tangent to the curve at xi.
i1ii
i xfxx
xf)tan(
i
ii1i xf
xfxx
Convergence Condition for NM
2xf
xfxfxfxf1xg
xf
xfxxg
NMforxf
xfxfxg
xf
xfxf11xg 22
if r is a simple root and derivative of f at x=r is different than zero then,
0rf
rfrfrg 2
Since r is a simple root and 0rf . This implies that
Always there is an interval I containing the root r in which |g'|<1 NM will produce always convergent sequence provided that x0 is in I (that is
sufficiently close to r)
Pi
Pi-1
xi+1 xi
x
a b
I
r x0
f(x)
Example
f(x)=ex-3x=0
wish to find r1and r2 by NM
i
ii1i xf
xfxx
i=0,1,2,...
3exfx3exf xx
3
31
i
i
xi
x
ii e
xexx 00.00 x
3
0*30
3
30
00
01 0
0
e
e
e
xexx
x
x
=0.5,
3
5.0*30
3
35.0
5.01
12 1
1
e
e
e
xexx
x
x
=0.61
62.0
62.0
62.0
61.0
50.0
00.0
1
4
3
2
1
0
r
x
x
x
x
x
54.1r
54.1x
54.1x
68.1x
00.2x
2
3
2
1
0
Rate of Convergence of NM
i
iii1i xf
xfxxgx
(I)
rgr (II)
(I)-(II) rgxgrx i1i (III) Taylor expansion of g(xi) about r
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxg 3
i2
iii
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxg 3
i2
iii (H.O.T stands for Higher Order Terms) And rewriting the Equation,
rgxgrx i1i
T.O.Hrxrg!3
1rxrg
!2
1rxrgrgxgrx 3
i2
iii1i
or
T.O.Hrxrg!3
1rxrg
!2
1rxrgrx 3
i2
ii1i
Where rxe 1i1i and rxe ii
Then error expression takes following form in general:
T.O.Herg!3
1erg
!2
1erge 3
i2ii1i
Where rg is zero in NM and 2ierg
!2
1 is dominating term, and rest of the terms may be
assumed as higher order terms and neglected, therefore error expression for NM is as follows:
2i1i erg
!2
1e Note that when i goes to infinity, rxx ii1i .
or
i
2i
ii g!2
rxrgrxrgxg
2ii1i rxg!2
1rx or 21 !2
1rx grx iii
2ii1i eg
!2
1e where i is between xi and r. Since ei has power of 2, NM converges
quadratically. So it is two times faster than LIM (since in linear iteration method the first term ierg is dominating term and the power of ei is 1 than means that the rate of convergence is
1 or linear.)
Now, it is shown that
Convergence of LIM is linear Convergence of NM is quadratic
Example
f(x)=ex-3x=0
LIM
3
ex
x
suitable for finding r1 in LIM
61.0
61.0
60.0
59.0
57.0
53.0
46.0
33.0
00.0 8
7
6
5
4
3
2
1
0
x
x
x
x
x
x
x
x
x
The root is found in 8 interations using LIM.
NM
62.0
62.061.0
50.0
00.0
4
3
2
1
0
x
xx
x
x
So 62.0xr 41 for two decimal place accuracy. The root is found in 4 iterations using NM.
Advantages of NM
xf
xfxxg
is given in NM
NM is suitable for finding all roots of f(x)=0
Example
2
2m
cr L
EIrP
rm is the absolutely smallest root of xtan(x)-6=0
x
6xtan06xtanxxf
20 mr
6tan xxxf x
xxxf
2costan and
i
iii xf
xfxx
1
Choose x0=1.2
x1=1.449
x2=1.394
x3=1.350 rm
x4=1.350
Convergence criteria
It may be shown that the sequence {xi}={x0,x1,x2...} generated by NM converges to r provided that x0 is chosen sufficiently close to r.
If ab)a(f
)a(f
and ab
)b(f
)b(f
and x0 is chosen
in [a,b], then the sequence {xi} generated by NM would converge to r
Example
f(x)=sin(x)-(x/2)2 . Chosen initial interval is [a,b]=[1.5,3]. 5.1ab64.0)5.1(f
)5.1(f
and
5.1ab84.0)3(f
)3(f
. Then, if x0 is chosen in [1.5,3], NM will converge to the root r.
Claim: The convergence of NM to a double root, if converges, would be linear.
Verification: Assume that r is a double root. Double root means that 0rf , and 0rf
but 0rf
NM xi+1 =xi+hi and i
ii xf
xfh
,
i
ii1i xf
xfxx
xf
xfxxg
And error expression is T.O.Herg!3
1erg
!2
1erge 3
i2ii1i
2xf
xfxfxfxf1xg
xf
xfxxg
b
a
b-a
|f(a)/f’(a)|
|f(b)/f’(b)|
r
y=f(x)
22 xf
xfxfxg
xf
xfxf11xg
rgxgLimrx
,
2rx2rx xf
xfLimrf
xf
xfxfLimrg
indefinite. Then by using
L'Hôpital's rule (LR)
xf*xf*2
xfLimrfrg
rx
= 2
1
rf2
1rf
T.O.Herg!3
1erg
!2
1erge 3
i2ii1i
i1i e2
1e errors at two successive iterations are related linearly, it means that the
convergence of NM to a double root is linear.
Claim: The convergence of the formula i
ii1i xf
xf*2xx
which is called modified NM-1
(MNM-1) to a double root would be quadratic; but the convergence of this formula to a simple root would be linear. Verify it!
Claim: But the convergence of the formula ii1i hxx with i
ii xu
xuh
and
i
ii xf
xfxu
which is called modified NM-2 (MNM-2) to a double root or simple root
would be quadratic. Summary:
i
ii1i xf
xfxx
Regular NM
i
ii1i xf
xf*2xx
Modified NM-1
i
ii1i xu
xuxx
,
i
ii xf
xfxu
Modified NM-2 Simple Root
Quadratic Linear Quadratic
Double Root
Linear Quadratic Quadratic
In general, for multiple roots: f(x)=0 and assume that r is a multiple root with the multiplicity M, there are M roots at x=r, i.e 0rf , 0rf , ..., 0rf 1M and 0rf M .
It can be written that )x(h)rx(xf M , where 0rh .
Claim: Then the convergence of the formula i
ii1i xf
xf*Mxx
to a multiple root r with
the multiplicity M would be quadratic but to a simple root it would be linear. And the error
relation is i1i eM
1Me
. Verify!
SECANT METHOD: Iterative formula could be derived geometrically by using following graph
Slope of straight line between ix and 1ix is
i1i
i1i
xx
xfxf
And between 1ix and ix is
1ii
i
xx
xf
.
Then by equating those slopes:
i1i
i1i
xx
xfxf
=
1ii
i
xx
xf
Then 1ii
1iii1i1i xfxf
xfxxfxx
.
To use the formula, two initial approximations or guesses which are 0x and 1x should be
considered . Iteration starts with 0x and 1x and result of the first iteration will be
01
01102 xfxf
xfxxfxx
And then second, third, fourth etc...
12
12213 xfxf
xfxxfxx
.
.
f(xi-1)
xi+1 xi xi-1
f(xi)
Example
f(x)=1+0.4sin(x)-2x=0
0.4sin(x)=2x-1
choose x0=0, x1=1 and 01
01102 xfxf
xfxxfxx
x2=0.60
x3=0.62
x4=0.62 r