steepest decent and cg

8/8/2019 Steepest Decent and CG

http://slidepdf.com/reader/full/steepest-decent-and-cg 1/68

Steepest Decent and Conjugate Gradients (CG)




Solving of the linear equation system b Ax !




Solving of the linear equation system

Problem: dimension n too big, or not enough time for

gauss eliminationIterative methods are used to get an

approximate solution.

b Ax !




Solving of the linear equation system

Problem: dimension n too big, or not enough time for

gauss eliminationIterative methods are used to get an

approximate solution.

Definition Iterative method: given starting point , do

stepshopefully converge to the right solution

b Ax !

0 x

-,,21x xp

x



starting issues



starting issues

Solving is equivalent to minimizingb Ax !

c xb Ax x x f T T !

2

1:)(



starting issues

Solving is equivalent to minimizing

A has to be symmetric positive definite:

b Ax !

c xb Ax x x f T T !

2

1:)(

00 {"! x Ax x A A T T



starting issues

02

1

2

1)(

!

!!! b Axb Ax x A x f symmetric A

T



starting issues

If A is also positive definite the solution of is theminimum

02

1

2

1)(

!


T

b Ax !



starting issues

If A is also positive definite the solution of is theminimum

02

1

2

1)(

!


T

b Ax !

-00

11

2

1

2

1)(

{"

!!d

T T Ad d cb Abd b A f



starting issues

error:

The norm of the error shows how far we are away from the

exact solution, but can¶t be computed without knowing of the exact solution .

x xe ii !:

x



starting issues

error:

The norm of the error shows how far we are away from the

exact solution, but can¶t be computed without knowing of the exact solution .

residual:

can be calculated

x xe ii !:

x

)(: x f Ae Axbr iii !!!



Steepest Decent



Steepest Decent

We are at the point . How do we reach ?i x 1i x



Steepest Decent

We are at the point . How do we reach ?

Idea: go into the direction in which decreases most

quickly ( )

i x 1i x

)( x f

ii r x f !! )(



Steepest Decent



quickly ( ) how far should we go?

i x 1i x

)( x f

ii r x f !! )(



Steepest Decent




Choose so that is minimized:

i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E



Steepest Decent





i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E

0)( ! ii r x f

d

d E

E



Steepest Decent





i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E

0)( ! ii r x f

d

d E

E0)( ! iii r r x f E



Steepest Decent





i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E

0)( ! ii r x f

d

d E


0))(( ! iii r br x E



Steepest Decent





i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E

0)( ! ii r x f

d

d E



i

r

iii r xbr r

i

)()(

!E



Steepest Decent





i x 1i x

)( x f

ii r x f !! )(

E )( ii r x f E

0)( ! ii r x f d

d E



i

r

iii r xbr r

i

)()(

!Ei

T

i

i

T

i

Ar r

r r !E



Steepest Decent

one step of steepest decent can be calculated as follows:

iiii

i

iii

ii

r x x

r r

r r

xbr

E

E

!

!

!

1



Steepest Decent

one step of steepest decent can be calculated as follows:

stopping criterion: or with an givensmall

It would be better to use the error instead of the residual,

but you can¶t calculate the error.

iiii

i

iii

ii

r x x

r r

r r

xbr

E

E

!

!

!

1

maxii " 0r r i I

I



Steepest Decent

Method of steepest decent:

1

)(

0

00max

0

0

n

n

n

n

"

nn

n

ii

Axbr

r x x

Ar r

r r

r r r r and iiwhile

r r Axbr

i

T

T

T T

E

E

I



Steepest Decent

As you can see the starting point is important!



Steepest Decent

As you can see the starting point is important!

When you know anything about the solution use it to guessa good starting point. Otherwise you can choose a starting

point you want e.g. .00! x



Steepest Decent - Convergence




Definition energy norm: A x x x A!:




Definition energy norm:

Definition condition:

( is the largest and the smallest eigenvalue of A)

A x x x A!:

min

max

: P

P

O !

maxP minP




Definition energy norm:

Definition condition:

( is the largest and the smallest eigenvalue of A)

convergence gets worse when the condition gets larger

A x x x A!:

min

max

: P

P

O !

maxP minP

A

i

Ai ee0

1

1¹ º

¸©ª

¨

e

O

O



Conjugate Gradients



Conjugate Gradients

is there a better direction?

Idea: orthogonal search directions110

,,, nd d d -



Conjugate Gradients


Idea: orthogonal search directions110

,,, nd d d -

§

!

!1

0

n

i

iid x P



Conjugate Gradients


Idea: orthogonal search directions

only walk once in each direction and minimize

maximal n steps are needed to reach the exact solution

110,,, nd d d -

§

!

!1

0

n

i

iid x P



Conjugate Gradients


Idea: orthogonal search directions

only walk once in each direction and minimize

maximal n steps are needed to reach the exact solution

has to be orthogonal to

110,,, nd d d -

§

!

!1

0

n

i

iid x P

1 ie id



Conjugate Gradients

example with the coordinate axes as orthogonal search

directions:



Conjugate Gradients

example with the coordinate axes as orthogonal search

directions:

Problem: can¶t becomputed because

(you don¶t know !)i

T

i

i

T

ii

d d

ed !E

ie



Conjugate Gradients

new idea: A-orthogonal110,,, nd d d -



Conjugate Gradients

new idea: A-orthogonal

Definition A-orthogonal: A-orthogonal

(reminder: orthogonal: )

110,,, nd d d -

ji d d , 0! j

T

i Ad d

ji d d , 0! j

T

i d d



Conjugate Gradients




now has to be A-orthogonal to

110,,, nd d d -

ji d d , 0! j

T

i Ad d

ji d d , 0! j

T

i d d

1ie id



Conjugate Gradients





110,,, nd d d -

ji d d , 0! j

T

i Ad d

ji d d , 0! j

T

i d d

1ie id

iT i

i

T

i

iT i

i

T

i

i Ad d

r d

Ad d

Aed !!E



Conjugate Gradients





can be computed!

110,,, nd d d -

ji d d , 0! j

T

i Ad d

ji d d , 0! j

T

i d d

1ie id

iT i

i

T

i

iT i

i

T

i

i Ad d

r d

Ad d

Aed !!E



Conjugate Gradients

A set of A-orthogonal directions can be found with n

linearly independent vectors and conjugate Gram-

Schmidt (same idea as Gram-Schmidt).iu



Conjugate Gradients

Gram-Schmidt:

linearly independent vectors10,, nuu -



Conjugate Gradients

Gram-Schmidt:

linearly independent vectors10,, nuu -

j

T

j

j

T

i

i j

i

j

ji jii

d d

d u

d ud i

ud

!

!"

!§

!

F

F1

0

00

:0



Conjugate Gradients

Gram-Schmidt:

linearly independent vectors

conjugate Gram-Schmidt:

10,, nuu -

j

T

j

j

T

i

i j Ad d

Ad u! F

j

T

j

j

T

i

i j

i

j

ji jii

d d

d u

d ud i

ud

!

!"

!

§

!

F

F1

0

00

:0



Conjugate Gradients



Schmidt (same idea as Gram-Schmidt).

CG works by setting (makes conjugate Gram-

Schmidt easy)

iu

ii r u !



Conjugate Gradients



Schmidt (same idea as Gram-Schmidt).

CG works by setting (makes conjugate Gram-

Schmidt easy)

with1! iiii d r d F11

!i

T

i

i

T

ii

r r

r r F

ii r u !

iu



Conjugate Gradients

0:1

0

1

!!!! §§

!!

!

n

jk

k

T

ik

n

jk

k k

T

ii

T

i j

T

i Ad d d Ad Aed r d ji

PP

§

!!

1

0

i

k k ik ii d ud F



Conjugate Gradients

0:1

0

1

!!!! §§

!!

!

n

jk

k

T

ik

n

jk

k k

T

ii

T

i j

T


PP

§

!!

1

0

i

k k ik ii d ud F

]

§

!!

!!

1

00

0:i

k jk

j

T

k ik j

T

i j

T

i r d r ur d ji F



Conjugate Gradients

0:1

0

1

!!!! §§

!!

!

n

jk

k

T

ik

n

jk

k k

T

ii

T

i j

T


PP

§

!!

1

0

i

k k ik ii d ud F

]

§

!!

!!

1

00

0:i

k jk

j

T

k ik j

T

i j

T

i r d r ur d ji F

jir u j

T

i ! 0



Conjugate Gradients

0:1

0

1

!!!! §§

!!

!

n

jk

k

T

ik

n

jk

k k

T

ii

T

i j

T


PP

§

!!

1

0

i

k k ik ii d ud F

]

§

!!

!!

1

00

0:i

k jk

j

T

k ik j

T

i j

T

i r d r ur d ji F

jir u j

T

i ! 0

i j j

T

i

j

T

iii

r r

jir r r u

H !

!! 0:



Conjugate Gradients

0:1

0

1

!!!! §§

!!

!

n

jk

k

T

ik

n

jk

k k

T

ii

T

i j

T


PP

i j j

T

i

j

T

iii

r r jir r r u

H !

!!0

:

]

i

T

i

i

k

jk

j

T

k ik i

T

ii

T

i r ur d r ur d !! §

!

!

1

00

F



Conjugate Gradients

ji Ad d

Ad r

j

T

j

j

T

i

i j "! F



Conjugate Gradients

ji Ad d

Ad r

j

T

j

j

T

i

i j "! F

j j j j j j j jAd r d e A Aer EE !!!

)(

11



Conjugate Gradients

ji Ad d

Ad r

j

T

j

j

T

i

i j "! F


)(

11

j

T

i j j

T

i j

T

i Ad r r r r r E!1



Conjugate Gradients

ji Ad d

Ad r

j

T

j

j

T

i

i j "! F


)(

11

j

T

i j j

T

i j

T

i Ad r r r r r E!1

1! j

T

i j

T

i j

T

i j r r r r Ad r E



Conjugate Gradients

1! j

T

i j

T

i j

T

i j r r r r Ad r E

±±±

°

±±±

¯

®

{{

!

!

!

!

10

11

ji ji

jir r

jir r

Ad r

r r

i

i

T

i

i

i

T

i

j

T

i

i j j

T

i

E

E

H



Conjugate Gradients

°¯

"!

10 jii j F



Conjugate Gradients

±°

±¯

®

!!

"

!

! 1

10

1111111

jir r

r r

r d

r r

Ad d

r r

ji

i

T

i

i

T

i

i

T

i

i

T

id ef

i

T

ii

i

T

ii j

E

E

F



Method of Conjugate Gradients:

0

0

0

n

nn

n

i

r r r d

Axbr

1

)(00max

n

n

n

n

n

n

n

"

ii

d r d

r r

r r

A xbr

r r

d x x

Ad d

r r

r r r r and iihile

old

T

old

T

old

T

T

T T

F

F

E

E

I



Conjugate Gradients - Convergence

steepest decent and cg

Documents