On implicit-factorization block preconditioners

Sue Dollar1,2, Nick Gould3,Wil Schilders2,4 and Andy Wathen1

1 Oxford University Computing Laboratory, Oxford, UK

2 CASA, Technische Universiteit Eindhoven, The Netherlands

3 Rutherford Appleton Laboratory, Chilton, UK

4 Philips Research Laboratory, Eindhoven, The Netherlands

Summary of the talk

• Introduction

• Direct versus Iterative Methods

• Preconditioned Conjugate Gradient Method

• Constraint Preconditioners

• Implicit Constraint Preconditioners

• Numerical Examples

• Future work and conclusions

Introduction

Interested in solving structured linear systems of equations

• • full rank• positive semi-definite

T

bH

x cA B

y dB C

n nA Rm nB Rm mC R

Example 1: Equality constrained minimization

Interior-point sub-problem

• • • Θ (small) barrier weights

First-order optimality

minimize ( ) subject to ( ) 0, 0nx

f x c x x

R

11minimize ( ) subject to

2n

T T

ss A s s g By c

R

( ), symmetric approximation to ( )x xxg f x A f x ( )xB c x

1

0

T s gA B

y cB

Factory

number of hours worked by person

hourly rate of pay for person

number of object that person makes each hour

number of object that need to be made each week

{ }, { }, { }

i

i

ji

j

i i ji

x i

c i

b j i

d j

x x c c B b

, { }jd d

min such that , 0T

xc x Bx d x

Example 2: Inequality constrained minimization

Interior-point sub-problem

• C (small) barrier weights

First-order optimality

1 11minimize ( ) ( )

2n

T T T T

ss A B C B s s g B C c

R

minimize ( ) subject to ( ) 0nx

f x c x

R

T s gA B

y cB C

Direct vs. Iterative Methods

• Direct methods– Gaussian elimination with a pivoting strategy

– Bunch-Parlett factorization

• Iterative methods– Krylov subspace methods

– MINRES & GMRES – find solution of (1) within n+m iterations

– CG may fail because of the indefiniteness of system

– Often advantageous to use a preconditioner P

PCGPossible to use the preconditioned conjugate-gradient method

•

•

•

•

•

rank( )C l, , and invertibleT m l l lC EDE E D R R

, we may rewrite (1) asTz DE y

1

0

0 0

0

T

T

A B x c

D E z

B E y d

1

2

forms a basis of null-space of Z x

Z B E ZwzZ

1(2) becomes T

ZA w Z c

11 1 2 2 positive definiteT T

ZA Z AZ Z D Z

(Gould, Hribar, Nocedal)

(2)

Projected PCG

– Can perform iteration in the original (x, z) space so long as preconditioner chosen carefully

… Solve

Iterate until convergence

– …

– Solve

– …

11 1 2 2 positive definiteT T

NG Z GZ Z D Z

0

T r cG B

uB C

T r cG B

u wB C

Constraint Preconditioners

Case C=0

–

, ,0 0

T TA B G BH P

B B

( ) is a basis for the nullspace of n n mZ B R

The matrix P-1H has

– an eigenvalue at 1 with multiplicity 2m

– n-m eigenvalues which are defined by the generalized eigenvalue problem

T Tz zZ AZx Z GZx

(Keller, Gould, Wathen)

Case rank(C)=l

, ,T TA B G B

H PB C B C

( )1

1 1 1

( )2

2 2 2

is a basis for the nullspace of

such that spans

is a basis for the nullspace of C

such that spans

n n m

n m n

m m l

m l m

Z B

Y Y Z

Z

Y Y Z

R

R R

R

R R

The matrix P-1H has• at least 2m-l eigenvalues at 1• n-m eigenvalues which are defined by the generalized eigenvalue problem

• m eigenvalues defined by the generalized eigenvalue problem

where w=[xy; xz], subject to Y2BY1xy≠0.

1 12 2 2 2 2 2 2 2( ( ) ) ( ( ) )T T T T T TA B Y Y CY Y B w G B Y Y CY Y B w

1 1 1 1T T

z zZ AZ x Z GZ x

CVXQP1_S: m=50, n=100, G=diag(A)

CVXQP1_S: m=50, n=100, G=diag(A)

Explicit vs. implicit constraint preconditioner

Explicit constraint preconditioners: choose G, and then factorize

Implicit constraint preconditioners: find easily-invertible factors R and S so that

always holds

Pick parts of R and S to match * (≡ G) to parts of A

TG BP

B C

* TT B

RSRB C

11 21 1

21 22 2

1 2

T T

TT T

G G BG B

RSR G G BB C

B B C

•Easily invertible B1

11 12 1 11 21 31

21 22 2 21 22 32

31 32 33 31 32 33

,

T T T

T T

R R B S S S

R R R B S S S S

R R R S S S

• Require that both R and S are easily block invertible

- some sub-blocks should be zero

Example 1: C=0

11 1 31

21 22 2 22 32

31 31 32 33

0 0 0

, 0

0 0

T T

T T

R B S

R R R B S S S

R S S S

11 11 31 1 1 31 11 1 33 1

21 2 31 11 2 33 1 21 31 1 22 32 1

22 2 31 21 2 32 22 2 33 2 21 31 2

22 32 2 22 22 22

T T T T

T T T T T

T T T T T T

T T

G R S B B S R B S B

G B S R B S B R S B R S B

G B S R B S R B S B R S B

R S B R S R

31 31 22 and +ve definite TR S S

• can recover any G (Schilders)

Example 21

1 31 33 31

22 2 221

31 33 33

0 0 ( ) 0 0

0 , 0 0

0 0 0

T T

T

B R C R R

R R B S S

R R R

133

22 22 22

0 0

0T

TG B R BR S R

Example 31 1

1 31 31 31

21 22 2 22

31 31

0 0 0

, 0 0

0 0 0

T T

T

T

B R CR R

R R R B S S

R C R

1 31 21

21 31 1 22 22 22 21 11 21 21 31 2 2 31 21

0 T T

T T T T T T

B S RG

R S B R S R R S R R S B B S R

Numerical ExamplesCUTEr test set

Case C=0Explicit G=I Implicit G22=I

name n m fact. iter. total fact. iter. total ratio

CVXQP1 10000 5000 1.73 2 2.98 0.066 27 1.47 0.49

CVXQP2 10000 2500 0.045 2 0.15 0.032 9 0.28 1.80

CVXQP3 10000 7500 3.11 2 6.09 0.085 16 1.35 0.22

POWELL20 10000 5000 0.12 1 0.19 0.040 314 6.52 33.0

PRIMALC1 239 9 0.0047 2 0.0092 0.0023 1 0.0046 0.50

PRIMALC2 238 7 0.0041 1 0.0071 0.0023 1 0.0042 0.58

QPNBOEI1 726 351 0.028 59 0.44 0.0044 13 0.060 0.13

QPNBOEI2 305 166 0.0071 55 0.11 0.0029 17 0.031 0.29

UBH1 9009 6000 0.11 9 0.50 0.049 1 0.16 0.33

26/40 problems solved faster (20 if take into account perm time)

Case C=I

Explicit G=I Implicit G22=I

name n m fact. iter. total fact. iter. total ratio

CVXQP1 10000 5000 1.98 2 3.41 0.062 14 1.02 0.30

CVXQP2 10000 2500 0.048 2 0.16 0.030 11 0.33 2.10

CVXQP3 10000 7500 6.73 2 10.5 0.083 8 1.09 0.10

POWELL20 10000 5000 0.081 2 0.17 0.040 2 0.13 0.77

PRIMALC1 239 9 0.0045 1 0.0080 0.0024 1 0.0044 0.55

PRIMALC2 238 7 0.0037 1 0.0069 0.0021 1 0.0038 0.56

QPNBOEI1 726 351 0.029 26 0.27 0.0040 17 0.077 0.28

QPNBOEI2 305 166 0.0080 6 0.031 0.0027 14 0.032 1.03

UBH1 9009 6000 0.12 2 0.26 0.043 2 0.22 0.83

29/40 problems solved faster (25 if take into account perm time)

Conclusions• New method for constructing preconditioners for CG methods for a variety

of important problems

• Preconditioners– implicitly respect crucial structure

– easy to apply

– flexible and capable of improving eigenvalue clusters

• Extend the class of problems for which CG is applicable

• Still under development…but will be available as part of GALAHAD

Open questions

• How to pick basis B1 efficiently and so as to improve eigenvalue clusters

• How to approximate blocks of H in G

• What about Mr Greedy?

Conclusions• New method for constructing preconditioners for CG methods for a variety

of important problems

• Preconditioners– implicitly respect crucial structure

– easy to apply

– flexible and capable of improving eigenvalue clusters

• Extend the class of problems for which CG is applicable

• Still under development…but will be available as part of GALAHAD

Open questions

• How to pick basis B1 efficiently and so as to improve eigenvalue clusters

• How to approximate blocks of H in G