preconditioned inverse iteration for hierarchical matrices

Hierarchical (H-)Matrices PINVIT Numerical Results

Preconditioned Inverse Iteration for H-Matrices

Peter Benner and Thomas Mach

Mathematics in Industry and TechnologyDepartment of Mathematics

TU Chemnitz

23rd Chemnitz FEM Symposium 2010

1/31 Thomas Mach Preconditioned Inverse Iteration for H-Matrices

FÜR DYNAMIK KOMPLEXERTECHNISCHER SYSTEME

MAGDEBURG

MAX−PLANCK−INSTITUT


Preconditioned Inverse Iteration for H-Matrices

Peter Benner and Thomas Mach

Computational Methods in Systems and Control TheoryMax Planck Institute for Dynamics of Complex Technical Systems

Magdeburg

23rd Chemnitz FEM Symposium 2010



Outline

1 Hierarchical (H-)Matrices

2 PINVIT

3 Numerical Results



H-Matrices [Hackbusch 1998]

Some dense matrices, e.g. BEM or FEM, can be approximated byH-matrices in a data-sparse manner.

hierarchical tree TI block H-tree TI× I

I = {1, 2, 3, 4, 5, 6, 7, 8}

{1, 2, 3, 4} {5, 6, 7, 8}

{1, 2} {3, 4} {5, 6} {7, 8}

{1}{2}{3}{4}{5}{6}{7}{8}

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 56 6 6 67 7 7 78 8 8 8

1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 56 6 6 67 7 7 78 8 8 8

dense matrices, rank-k-matrices

rank-k-matrix: Ms×t = ABT , A ∈ Rn×k ,B ∈ Rm×k (k � n,m)



H-Matrices [Hackbusch 1998]

Hierarchical matrices

H(TI× I, k) ={

M ∈ RI× I∣∣ rank (Ma×b) ≤ k ∀a× b admissible

}22

3 3

7 10

3 7

3 10

19 10

10 31

14 8

11 11

14 118 11

19 10

10 31 11

11 31

11 9

9 16 12

11 1611 8

9 16 11

11 16

619 73 3

8

11 11

9 3

7 3

11

8 11

25 10

10 19 11

11 31

8 5

11 8

8 15 8 12

116 515 6

13

13

7 5

13 8

8 11 8 11

116 5

15 6

8

11 8

8 15

5 8 11

126 15

5 6 13

137

13 8

8 11

5 8 11

116 15

5 6

6110 103

6 14

10 3 6

10 14

25 10 6

10

6

19 10

10 31

15 9

9 11 10

10 1615 9

8 11 10

10 16

5110 107 9 7

3 3

10 7

109 3

7 3

25 11

1125 10

10 19

11 811 8

8 15 9

8 15 12 13

1310 7

13 8

8 11 9

9 15 1119

2010 7

13 8

9

11 8

8 15 11 12

139 7

13 8

9

13 8

8 1111

1011 8

8 15 8

9 15

7 12 13

1310

13 8

8 11 8

9 15

7 1120

199

13 9

8

11 8

8 15

7 11 12

129

13 9

813 8

8 11

7 11

39 10

10 25

3 7

3 10 107 10 6

3 3

7 10 7

1010

6

22

3 3

7 10

3 7

3 10

19 10

10 31

15 10

11

11 9

9 16

15 11

1011 8

9 16

34 10

10 25

13 10

7 1113 7

10 11 61

6 513 6 11

12

8 5

11 8

8 15 812

136 5

13 6 11

11 236 13

5 6 12

118

11 8

8 15

5 813

12

6 13

5 6 10

11 23

20 9

9 39

9 7

103 7

3 10

9 107

3 3

7 10 61

15 10

1015 9

9 11

15 10

1015 9

8 11

20 9

9 34

9 7

10 13

9 107 13 51

adaptive rank k(ε)

storage NSt,H(T , k) = O(n log n k(ε))

complexity of approximate arithmetic

MHv O(n log n k(ε))+H,−H O(n log n k(ε)2)

∗H,HLU(·), (·)−1H O(n (log n)2 k(ε)2)



Goal: Eigenvalues of symmetric H-Matrices

M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λi in O(n (log n)α kβ)?

X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

X



Preconditioned Inverse Iteration[Knyazev, Neymeyr, et al.]

Definition

The function

µ(x) = µ(x ,M) =xTMx

xT x

is called the Rayleigh quotient.

Minimize the Rayleigh quotient by a gradient method:

xi+1 := xi − α∇µ(xi ), ∇µ(x) =2

xT x(Mx − xµ(x)) ,

+ preconditioning ⇒ update equation:

xi+1 := xi − B−1 (Mxi − xiµ(xi )) .

Residual r(x) = Mx − xµ(x).

Preconditioned residual B−1r(x) = B−1 (Mx − xµ(x)).



Preconditioned Inverse Iteration[Knyazev, Neymeyr 2009]

xi+1 := xi − B−1 (Mxi − xiµ(xi ))

If

M symmetric positive definite and

B−1 approximates the inverse of M, so that∥∥I − B−1M∥∥

M≤ c < 1,

then Preconditioned INVerse ITeration (PINVIT) converges.



Preconditioned Inverse Iteration[Knyazev, Neymeyr, et al.]

The residual

ri = Mxi − xiµ(xi )

converges to 0, so that

‖ri‖2 < ε

is a useful termination criterion.



Why is it called Preconditioned Inverse Iteration?

Inverse Iteration:

xi+1 = µ(xi )A−1xi ,

xi+1 = xi − A−1Axi︸︷︷︸=0

+µ(xi )A−1xi ,

xi+1 = xi − A−1 (Axi + µ(xi )xi ) .

Replace A−1 with the inexact solution by the preconditioner B:

xi+1 = xi − B−1 (Axi + µ(xi )xi ) .



Variants of PINVIT[Neymeyr 2001: A Hierarchy of Preconditioned Eigensolvers for Elliptic Diff. Operators]

Classification by Neymeyr:

PINVIT(1): xi+1 := xi − B−1ri .

PINVIT(2): xi+1 := arg minv∈span{xi ,B−1ri} µ(v).

PINVIT(3): xi+1 := arg minv∈span{xi−1,xi ,B−1ri} µ(v).

PINVIT(n): Analogously.

PINVIT(·,d): Replacing x by a rectangular full rank matrixX ∈ Rn×d one gets the subspace version of PINVIT(·).



Algorithm

H-PINVIT(1,d)

Input: M ∈ Rn×n, X0 ∈ Rn×d (e.g. randomly chosen)Output: Xp ∈ Rn×d , µ ∈ Rd×d , with ‖MXp − Xpµ‖ ≤ εB−1 = (M)−1

HOrthogonalize X0

R := MX0 − X0µ, µ = XT0 MX0

for (i := 1;‖R‖F > ε; i + +) doXi := Xi−1 − B−1ROrthogonalize Xi

R := MXi − Xiµ, µ = XTi MXi

end



Complexity

The number of iterations is independent of matrix size n.

Once:

adaptive H-matrix inversion: O(n (log n)2 k (c)2).

Per iteration:

H-matrix-vector products with M and B−1: O(n (log n) k (c))and O(n (log n) k (c)2), and

some dense arithmetic handling vectors of length n.

The complexity of the algorithm is determined by the H-matrixinversion: ⇒ O(n (log n)2 k (c)2).

Competitive to MATLAB ® eigs.

Expensive.




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)?

X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λn in O(n (log n)α kβ)? X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0


X



Folded Spectrum Method

How to find λi in O(n (log n)α kβ)?

If i = n − d with d < O(log n),

use subspace version PINVIT(·,d).

0λn λn−1λn−2. . .

λ1

But (M − σI) is not positive definite!





If i = n − d with d � log n?

shift with σ near λi .

0λn. . .

λi+1 λi λi−1. . .

λ1






If i = n − d with d � log n,

shift with σ near λi .

0λn. . .

λi+1 λi λi−1. . .

λ1σ





Folded Spectrum Method [Wang, Zunger 1994]

Mσ = (M − σI)2

Mσ is s.p.d., if M is s.p.d. and σ 6= λi .

Assume all eigenvalues of Mσ are simple.

Mv = λv ⇔ Mσv = (M − σI)2 v

= M2v − 2σMv + σ2v

= λ2v − 2σλv + σ2v

= (λ− σ)2v




1 Choose σ.2 Compute

a) Mσ = (M − σI)2 andb) M−1

σ .

3 Use PINVIT to find the smallest eigenpair (µσ, v) of Mσ.4 Compute µ = vTMv/vTv .

(µ, v) is the nearest eigenpair to σ.

H-arithmetic enables us to shift, square and invert M resp.Mσ in linear-polylogarithmic complexity.




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0


X




M = MT ∈ H(T , k),

Λ(M) = {λ1, λ2, . . . , λn},λ1 ≥ λ2 ≥ · · · ≥ λn > 0

⇓How to find λi in O(n (log n)α kβ)? X



Numerical Results

Numerical Results



Hlib

Hlib [Borm, Grasedyck, et al.]

We use the Hlib1.3 (www.hlib.org) for theH-arithmetic operations and some examples out ofthe library for testing the eigenvalue algorithm.


www.hlib.org


2D Laplace, smallest eigenvalues

2D Laplace over [−1, 1]× [−1, 1]

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 4.3466E-08 0.006

FEM16 256 1.2389E-07 0.014 2.33 80.00

FEM32 1 024 4.4308E-07 0.036 2.57 21.67

FEM64 4 096 1.8003E-06 0.228 6.33 6.72

FEM128 16 384 6.7069E-06 0.992 4.35 4.67

FEM256 65 536 1.6631E-05 2.397 2.42 4.57

FEM512 262 144 4.4015E-05 8.120 3.39 5.79

d = 4, c = 0.2, εeig = 10−4, N(ni ) = ni (log2 ni ) CspCid

Time t only PINVIT (without H-inversion)




FEM8

FEM16

FEM32

FEM64

FEM12

8

FEM25

6

FEM51

210−3

10−2

10−1

100

101

102

103

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)

O(N(ni ))H-inversion

O(N(ni ) log ni )




FEM8

FEM16

FEM32

FEM64

FEM12

8

FEM25

6

FEM51

210−3

10−2

10−1

100

101

102

103

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)


O(N(ni ) log ni )eigs



Complexity

The number of iterations is independent of matrix size n.

Once:

adaptive H-matrix inversion: O(n (log n)2 k (c)2).

Per iteration:

H-matrix-vector products with M and B−1: O(n (log n) k (c))and O(n (log n) k (c)2), and

some dense arithmetic handling vectors of length n.

The complexity of the algorithm is determined by the H-matrixinversion: ⇒ O(n (log n)2 k (c)2).

Competitive to MATLAB eigs.

Expensive.



2D Laplace, vn, FEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



2D Laplace, vn−1, FEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2




−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



2D Laplace, inner eigenvalues

2D Laplace over [−1, 1]× [−1, 1]

Name ni error titi

ti−1

N(ni )N(ni−1)

FEM8 64 2.2560E-08 <0.01

FEM16 256 3.4681E-06 0.01 — 80.00

FEM32 1 024 1.1875E-04 0.04 4.00 21.67

FEM64 4 096 7.2115E-04 0.20 5.00 6.72

FEM128 16 384 2.0747E-05 0.93 4.65 4.67

FEM256 65 536 9.2370E-05 3.59 3.86 4.57





2D Laplace, L-shape, smallest eigenvalues

2D Laplace over [−1, 1]× [−1, 1] \ [0, 1]× [0, 1]

Name n error t titi−1

N(ni )N(ni−1)

LFEM8 48 6.0999E-07 0.023

LFEM16 192 5.7426E-07 0.037 1.57 10.86

LFEM32 768 1.4644E-06 0.180 4.91 10.11

LFEM64 3 072 4.4753E-06 0.497 2.76 108.78

LFEM128 12 288 1.8770E-04 0.770 1.55 9.49

LFEM256 49 152 — 2.133 2.77 4.59






LFEM8

LFEM16

LFEM32

LFEM64

LFEM12

8

LFEM25

610−3

10−2

10−1

100

101

102

103

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)


O(N(ni ) log ni )




LFEM8

LFEM16

LFEM32

LFEM64

LFEM12

8

LFEM25

610−3

10−2

10−1

100

101

102

103

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)





2D Laplace, L-shape, vn, LFEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



2D Laplace, L-shape, vn−1, LFEM64

−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2




−1 −0.5 00.5 1−1

0

1−5

0

5

·10−2

−6

−4

−2

0

2

4

6·10−2



BEM example, smallest eigenvalues

dense matrix, but approximable by an H-matrix ⇒ MATLAB eigsis expensive

Name ni error titi

ti−1

N(ni )N(ni−1)

BEM8 258 2.0454E-05 0.01

BEM16 1 026 3.5696E-05 0.03 3.00 26.81

BEM32 4 098 7.1833E-05 0.14 4.67 14.42

BEM64 16 386 — 0.46 3.29 16.51

BEM128 65 538 — 2.00 4.36 26.35






BEM8

BEM16

BEM32

BEM64

BEM12

8

10−2

100

102

104

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)


O(N(ni ) log ni )




BEM8

BEM16

BEM32

BEM64

BEM12

8

10−2

100

102

104

CP

Uti

me

ins

PINVIT(1,4)

PINVIT(3,4)





Conclusions

Once we have inverted M ∈ H(T , k) finding the eigenvaluesby PINVIT is cheap and storage efficient.

For dense, data-sparse matrices, H-PINVIT can solveproblems where eigs (implicitly, restarted Arnoldi/Lanczositeration) is not applicable.

The folded spectrum method enables us to compute innereigenvalues, too.

We will investigate the use of H-Cholesky factorizationsinstead of the H-inversion.

Thank you for your attention.


Appendix

Adaptive H-Inversion [Grasedyck 2001]

Adaptive H-Inversion

Input: M ∈ H (TI×I ), c = c√‖M‖2

∈ R

Output: M−1H , with

∥∥I −M−1H M

∥∥2< c

Compute M−1H with εlocal = c/(Csp(M) ‖M‖H2 )

δ0M :=∥∥I −M−1

H M∥∥H

2/c

while δiM > 1 doCompute M−1

H with εlocal = εlocal/δiM

δiM :=∥∥I −M−1

H M∥∥H

2/c

end

∥∥I −M−1H M

∥∥2< c =

c√‖M‖2

⇔∥∥I −M−1

H M∥∥

M≤ c


preconditioned inverse iteration for hierarchical matrices

Technology

hmatrices peter benner

inverse iteration knyazev

matrices rank

dense matrices

hmatricespeter benner

preconditioned residual

function x t mx x

hierarchical matriceshti