nick higham research matters school of mathematics the ...higham/talks/talk11...research matters...
TRANSCRIPT
-
Research Matters
February 25, 2009
Nick HighamDirector of Research
School of Mathematics
1 / 6
The f (A)b Problem
Nick HighamSchool of Mathematics
The University of Manchester
[email protected]://www.ma.man.ac.uk/~higham/
ICIAM 2011, Vancouver, July 2011.
http://www.ma.man.ac.uk/~higham/http://www.ma.man.ac.ukhttp://www.man.ac.ukmailto:[email protected]://www.ma.man.ac.uk/~higham/
-
The f (A)b Problem
Givenmatrix function f : Cn×n → Cn×n,A ∈ Cn×n, b ∈ Cn,
compute f (A)b without first computing f (A).
Most important casesf (x) = x−1
f (x) = ex .f (x) = log(x).f (x) = x1/2.f (x) = sign(x).
MIMS Nick Higham The f (A)b Problem 2 / 29
http://www.mims.manchester.ac.uk/
-
Application: Second Order ODE
d2ydt2
+ Ay = 0, y(0) = y0, y ′(0) = y ′0
has solution
y(t) = cos(√
At)y0 +(√
A)−1 sin(√At)y ′0.
But [y ′
y
]= exp
([0 −tA
t In 0
])[y ′0y0
].
MIMS Nick Higham The f (A)b Problem 3 / 29
http://www.mims.manchester.ac.uk/
-
Application: Second Order ODE
d2ydt2
+ Ay = 0, y(0) = y0, y ′(0) = y ′0
has solution
y(t) = cos(√
At)y0 +(√
A)−1 sin(√At)y ′0.
But [y ′
y
]= exp
([0 −tA
t In 0
])[y ′0y0
].
MIMS Nick Higham The f (A)b Problem 3 / 29
http://www.mims.manchester.ac.uk/
-
A1/2b: Application in StatisticsChen, Anitescu & Saad (2011): sample x ∼ N(µ,C),C ∈ Rm×m, m ∈ [1012,1015].Let x ∼ N(0, I).
If C = LLT is Cholesky factorization,y = µ+ Lx ∼ N(µ,C).
Cholesky factorization may not be computable orstorable.
y = µ+ C1/2x ∼ N(µ,C).
MIMS Nick Higham The f (A)b Problem 4 / 29
http://www.mims.manchester.ac.uk/
-
A1/2 b via Contour Integration
f (A)b =1
2πi
∫Γ
f (z)(zI − A)−1b dz.
A 5× 5 Pascal matrix: λ(A) ∈ [0.01,92.3], f (z) = z1/2.Use repeated trapezium rule to integrate around circlecentre (λmin + λmax)/2, radius λmax/2.Need 32,000 (262,000) points for 2 (13) decimal digits.Hale, H & Trefethen (2008): conformally map
Ω = C \{(−∞,0] ∪ [m,M ]
}.
to an annulus: [m,M ]→ inner circle, (−∞,0]→ outercircle. Now 5 (35) points for 2 (13) decimal digits.Further refinements: 20 points yield 14 digits.
MIMS Nick Higham The f (A)b Problem 5 / 29
http://www.mims.manchester.ac.uk/
-
Aα b via Binomial Expansion
Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .
(I − C)α =∞∑
j=0
(α
j
)(−C)j , ρ(C) < 1.
So
Aαb = sα∞∑
j=0
(α
j
)(−C)jb.
For M-matrices, required splitting with C ≥ 0 always exists.
MIMS Nick Higham The f (A)b Problem 6 / 29
http://www.mims.manchester.ac.uk/
-
Aα b via Binomial Expansion
Write A = s(I − C).If λi > 0, s = (λmin + λmax)/2 yieldsρmin = (λmax − λmin)/(λmax + λmin).For any A, s = trace(A∗A)/trace(A∗) minimizes ‖C‖F .
(I − C)α =∞∑
j=0
(α
j
)(−C)j , ρ(C) < 1.
So
Aαb = sα∞∑
j=0
(α
j
)(−C)jb.
For M-matrices, required splitting with C ≥ 0 always exists.
MIMS Nick Higham The f (A)b Problem 6 / 29
http://www.mims.manchester.ac.uk/
-
Aα b via ODE IVP
dydt
= α(A− I)[t(A− I) + I
]−1y , y(0) = bhas unique solution
y(t) =[t(A− I) + I
]αbso
y(1) = Aαb.
Used by Allen, Baglama & Boyd (2000) for α = 1/2, spd A.
MIMS Nick Higham The f (A)b Problem 7 / 29
http://www.mims.manchester.ac.uk/
-
Exponential Integrators
u ′(t) = Au(t) + g(t ,u(t)), u(0) = u0, t ≥ 0,
Solution can be written
u(t) = etAu0 +∞∑
k=1
ϕk(tA)tk uk ,
where uk = g(k−1)(t ,u(t)) |t=0 and ϕ`(z) =∑∞
k=0 zk/(k + `)!.
u(t) ≈ û(t) = etAu0 +p∑
k=1
ϕk(tA)tk uk .
Exponential time differencing (ETD) Euler (p = 1):
yn+1 = ehAyn + hϕ1(hA)g(tn, yn).
MIMS Nick Higham The f (A)b Problem 8 / 29
http://www.mims.manchester.ac.uk/
-
Exponential Integrators
u ′(t) = Au(t) + g(t ,u(t)), u(0) = u0, t ≥ 0,
Solution can be written
u(t) = etAu0 +∞∑
k=1
ϕk(tA)tk uk ,
where uk = g(k−1)(t ,u(t)) |t=0 and ϕ`(z) =∑∞
k=0 zk/(k + `)!.
u(t) ≈ û(t) = etAu0 +p∑
k=1
ϕk(tA)tk uk .
Exponential time differencing (ETD) Euler (p = 1):
yn+1 = ehAyn + hϕ1(hA)g(tn, yn).
MIMS Nick Higham The f (A)b Problem 8 / 29
http://www.mims.manchester.ac.uk/
-
Saad’s Trick (1992)
ϕ1(z) =ez − 1
z.
exp([
A b0 0
])=
[eA ϕ1(A)b0 1
]
MIMS Nick Higham The f (A)b Problem 9 / 29
http://www.mims.manchester.ac.uk/
-
Theorem (Al-Mohy & H, 2011)
Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define
B =[
A U0 J
]∈ C(n+p)×(n+p), J =
[0 Ip−10 0
]∈ Cp×p.
Then for X = eτB we have
X (1 : n,n + j) =∑j
k=1 τk ϕk(τA)uj−k+1, j = 1 : p.
Completely removes the need toevaluate ϕk functions!
-
Theorem (Al-Mohy & H, 2011)
Let A ∈ Cn×n, U = [u1,u2, . . . ,up] ∈ Cn×p, τ ∈ C, and define
B =[
A U0 J
]∈ C(n+p)×(n+p), J =
[0 Ip−10 0
]∈ Cp×p.
Then for X = eτB we have
X (1 : n,n + j) =∑j
k=1 τk ϕk(τA)uj−k+1, j = 1 : p.
Completely removes the need toevaluate ϕk functions!
-
Implementation of Exponential Integrators
We compute
û(t) = etAu0 +p∑
k=1
ϕk(tA)tk uk
as, with U = [up, . . . ,u1],
û(t) =[
In 0]
exp(
t[
A ηU0 J
])[u0
η−1ep
].
Choose η so that η‖U‖ ≈ 1.
MIMS Nick Higham The f (A)b Problem 11 / 29
http://www.mims.manchester.ac.uk/
-
Assumptions
Choice of algorithm depends on assumptions about A.
Examples:
Can Schur-factorize A.“Backslash matrix”: can solve (zI −A)x = b by (sparse)direct methods.A symmetric, can estimate [λmin, λmax], onlymatrix–vector products available.A is general, only matrix–vector products available.
MIMS Nick Higham The f (A)b Problem 12 / 29
http://www.mims.manchester.ac.uk/
-
Formulae for eA, A ∈ Cn×n
Taylor series Limit Scaling and squaring
I + A +A2
2!+
A3
3!+ · · · lim
s→∞(I + A/s)s (eA/2
s)2
s
Cauchy integral Jordan form Interpolation
12πi
∫Γ
ez(zI − A)−1 dz Zdiag(eJk )Z−1n∑
i=1
f [λ1, . . . , λi ]i−1∏j=1
(A − λj I)
Differential system Schur form Padé approximation
Y ′(t) = AY (t), Y (0) = I QeT Q∗ pkm(A)qkm(A)−1
Krylov methods: Arnoldi fact. AQk = Qk Hk + hk+1,k qk+1eTk withHessenberg H: eAb ≈ Qk eHk Q∗k b.
MIMS Nick Higham The f (A)b Problem 13 / 29
http://www.mims.manchester.ac.uk/
-
The Sixth Dubious WayMoler & Van Loan (1978, 2003)
MIMS Nick Higham The f (A)b Problem 14 / 29
http://www.mims.manchester.ac.uk/
-
Computing eAB
A︸︷︷︸n×n
, B︸︷︷︸n×n0
, n0 � n. Exploit, for integer s,
eAB = (es−1A)sB = es
−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
How to choose s and m?
MIMS Nick Higham The f (A)b Problem 15 / 29
http://www.mims.manchester.ac.uk/
-
Computing eAB
A︸︷︷︸n×n
, B︸︷︷︸n×n0
, n0 � n. Exploit, for integer s,
eAB = (es−1A)sB = es
−1Aes−1A · · · es−1A︸ ︷︷ ︸s times
B.
Choose s so Tm(s−1A) =∑m
j=0(s−1A)j
j!≈ es−1A. Then
Bi+1 = Tm(s−1A)Bi , i = 0 : s − 1, B0 = B
yields Bs ≈ eAB.
How to choose s and m?
MIMS Nick Higham The f (A)b Problem 15 / 29
http://www.mims.manchester.ac.uk/
-
Backward Error Analysis
Lemma (Al-Mohy & H, 2009)
Tm(s−1A)sB = eA+∆AB, where ∆A = shm+1(s−1A) andhm+1(x) = log(e−x Tm(x)) =
∑∞k=m+1 ck x
k . Moreover,
‖∆A‖ ≤ s∞∑
k=m+1
|ck | αp(s−1A)k
if m + 1 ≥ p(p − 1), where
αp(A) = max(dp,dp+1), dp = ‖Ap‖1/p.
MIMS Nick Higham The f (A)b Problem 16 / 29
http://www.mims.manchester.ac.uk/
-
Why Use dp = ‖Ap‖1/p?ρ(A) ≤ dp ≤ ‖A‖.
‖Ak‖ ≤ ‖Apk‖1/p ≤ dkp .
With A =[
0.9 5000 −0.5
]:
k 2 5 10‖Ak‖1 2.0e2 2.2e2 1.2e2‖A‖k1 2.5e5 3.1e13 9.8e26
dk2 =(‖A2‖1/21
)k 2.0e2 5.7e5 3.2e11dk3 =
(‖A3‖1/31
)k 4.5e1 1.3e4 1.9e8Cheaply estimate ‖Ak‖1, for a few k (H & Tisseur,2000).
MIMS Nick Higham The f (A)b Problem 17 / 29
http://www.mims.manchester.ac.uk/
-
Why Use dp = ‖Ap‖1/p? — cont.
dp = ‖Ap‖1/p provide information about thenonnormality of A.Their use helps avoid overscaling.What other uses do they have?
MIMS Nick Higham The f (A)b Problem 18 / 29
http://www.mims.manchester.ac.uk/
-
Size of Taylor Series Argument
Constant
θm :=max
{θ :
∞∑k=m+1
|ck |θk−1 ≤ �}.
Computed via symbolic, high precision:
m 10 20 30 40 55single 1.0e0 3.6e0 6.3e0 9.1e0 1.3e1
double 1.4e-1 1.4e0 3.5e0 6.0e0 9.9e0
MIMS Nick Higham The f (A)b Problem 19 / 29
http://www.mims.manchester.ac.uk/
-
Choice of s and m
αp(A) :=max(dp,dp+1
)‖∆A‖‖A‖ ≤ � if m + 1 ≥ p(p − 1) and s
−1αp(A) ≤ θm.
Computational cost for Bs ≈ eAB is
Cm(A) = m max(dαp(A)/θme,1
)matrix products.
Cost decreases with m.Restrict 2 ≤ p ≤ pmax, p(p − 1)− 1 ≤ m ≤ mmax.Minimize cost over p, m.
MIMS Nick Higham The f (A)b Problem 20 / 29
http://www.mims.manchester.ac.uk/
-
Preprocessing
Expand the Taylor series about µ ∈ C:
eµ∞∑
k=0
(A− µI)k/k !
Choose µ so ‖A− µI‖ ≤ ‖A‖.Alg is based on 1-norm, but minimizing ‖A− µI‖F doesbetter empirically at minimizing dp(A− µI).Recover eA from
eµ[Tm(s−1(A− µI))
]s,
[eµ/sTm(s−1(A− µI))
]s.
First expression prone to overflow, so prefer second.Balancing is an option.
MIMS Nick Higham The f (A)b Problem 21 / 29
http://www.mims.manchester.ac.uk/
-
Termination Criterion
In evaluating
Tm(s−1A)Bi =m∑
j=0
(s−1A)j
j!Bi
we accept Tk(A)Bi for the first k such that
‖Ak−1Bi‖(k − 1)! +
‖AkBi‖k !
≤ �‖Tk(A)Bi‖.
MIMS Nick Higham The f (A)b Problem 22 / 29
http://www.mims.manchester.ac.uk/
-
Algorithm for F = etAB
1 µ = trace(A)/n2 A = A− µI3 [m∗, s] = parameters(tA) % Includes norm estimation.4 F = B, η = etµ/s
5 for i = 1: s6 c1 = ‖B‖∞7 for j = 1:m∗8 B = t AB/(sj), c2 = ‖B‖∞9 F = F + B
10 if c1 + c2 ≤ tol‖F‖∞, quit, end11 c1 = c212 end13 F = ηF , B = F14 end
MIMS Nick Higham The f (A)b Problem 23 / 29
http://www.mims.manchester.ac.uk/
-
George Forsythe “Pitfalls” (1970)
MIMS Nick Higham The f (A)b Problem 24 / 29
http://www.mims.manchester.ac.uk/
-
Conditioning of eAB
Relative forward error due to roundoff bounded by
ue‖A‖2‖B‖2/‖eAB‖F .
A normal implies κexp(A) = ‖A‖2. Then instability ife‖A‖2 � ‖eA‖2.A Hermitian implies spectrum of A− n−1trace(A)Ihas λmax = −λmin ⇒ (normwise) stability!
MIMS Nick Higham The f (A)b Problem 25 / 29
http://www.mims.manchester.ac.uk/
-
Comparison with the Sixth Dubious Way
Advantages of our method over the one-step ODEintegrator:
Fully exploits the linearity of the ODE.Backward error based; ODE integrator controls local(forward) errors.Overscaling avoided.
MIMS Nick Higham The f (A)b Problem 26 / 29
http://www.mims.manchester.ac.uk/
-
Experiment 1
Trefethen, Weideman & Schmelzer (2006):A ∈ R9801×9801, 2D Laplacian,-2500*gallery(’poisson’,99).
Compute eαAb, tol = ud .
α = 0.02 α = 1speed mv diff speed mv diff
Alg AH 1 1010 1 47702expv 2.8 403 7.7e-15 1.3 8835 4.2e-15phipm 1.1 172 3.1e-15 0.2 2504 4.0e-15rational 3.8 7 3.3e-14 0.1 7 1.2e-12
MIMS Nick Higham The f (A)b Problem 27 / 29
http://www.mims.manchester.ac.uk/
-
Experiment 2A = -gallery(’triw’,20,4.1), bi = cos i , tol = ud .
0 20 40 60 80 100
10−10
10−5
100
105
t
‖etAb‖2
Alg AH
phipm
rational
expv
MIMS Nick Higham The f (A)b Problem 28 / 29
http://www.mims.manchester.ac.uk/
-
Conclusions
f (A)b problem of growing interest.Many possible approaches (including Krylov—notdiscussed here).Method design/choice must take into account
assumptions about A,desired accuracy.
New Taylor series-based alg for eAB very competitivewith existing methods. Well suited for “black box” code.
Al-Mohy & H. Computing the action of the matrixexponential, with an application to exponentialintegrators. SISC, 2011.
MIMS Nick Higham The f (A)b Problem 29 / 29
http://www.mims.manchester.ac.uk/
-
References I
A. H. Al-Mohy and N. J. Higham.Computing the action of the matrix exponential, with anapplication to exponential integrators.SIAM J. Sci. Comput., 33(2):488–511, 2011.
E. J. Allen, J. Baglama, and S. K. Boyd.Numerical approximation of the product of the squareroot of a matrix with a vector.Linear Algebra Appl., 310:167–181, 2000.
J. Chen, M. Anitescu, and Y. Saad.Computing f (A)b via least squares polynomialapproximations.SIAM J. Sci. Comput., 33(1):195–222, 2011.
MIMS Nick Higham The f (A)b Problem 25 / 29
http://www.mims.manchester.ac.uk/
-
References II
P. I. Davies and N. J. Higham.Computing f (A)b for matrix functions f .In A. Boriçi, A. Frommer, B. Joó, A. Kennedy, andB. Pendleton, editors, QCD and Numerical Analysis III,volume 47 of Lecture Notes in Computational Scienceand Engineering, pages 15–24. Springer-Verlag, Berlin,2005.
G. E. Forsythe.Pitfalls in computation, or why a math book isn’tenough.Amer. Math. Monthly, 77(9):931–956, 1970.
MIMS Nick Higham The f (A)b Problem 26 / 29
http://www.mims.manchester.ac.uk/
-
References III
N. Hale, N. J. Higham, and L. N. Trefethen.Computing Aα, log(A) and related matrix functions bycontour integrals.SIAM J. Numer. Anal., 46(5):2505–2523, 2008.
N. J. Higham.Functions of Matrices: Theory and Computation.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, 2008.ISBN 978-0-898716-46-7.xx+425 pp.
N. J. Higham and A. H. Al-Mohy.Computing matrix functions.Acta Numerica, 19:159–208, 2010.
MIMS Nick Higham The f (A)b Problem 27 / 29
http://www.mims.manchester.ac.uk/
-
References IV
N. J. Higham and F. Tisseur.A block algorithm for matrix 1-norm estimation, with anapplication to 1-norm pseudospectra.SIAM J. Matrix Anal. Appl., 21(4):1185–1201, 2000.
C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix.SIAM Rev., 20(4):801–836, 1978.
C. B. Moler and C. F. Van Loan.Nineteen dubious ways to compute the exponential of amatrix, twenty-five years later.SIAM Rev., 45(1):3–49, 2003.
MIMS Nick Higham The f (A)b Problem 28 / 29
http://www.mims.manchester.ac.uk/
-
References V
Y. Saad.Analysis of some Krylov subspace approximations tothe matrix exponential operator.SIAM J. Numer. Anal., 29(1):209–228, Feb. 1992.
L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer.Talbot quadratures and rational approximations.BIT, 46(3):653–670, 2006.
MIMS Nick Higham The f (A)b Problem 29 / 29
http://www.mims.manchester.ac.uk/