Texts in Applied Mathematics 12

Springer Science+Business Media, LLC

Editors J.E. Marsden

L. Sirovich M. Golubitsky

S.S. Antman

Advisors G.Iooss

P. Holmes D. Barkley

M. Dellnitz P. Newton

Texts in Applied Mathematics

1. Sirovich: Introduction to Applied Mathematics.

2. Wiggins: Introduction to Applied Nonlinear Dynamical Systems and Chaos. 3. HalelKorak: Dynamics and Bifurcations. 4. ChorinlMarsden: A Mathematical Introduction to Fluid Mechanics, 3rd ed.

5. HubbardlWest: Differential Equations: A Dynamical Systems Approach: Ordinary Differential Equations.

6. Sontag: Mathematical Control Theory: Deterministic Finite Dimensional Systems, 2nd ed.

7. Perko: Differential Equations and Dynamical Systems, 3rd ed. 8. Seaborn: Hypergeometric Functions and Their Applications. 9. Pipkin: A Course on Integral Equations. 10. HoppensteadtlPeskin: Modeling and Simulation in Medicine and the Life Sciences,

2nd ed. 11. Braun: Differential Equations and Their Applications, 4th ed.

12. StoerlBulirsch: Introduction to Numerical Analysis, 3rd ed. 13. RenardylRogers: An Introduction to Partial Differential Equations. 14. Banks: Growth and Diffusion Phenomena: Mathematical Frameworks and

Applications. 15. Brenner/Scott: The Mathematical Theory of Finite Element Methods, 2nd ed.

16. Van de Velde: Concurrent Scientific Computing. 17. MarsdenlRatiu: Introduction to Mechanics and Symmetry, 2nd ed. 18. Hubbard/West: Differential Equations: A Dynamical Systems Approach: Higher-

Dimensional Systems. 19. KaplanlGlass: Understanding Nonlinear Dynamics. 20. Holmes: Introduction to Perturbation Methods. 21. Curtain/Zwart: An Introduction to Infinite-Dimensional Linear Systems Theory.

22. Thomas: Numerical Partial Differential Equations: Finite Difference Methods. 23. Taylor: Partial Differential Equations: Basic Theory. 24. Merkin: Introduction to the Theory of Stability of Motion. 25. Naber: Topology, Geometry, and Gauge Fields: Foundations. 26. Polderman/Willems: Introduction to Mathematical Systems Theory: A Behavioral

Approach. 27. Reddy: Introductory Functional Analysis with Applications to Boundary-Value

Problems and Finite Elements. 28. Gustafson/Wilcox: Analytical and Computational Methods of Advanced

Engineering Mathematics. 29. Tveito/Winther: Introduction to Partial Differential Equations: A Computational

Approach. 30. GasquetlWitomski: Fourier Analysis and Applications: Filtering, Numerical

Computation, Wavelets.

(continued after index)

]. Stoer R. Bulirsch

Introduction to Numerical Analysis

Third Edition

Translated by R. Bartels, W. Gautschi, and C. Witzgall

With 39 Illustrations

Springer

J. Stoer Institut fUr Angewandte Mathematik Universtiit Wurzburg AM Hubland D-97074 Wurzburg Germany

R. Bartels W. Gautschi

R. Bulirsch Institut fUr Mathematik Technische Universitiit 8000 Munchen Germany

C. Witzgall Department of Computer Department of Computer Center for Applied

Mathematics National Bureau of

Standards Washington, DC 20234 USA

Science University of Waterloo Waterloo, Ontario N2L 3Gl Canada

Sciences Purdue University West Lafayette, IN 47907 USA

Series Editors J.E. Marsden Control and Dynamical Systems, 107-81 California Institute of Technology Pasadena, CA 91125 USA

M. Golubitsky Department of Mathematics University of Houston Houston, TX 77204-3476 USA

L. Sirovich Division of Applied Mathematics Brown University Providence, RI 02912 USA

S.S. Antman Department of Mathematics and Institute for Physical Science

and Technology University of Maryland College Park, MD 20742-4015 USA

Mathematics Subject Classification (2000): 44-01, 44AIO, 44A15, 65R1O

Library of Congress Cataloging.in.Publication Data Stoer, Josef.

[EinfOhrung in die numerische Mathematik. English] Introduction to numerica! analysis / J. Stoer, R. Bulirsch. - 3rd ed.

p. cm. - (Texts in applied mathematics ; 12) Includes bibliographical references and index. ISBN 978-1-4419-3006-4 ISBN 978-0-387-21738-3 (eBook) DOI 10.1007/978-0-387-21738-3 1. Numerica! analysis. I. Bulirsch, Roland, II. Title. III. Series.

QA297 .S8213 2002 519.4-<1c21 2002019729

Printed on acid-free paper. © 2002, 1980, 1993 Springer Science+Business Media New York Originally published by Springer-Verlag New York, Inc. in 2002 Softcover reprint ofthe hardcover 3rd edition 2002 AII rights reserved. This work may nat be translated or copied in whole ar in part without the written pennission of the publisher Springer Science+Business Media, LLC, except for brief excerpts in connection with reviews or scholarly analysis. Vse in connection with any fonn of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use of general descriptive names, tracle names, trademarks, etc., in this publication. even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.

Production managed by Timothy Taylor; manufactu!:!.ns. supernsed by Jacqui Ashri. Photocomposed copy prepared from the author's U:l~ files using Springer's svsing.sty macro.

9 8 7 6 5 4 3 2 1

ISBN 978-1-4419-3006-4 SPIN 10867658

Series Preface

Mathematics is playing an ever more important role in the physical and biological sciences, provoking a blurring of boundaries between scientific disciplines and a resurgence of interest in the modern as well as the classical techniques of applied mathematics. This renewal of interest, both in re­search and teaching, has led to the establishment of the series Texts in Applied Mathematics (TAM).

The development of new courses is a natural consequence of a high level of excitement on the research frontier as newer techniques, such as numeri­cal and symbolic computer systems, dynamical systems, and chaos, mix with and reinforce the traditional methods of applied mathematics. Thus, the purpose of this textbook series is to meet the current and future needs of these advances and to encourage the teaching of new courses.

TAM will publish textbooks suitable for use in advanced undergraduate and beginning graduate courses, and will complement the Applied Mathe­matical Sciences (AMS) series, which will focus on advanced textbooks and research-level monographs.

Pasadena, California Providence, Rhode Island Houston, Texas College Park, Maryland

J .E. Marsden L. Sirovich M. Golubitsky S.S. Antman

Preface to the Third Edition

A new edition of a text presents not only an opportunity for corrections and minor changes but also for adding new material. Thus we strived to improve the presentation of Hermite interpolation and B-splines in Chap­ter 2, and we added a new Section 2.4.6 on multi-resolution methods and B-splines, using, in particular, low order B-splines for purposes of illustra­tion. The intent is to draw attention to the role of B-splines in this area, and to familiarize the reader with, at least, the principles of multi-resolution methods, which are fundamental to modern applications iI:. signal- and im­age processing.

The chapter on differential equations was enlarged, too A new Section 7.2.18 describes solving differential equations in the presence of disconti­nuities whose locations are not known at the outset. Such discontinuities occur, for instance, in optimal control problems where the character of a differential equation is affected by control function changes in response to switching events.

Many applications, such as parameter identification, lead to differential equations which depend on additional parameters. Users then would like to know how sensitively the solution reacts to small changes in these pa­rameters. Techniques for such sensitivity analyses are the subject of the new Section 7.2.19.

Multiple shooting methods are among the most powerful for solving boundary value problems for ordinary differential equatiom:. We dedicated, therefore, a new Section 7.3.8 to new advanced techniques in multiple shoot­ing, which especially enhance the efficiency of these methods when applied to solve boundary value problems with discontinuities, which are typical for optimal contol problems.

Among the many iterative methods for solving large sparse linear equa­tions, Krylov space methods keep growing in importance. We therefore treated these methods in Section 8.7 more systematically by adding new subsections dealing with the GMRES method (Section 8.7.2), the biorthog­onalization method of Lanczos and the (principles of the) QMR method (Section 8.7.3), and the Bi-CG and Bi-CGSTAB algorithmo: (Section 8.7.4). Correspondingly, the final Section 8.10 on the comparison of iterative meth­ods was updated in order to incorporate the findings for all Krylov space methods described before.

The authors are greatly indebted to the many who have contributed

Vll

viii Preface to the Third Edition

to the new edition. We thank R. Grigorieff for many critical remarks on earlier editions, M. v. Golitschek for his recommendations concerning B­splines and their application in multi-resolution methods, and Ch. Pflaum for his comments on the chapter dealing with the iterative solution of linear equations. T. Kronseder and R. Callies helped substantially to establish the new sections 7.2.18, 7.2.19, and 7.3.8. Suggestions by Ch. Witzgall, who had helped translate a previous edition, were highly appreciated and went beyond issues of language. Our co-workers M. Preiss and M. Wenzel helped us read and correct the original german version. In particular, we appreciate the excellent work done by J. Launer and Mrs. W. Wrschka who were in charge of transcribing the full text of the new edition in TEX.

Finally we thank the Springer-Verlag for the smooth cooperation and expertise that lead to a quick realization of the new edition.

Wiirzburg, Miinchen January 2002

J. Stoer R. Bulirsch

Preface to the Second Edition

On the occasion of the new edition, the text was enlarged by several new sections. Two sections on B-splines and their computation were added to the chapter on spline functions: due to their special properties, their flex­ibility, and the availability of well tested programs for their computation, B-splines play an important role in many applications.

Also, the authors followed suggestions by many readers to supplement the chapter on elimination methods by a section dealing with the solution of large sparse systems of linear equations. Even though such systems are usually solved by iterative methods, the realm of elimination methods has been widely extended due to powerful techniques for handling sparse matrices. We will explain some of these techniques in connection with the Cholesky algorithm for solving positive definite linear systems.

The chapter on eigenvalue problems was enlarged by a section on the Lanczos algorithm; the sections on the LR- and QR algorit.hm were rewrit­ten and now contain also a description of implicit shift teehniques.

In order to take aeeount of the progress in the area of ordinary dif­ferential equations to some extent, a new section on implieit differential equations and differential-algebraic systems was added, and the seetion on stiff differential equations was updated by describing further methods to solve such equations.

Also the last chapter on the iterative solution of linear equations was improved. The modern view of the eonjugate gradient algorithm as an iterative method was stressed by adding an analysis of its convergence rate and a description of some preconditioning techniques. Finally, a new section on multigrid methods was incorporated: It contains a description of their basic ideas in the context of a simple boundary value problem for ordinary differential equations.

ix

x Preface to the Second Edition

Many of the changes were suggested by several colleagues and readers. In particular, we would like to thank R. Seydel, P. Rentrop and A. Neumaier for detailed proposals, and our translators R. Bartels, W. Gautschi and C. Witzgall for their valuable work and critical commentaries. The original German version was handled by F. Jarre, and 1. Brugger was responsible for the expert typing of the many versions of the manuscript.

Finally we thank the Springer-Verlag for the encouragement, patience and close cooperation leading to this new edition.

Wiirzburg, Miinchen May 1991

J. Stoer R. Bulirsch

Contents

Preface to the Third Edition

Preface to the Second Edition

VII

IX

1 Error Analysis 1

1.1 Representation of Numbers 2 1.2 Roundoff Errors and Floating-Point Arithmetic 4 1.3 Error Propagation 9 1.4 Examples 21 1.5 Interval Arithmetic; Statistical Roundoff Estimation 27

Exercises for Chapter 1 33 References for Chapter 1 36

2 Interpolation

2.1 Interpolation by Polynomials 38

37

2.1.1 Theoretical Foundation: The Interpolation Formula of Lagrange 38 2.1.2 Neville's Algorithm 40 2.1.3 Newtons Interpolation Formula: Divided Differences 43 2.1.4 The Error in Polynomial Interpolation 48 2.1.5 Hermite Interpolation 51 2.2 Interpolation by Rational Functions 59 2.2.1 2.2.2 2.2.3 2.2.4 2.3 2.3.1 2.3.2 2.3.3

General Properties of Rational Interpolation 59 Inverse and Reciprocal Differences. Thiele's Continued Fraction Algorithms of the Neville Type 68 Comparing Rational and Polynomial Interpolation 73

Trigonometric Interpolation 74 Basic Facts 74 Fast Fourier Transforms 80 The Algorithms of Goertzel and Reinsch 88

2.3.4 The Calculation of Fourier Coefficients. Attenuation Factors 92

64

xi

XII Contents

2.4 Interpolation by Spline Functions 97 2.4.1 Theoretical Foundations 97

2.4.2 Determining Interpolating Cubic Spline Functions 101 2.4.3 Convergence Properties of Cubic Spline Functions 107 2.4.4 B-Splines 111

2.4.5 The Computation of B-Splines 117 2.4.6 lVIulti-Resolution Methods and B-Splines 121

Exercises for Chpater 2 134 References for Chapter2 143

3 Topics in Integration 145

3.1 The Integration Formulas of Newton and Cotes 146 3.2 Peano's Error Representation 151

3.3 The Euler-Maclaurin Summation Formula 156

3.4 3.5

Integration by Extrapolation About Extrapolation Methods

160

165 3.6 Gaussian Integration Methods 171 3.7 Integrals with Singularities 181

Exercises for Chapter 3 184

References for Chapter;{ 188

4 Systems of Linear Equations 190

4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 190 4.2 The Gauss-Jordan Algorithm 200

4.3 The Choleski Decompostion 204

4.4 Error Bounds 207 4.5 Roundoff-Error Analysis for Gaussian Elimination 215 4.6 Roundoff Errors in Solving Triangular Systems 221 4.7 Orthogonalization Techniques of Householder and Gram-Schmidt 223

4.8 Data Fitting 231 4.8.1 Linear Least Squares. The Normal Equations 232 4.8.2 The Use of Orthogonalization in Solving Linear Least-Squares

Problems 235 4.8.3 The Condition of the Linear Least-Squares Problem 236 4.8.4 Nonlinear Least-Squares Problems 241 4.8.5 The Pseudoinverse of a Matrix 243 4.9 Modification Techniques for Matrix Decompositions 247 4.10 The Simplex Method 256 4.11 Phase One of the Simplex Method 268 4.12 Appendix: Elimination Methods for Sparse Matrices 272

Exercises for Chapter 4 280 References for Chapter 1 286

Contents xiii

5 Finding Zeros and Minimum Points by Iterative Methods 289

5.1 The Development of Iterative Methods 290 5.2 General Convergence Theorems 293 5.3 The Convergence of Newton's Method in Several Variables 298 5.4 A Modified Newton Method 302 5.4.1 On the Convergence of Minimization Methods 303 5.4.2 Application of the Convergence Criteria to the Modified

Newton Method 308 5.4.3 Suggestions for a Practical Implementation of the Modified

Newton Method. A Rank-One Method Due to Broyden 313 5.5 Roots of Polynomials. Application of Newton's Method 316 5.6 Sturm Sequences and Bisection Methods 328 5.7 Bairstow's Method 333 5.8 The Sensitivity of Polynomial Roots 335 5.9 Interpolation Methods for Determining Roots 338 5.10 The 6,2-Method of Aitken 344

5.11 Minimization Problems without Constraints 349 Exercises for Chapter 5 358 References for Chapter 5 361

6 Eigenvalue Problems 364

6.0 Introduction 364 6.1 Basic Facts on Eigenvalues 366 6.2 The Jordan Normal Form of a Matrix 369 6.3 The Frobenius Normal Form of a Matrix 375 6.4 The Schur Normal Form of a Matrix; Hermitian and

Normal Matrices; Singular Values of Matrixes 379 6.5 Reduction of Matrices to Simpler Form 386 6.5.1 Reduction of a Hermitian Matrix to Tridiagonal Form:

The Method of Householder 388 6.5.2 Reduction of a Hermitian Matrix to Tridiagonal or Diagonal

Form: The Methods of Givens and Jacobi 394 6.5.3 Reduction of a Hermitian Matrix to Tridiagonal Form:

The Method of Lanczos 398 6.5.4 Reduction to Hessenberg Form 402 6.6 Methods for Determining the Eigenvalues and Eigenvectors 405 6.6.1 Computation of the Eigenvalues of a Hermitian

Tridiagonal Matrix 405 6.6.2 Computation of the Eigenvalues of a Hessenberg Matrix.

The Method of Hyman 407 6.6.3 Simple Vector Iteration and Inverse Iteration of Widandt 408 6.6.4 The LR and QR Methods 415 6.6.5 The Practical Implementation of the QR Method 425

xiv Contents

6.7 Computation of the Singular Values of a Matrix 436

6.8 Generalized Eigenvalue Problems 440 6.9 Estimation of Eigenvalues 441

Exercises for Chapter 6 455 References for Chapter 6 462

7 Ordinary Differential Equations 465 7.U Introduction 465

7.1 Some Theorems from the Theory of Ordinary Differential Equations 467

7.2 Initial-Value Problems 471 7.2.1 One-Step Methods: Basic Concepts 471

7.2.2 Convergence of One-Step Methods 477 7.2.3 Asymptotic Expansions for the Global Discretization Error

of One-Step Methods 480

7.2.4 The Influence of Rounding Errors in One-Step Methods 483 7.2.5 Practical Implementation of One-Step Methods 485

7.2.6 Multistep Methods: Examples 492 7.2.7 General Multistep Methods 495

7.2.8 An Example of Divergence 498 7.2.9 Linear Difference Equations 501

7.2.10 Convergence of Multistep Methods 504 7.2.11 Linear Multistep rvIethods 508

7.2.12 Asymptotic Expansions of the Global Discretization Error for Linear Multistep Methods 513

7.2.13 Practical Implementation of Multistep Methods 517 7.2.14 Extrapolation Methods for the Solution of the Initial-Value

Problem 521 7.2.15 Comparison of Methods for Solving Initial-Value Problems 524

7.2.16 Stiff Differential Equations 525

7.2.17 7.2.18 7.2.19

7.3 7.3.0

7.3.1

Implicit Differential Equations. Differential-Algebraic Equations Handling Discontinuities in Differential Equations 536

Sensitivity Analysis of Initial-Value Problems 538

Boundary-Value Problems 539 Introduction 539

The Simple Shooting Method 542 7.3.2 The Simple Shooting Method for Linear Boundary-Value

Problems 548 7.3.3 An Existence and Uniqueness Theorem for the Solution of

Boundary-Value Problems 550 7.3.4 Difficulties in the Execution of the Simple Shooting

Method 552 7.3.5 The Multiple Shooting Method 557

531

Contents xv

7.3.6 Hints for the Practical Implementation of the Multiple Shooting Method 561

7.3.7 An Example: Optimal Control Program for a Lifting Reentry Space Vehicle 565

7.3.8 Advanced Techniques in Multiple Shooting 572 7.3.9 The Limiting Case m -+ 00 of the Multiple Shooting Method

(General Newton's Method, Quasilinearization) 577 7.4 Difference Methods 582 7.5 Variational Methods 586 7.6 Comparison of the Methods for Solving Boundary-Value Problems

for Ordinary Differential Equations 596 7.7 Variational Methods for Partial Differential Equations

The Finite-Element Method 600

8

Exercises for Chapter 7 607 References for Chapter 7 613

Iterative Methods for the Solution of Large Systems of Linear Equations. Additional Methods

8.0 Introduction 619 8.1 General Procedures for the Construction of Iterative Methods 621 8.2 Convergence Theorems 623 8.3 Relaxation Methods 629 8.4 Applications to Difference Methods-An Example 63~)

8.5 Block Iterative Methods 645 8.6 The ADI-Method of Peaceman and Rachford 647 8,7 Krylov Space Methods for Solving Linear Equations 657 8.7.1 The Conjugate-Gradient Method of Hestenes and Stiefel 658 8.7.2 The GMRES Algorithm 667 8.7.3 The Biorthogonalization Method of Lanczos and the QMR

algorithm 680 8.7.4 The Bi-CG and BI-CGSTAB Algorithms 686 8.8 Buneman's Algorithm and Fourier Methods for Solving the

Discretized Poisson Equation 691 8.9 Multigrid Methods 702 8.10 Comparison of Iterative Methods 712

Exercises for Chapter 8 719 References for Chapter 8 727

General Literature on Numerical Methods

Index

619

730

732