Numerical Integration (Quadrature) Why? many integrals cannot be evaluated analytically even if you can, you might need to check your answer(!) even if you can, numerical evaluation of the answer can be bothersome sometimes the integrand is known only for a sample of points

Examples:

2/2

2 20

sin1 tan

xe x dxx x

π −

+∫

0 0

( 1)2cosh 2 1k

k

dxx x k

π∞∞

=

−=+∑∫

An example of an integral that needs checking:

Here is an example of an integrand known only at a set of points:

(and it is also a simple ODE, so see Kyle’s lectures for that.

So we have an array of possible problems to solve:

the integrand is some sort of table of numbers regularly spaced irregularly spaced contaminated with noise

the integrand is computable everywhere in the range of integration

but there may be infinite range of integration local discontinuities integrable singularities

considerations

time to compute the integral estimate of the error due to

♦ truncation ♦ round-off ♦ noise in tabulated values

0. Integral as Riemann sum

01

0

( ) lim ( )

( )

nbi ina i

Ni i

i

f x dx f x x

f x x

→∞=

−

=

= Δ

Δ

∑∫

∑

for some large N.

Let’s try a simple example: /2/2

0 0cos sin 1xdx x

ππ= =∫

n intervals dx error 1 2 0.785398 -0.3407592 4 0.392699 -0.1834653 8 0.196350 -0.0949604 16 0.098175 -0.0482845 32 0.049087 -0.0243436 64 0.024544 -0.0122227 128 0.012272 -0.0061238 256 0.006136 -0.0030659 512 0.003068 -0.001533

10 1024 0.001534 -0.000767 Note that the error is decreasing by a factor 2, just like our discretization interval dx. So what is exact and rigorous in a math book is sometimes, in fact almost always, not so simple on a computer.

Instead of having the top of the rectangle hit the left (or right) edge we could also have it hit the function at the midpoint of each interval:

12

1

0

( )

( )i ix x

baN

ii

f x dx

f x++−

=Δ

∫

∑

Note that the lines at the top of the rectangles can have any slope whatsoever and we will always get the same answer.

n intervals dx error 1 2 0.785398 -0.0261721532 4 0.392699 -0.0064545433 8 0.196350 -0.0016081894 16 0.098175 -0.0004017085 32 0.049087 -0.0001004066 64 0.024544 -0.0000251007 128 0.012272 -0.0000062758 256 0.006136 -0.0000015699 512 0.003068 -0.000000392

10 1024 0.001534 -0.000000098 now the error is falling by a factor 4 with each halving of the interval dx.

Let’s try something else:

the area under the first trapezoid is 1 1( ) ( ( ) ( )) ( ( ) ( ))2 2hf a h f a h f a h f a f a h+ + − = + +

so adding up similar terms for all the trapezoids we get

n intervals dx error 1 2 0.78539816 0.051940552 4 0.39269908 0.012884203 8 0.19634954 0.003214834 16 0.09817477 0.000803325 32 0.04908739 0.000200816 64 0.02454369 0.000050207 128 0.01227185 0.000012558 256 0.00613592 0.000003149 512 0.00306796 0.00000078

10 1024 0.00153398 0.00000020 Now the error is again decreasing by a factor 4, so like dx2.

Numerical integration using the trapezoid rule is very old. Earliest known use: Archimedes (287 B.C. – 212 B.C.), Measurement of a Circle. He alternately circumscribes and inscribes regular polygons around a circle, thus estimating the circumference of a circle relative to its diameter: What he is actually doing is using the trapezoid rule to estimate the integral

1

20

12 1 xπ=

−∫

Let’s try something that curves, e.g. a parabola between (f(a),f(a+h),f(a+2h)), etc.:

the area under each parabola can be easily computed and we get

n intervals dx error 1 2 0.785398 -0.002279877492210 2 4 0.392699 -0.000134584974194 16.943 8 0.19635 -0.000008295523968 16.224 16 0.098175 -0.000000516684707 16.065 32 0.049087 -0.000000032265001 16.016 64 0.024544 -0.000000002016129 16.007 128 0.012272 -0.000000000126001 16.008 256 0.006136 -0.000000000007875 16.009 512 0.003068 -0.000000000000492 16.01

10 1024 0.001534 -0.000000000000029 16.97 and this time the error is decreasing by a factor of 16 each time.

SUBROUTINE trapzd(func,a,b,s,n) INTEGER n REAL a,b,s,func EXTERNAL func INTEGER it,j REAL del,sum,tnm,x if (n.eq.1) then s=0.5*(b-a)*(func(a)+func(b)) else it=2**(n-2) tnm=it del=(b-a)/tnm x=a+0.5*del sum=0. do 11 j=1,it sum=sum+func(x) x=x+del 11 continue s=0.5*(s+(b-a)*sum/tnm) endif return END C (C) Copr. 1986-92 Numerical Recipes Software D..W$5+%.

Typically you don’t actually know the error, so you just keep making dx smaller until successive estimates differ by less than some acceptable value, e.g. 10-6 or so. Note that you can generally reuse earlier results. SUBROUTINE qtrap(func,a,b,s) INTEGER JMAX REAL a,b,func,s,EPS EXTERNAL func PARAMETER (EPS=1.e-6, JMAX=20) CU USES trapzd INTEGER j REAL olds olds=-1.e30 do 11 j=1,JMAX call trapzd(func,a,b,s,j) if (abs(s-olds).lt.EPS*abs(olds)) return olds=s 11 continue pause 'too many steps in qtrap' END C (C) Copr. 1986-92 Numerical Recipes Software D..W$5+%.

Derivations of Formulae One way is to insert the Taylor series, integrate, and simplify terms. Another way uses the Lagrange interpolation polynomials, and you just do the integrations and simplify. One by-product is that an error bound is provided in the process. Another way uses the Newton divided difference scheme to get the interpolating polynomial, which is then integrated term by term. Another way, usually referred to as the method of moments, assumes that

0

( ) ( )nb

ia if x dx w f a ih

== +∑∫

and you get a set of linear equations for the wi’s by requiring the equality to be exact for

2 3( ) 1, , , ,..., nf x x x x x= .

There are higher order rules, but they are of marginal usefulness (as far as I know).

For the trapezoidal rule the error ( )n tI I E n= + is of the form

1

32

2

( )( )( )12

n

iit

n

fb aE n nn

C I In

ζ=

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

=

′′−=

−

∑

If we double n to 2n we get 22(2 )

(2 )t nCE n I In′= −

Combining these (and assuming C C′) gives

22 3

nnn

I II I −+

Repeating gives

4 24 15

n nn

I II I −+

then 8 4n8 63

nn

I II I −+ , etc

This is called Richardson extrapolation or Romberg’s method, depending where you look.

Trapezoid rule

n dx integral error Romberg error 1 2 0.785398 0.948059448969 0.051940551031 2 4 0.392699 0.987115800973 0.012884199027 1.000134584974 -0.00013458497423 8 0.19635 0.996785171886 0.003214828114 1.000008295524 -0.00000829552404 16 0.098175 0.999196680485 0.000803319515 1.000000516685 -0.00000051668475 32 0.049087 0.999799194320 0.000200805680 1.000000032265 -0.00000003226506 64 0.024544 0.999949800092 0.000050199908 1.000000002016 -0.00000000201617 128 0.012272 0.999987450118 0.000012549882 1.000000000126 -0.00000000012608 256 0.006136 0.999996862535 0.000003137465 1.000000000008 -0.00000000000799 512 0.003068 0.999999215634 0.000000784366 1.000000000000 -0.0000000000005

10 1024 0.001534 0.999999803909 0.000000196091 1.000000000000 0.0000000000000 Simpson's Rule

n dx integral error 1 2 0.785398 1.002279877492 -0.002279877492 2 4 0.392699 1.000134584974 -0.000134584974 3 8 0.19635 1.000008295524 -0.000008295524 4 16 0.098175 1.000000516685 -0.000000516685 5 32 0.049087 1.000000032265 -0.000000032265 6 64 0.024544 1.000000002016 -0.000000002016 7 128 0.012272 1.000000000126 -0.000000000126 8 256 0.006136 1.000000000008 -0.000000000008 9 512 0.003068 1.000000000000 0.000000000000

10 1024 0.001534 1.000000000000 0.000000000000

Are we getting something for nothing?

2 0 2

4 0 1 22 2

I hf hfh hI f hf f

= +

= + +

Then

4 24

0 1 20 1 2

0 1 2

0 1 2

3

2 22 2 3

43 3 3

( 4 ) Simpson's rule3

I II Ih hf hf fh hf hf f

h h hf f fh f f f

−= +

− + −= + + +

= + +

= + + =

so we are really getting Simpson’s rule in disguise and of course the error is O(h5). If you do the same with two applications of Simpson’s rule you get Bode’s rule, etc.

Adaptive Quadrature, the trick being to minimize the number of integrand computations

SUBROUTINE qsimp(func,a,b,s) INTEGER JMAX REAL a,b,func,s,EPS EXTERNAL func PARAMETER (EPS=1.e-6, JMAX=20) CU USES trapzd INTEGER j REAL os,ost,st ost=-1.e30 os= -1.e30 do 11 j=1,JMAX call trapzd(func,a,b,st,j) s=(4.*st-ost)/3. if (abs(s-os).lt.EPS*abs(os)) return os=s ost=st 11 continue pause 'too many steps in qsimp' END C (C) Copr. 1986-92 Numerical Recipes Software D..W$5+%.

SUBROUTINE qromb(func,a,b,ss) INTEGER JMAX,JMAXP,K,KM REAL a,b,func,ss,EPS EXTERNAL func PARAMETER (EPS=1.e-6, JMAX=20, JMAXP=JMAX+1, K=5, KM=K-1) CU USES polint,trapzd INTEGER j REAL dss,h(JMAXP),s(JMAXP) h(1)=1. do 11 j=1,JMAX call trapzd(func,a,b,s(j),j) if (j.ge.K) then call polint(h(j-KM),s(j-KM),K,0.,ss,dss) if (abs(dss).le.EPS*abs(ss)) return endif s(j+1)=s(j) h(j+1)=0.25*h(j) 11 continue pause 'too many steps in qromb' END

You can still have trouble, e.g. if the high-order derivatives are not bounded. Consider 2( ) 4 (1 )f x x= − The integral over [0,1] is obviously π, but the derivatives are unbounded near x = 1.

0

1

2

3

4

5

0 0.2 0.4 0.6 0.8 1

Gaussian Integration (or Quadrature) We want ( ) ( )

b

aI f x w x dx= ∫

which we approximate by

1

( ) ( ) ( )nb

k kak

f x w x dx A f x=∑∫

We replace f(x) by an interpolating polynomial P(x) of degree n–1 and a remainder term: ( ) ( ) ( )f x P x r x= + so that at the n points xk ( ) ( )k kP x f x= :

1

( )( ) ( )( ) ( )

n

kk k k

xP x f xx x x

αα=

=′−∑

where α(x) is a completely factored nth-degree polynomial: 1 2( ) ( )( ) ( )nx x x x x x xα = − − ⋅⋅⋅ − Note that

( )lim 1

( ) ( )kx xk k

xx x x

αα→

=′−

So if f(x) is a polynomial of degree n–1 then r(x) = 0 and the approximation is exact.

But whenever r(x) is small, we have approximately

1

1

1

( ) ( ) ( ) ( )

( )( ) ( )( ) ( )

( )( ) ( )( ) ( )

( )

b b

a a

nb

kak k k

n b

k ak k kn

k kk

f x w x dx P x w x dx

xf x w x dxx x x

xf x w x dxx x x

A f x

αα

αα

=

=

=

=′−

=′−

=

∫ ∫

∑∫

∑ ∫

∑

(1)

This will be exact if f(x) is a polynomial of degree n–1. Note that the values of the xk have not been specified. If they are uniformly spaced we recover the various Newton-Cotes formulae.

Gauss noted that if we pick the xk such that the n-th order polynomial α(x) is orthogonal to all lower degree polynomials, i.e. if ( ) ( ) 0

b m

ax x w x dxα =∫

for m < n, then our formula (1) is exact when f(x) is any polynomial of degree 2n–1. Thus the Theorem A necessary and sufficient condition that an interpolatory formula like (1) be exact for all polynomials of degree 2 1n≤ − is that the α(x) satisfy

for all m < n. ( ) ( ) 0b m

ax x w x dxα =∫

Necessity. Assume eq. (1) is exact for f(x) = d(2n–1) and let Q(x) = d(n–1). Then ( ) ( ) ( ) (2 1)f x x Q x d nα= = − . Then

k1

( ) ( ) ( ) ( ) ( )

0

nb

k kak

x Q x w x dx A x Q xα α=

=∑∫

=is exact (by assumption) and the 0 comes from α(xk) = 0. Sufficiency. Assume that α(x) is orthogonal to all polynomial of d(n–1). Let f(x) = d(2n–1) and divide f(x) by α(x) to get

( ) ( )( )( ) ( )

f x xQ xx x

ρα α

= +

or ( ) ( ) ( ) ( )f x x Q x xα ρ= + where both Q(x) and ρ(x) are d(n–1). Note that since α(xk) = 0, we must have ( ) ( )k kf x xρ= .

Then ( ) ( ) ( ) ( ) ( ) ( ) ( )

b b b

a a af x w x dx x Q x w x dx x w x dxα ρ= +∫ ∫ ∫

The first integral on the r.h.s. vanishes by the orthogonality assumption and the second integral is exact since ρ(x) = d(n–1), and since )kf x x( ) (k ρ= , we must have

1

1

( ) ( ) ( )

( )

nb

k kak

n

k kk

f x w x dx A x

A f x

ρ=

=

=

=

∑∫

∑

and this is therefore exact for any f(x) = degree 2n – 1. Gaussian integration will be useful when

(a) the values of f(xk) are available (b) f(x) is well-approximated by a degree 2n – 1 polynomial on [a,b].

In practice there are three major systems of classical orthogonal polynomials: Jacobi

1 ( , ) ( , )

1(1 ) (1 ) ( ) ( )n mx x P x P x dxα β α β α β

−− +∫

(associated) Laguerre ( ) ( )

0( ) ( ) , 1x

n mx e L x L x dxα α α α∞ − > −∫

Hermite

2

( ) ( )xn me H x H x dx

∞ −

−∞∫

special cases of Jacobi: α = β = 0 Legendre α = β = –1/2 Chebyshev 1st kind α = β = +1/2 Chebyshev 2nd kind

general properties:

(a) if on [–1, 1] then the zeros x( ) 0w x ≥ k all lie in the interval (–1, 1). (b) the zeros of pn are interlaced within the zeros of pn+1. (c) the weights Ak are all positive. (d) the zeros xk and weights Ak are symmetric about zero.

Electrostatic analogy put a charge α+1 at +1, a charge β+1 at –1, and n charges of charge +2 on the interior line. Let the force between the charges be coulomb in 2-d, so

1/( )ij i jF x x−∼ . Then the equilibrium position of the n charges will be at the zeros xk of the Jacobi polynomial Pn

(α,β)(x).

With various changes of variable one can get many other formulae, e.g.

( ) ,2 2

b

a

b a b af y dy y x− += +∫

1

0( )kx f x dx∫

( ) , ( )b

ab y f y dy y a b a x− = + −∫

( ) , ( )

b

a

f y dy y a b a xb y

= + −−∫

( ) ( ),

2 2( )( )b

a

f y b a b ady y xy a b y

+ −= +

− −∫

( )( )( ) ( ) ,

2 2b

a

b a b ay a b y f y dy y x+ −− − = +∫

( )( ) ,

2 2b

a

x a b a b af y dy y xb x− + −

= +−∫

The formulae so far are open, so the function is not evaluated at the endpoints x = ≤1. The closed formulae are Radau (degree 2n – 2 is exact)

11

211

2( ) ( 1) ( )n

k kk

f x dx f A f xn

−

−=

= − +∑∫

Lobatto (degree 2n – 3 is exact)

11

12

2( ) [ ( 1) (1)] ( )( 1)

n

k kk

f x dx f f A f xn n

−

−=

= − + +− ∑∫

What about reuse of earlier function evaluations? Gauss–Kronrod: start with Gn (degree 2n – 1 is exact) and add n + 1 abscissas so that degree 3n – 1 is exact. All the weights change, but this enables reuse of the function values.

Some examples from Davis and Rabinowicz

an interesting alternative is Clenshaw-Curtis integration: http://en.wikipedia.org/wiki/Clenshaw-Curtis_quadrature and Num. Recipes expand f(x) in a Chebyshev series

1 11

1( ) ( )2

n

k kk

f x c T x c−=

= −∑

1

1 ( ) ( )n

j k j kk

c f x T xn =

= ∑

cos( ) 0,1,...,kkx k nnπ

= =

Then the integral

1 3 2 11 1 1( ) ( )[ ... ...)2 3 (2 1)(2 1)

b

kaf x dx b a c c c

k k += − − − − −− +∫

If you double the number of points you get complete reuse of the coefficients and for large n you can use an FFT to compute the initial transform.

Some results from Trefethen http://web.comlab.ox.ac.uk/oucl/work/nick.trefethen/CC_trefethen_revised2.pdfhttp://web.comlab.ox.ac.uk/oucl/work/nick.trefethen/cctalk.pdf

http://en.wikipedia.org/wiki/Image:Legendre_poly.svg

http://en.wikipedia.org/wiki/Image:Legendre_poly.svg

http://en.wikipedia.org/wiki/Image:Legendre_poly.svg

Monte Carlo Integration

Recall the rectangle rule in 1–d:

1

0

( )( ) ( )

( ) ( )

nb

iai

b af x dx f xn

b a f x

−

=

−=

= −

∑∫

and recall that the error decreases as 1/n. By the second mean-value theorem of calculus, the integral would be exact if we really knew the average of f(x) over [a, b], and the error arises from our (inaccurate) estimation of ( )f x .

Of course, there are other ways to compute an average. For example, suppose we have a large field of known area A which contains a pond of unknown area. We can estimate the pond area a by randomly throwing n stones into the field and counting the number of splashes s, and so

sa An

=

This is usually called the hit-or-miss method.

To use this to estimate an integral ( )b

af x dx∫ we could imagine a box of width

(b–a) and height h (and hence area A = h(b–a) )

We would then pick a number n random points (x,y) in

[b–a,h] and count the number of times s that y < f(x). Then an estimate of the integral is As/n.

Trying this on 2( ) 4 (1 )f x x= − gives

n npts integral error err*sqrt(npts)

1 2 4.000000 -0.858407 -1.213971 2 4 3.000000 0.141593 0.283185 3 8 3.000000 0.141593 0.400485 4 16 3.000000 0.141593 0.566371 5 32 3.000000 0.141593 0.800969 6 64 3.000000 0.141593 1.132741 7 128 3.218750 -0.077157 -0.872936 8 256 3.156250 -0.014657 -0.234518 9 512 3.195313 -0.053720 -1.215541

10 1024 3.144531 -0.002939 -0.094035 11 2048 3.093750 0.047843 2.165111 12 4096 3.124023 0.017569 1.124430 13 8192 3.157227 -0.015634 -1.415020 14 16384 3.142578 -0.000985 -0.126140 15 32768 3.128540 0.013053 2.362776 16 65536 3.145447 -0.003854 -0.986656 17 131072 3.145081 -0.003488 -1.262759 18 262144 3.141052 0.000540 0.276689 19 524288 3.142059 -0.000467 -0.337907 20 1048576 3.142883 -0.001291 -1.321623

Another way is to pick some number n points xi in [a, b] and compute the average of the function values at those points. Then the integral is approximately

1

0

1( ) ( ) ( )

( ) ( )

nb

iai

f x dx b a f xn

b a f x

−

=

= −

= −

∑∫

This is usually called the sample-mean method.

This is just like the rectangle method except the xi are randomly spaced instead of uniformly spaced.

Trying this on 2( ) 4 (1 )f x x= − gives

n npts integral error err*sqrt(npts) 1 2 3.508162 -0.36657 -0.518412 4 3.367358 -0.22577 -0.451533 8 2.562617 0.578976 1.637594 16 3.055518 0.086074 0.3442985 32 3.06597 0.075622 0.4277856 64 3.07479 0.066803 0.534427 128 2.998369 0.143224 1.6203958 256 3.236035 -0.09444 -1.511089 512 3.170993 -0.0294 -0.66526

10 1024 3.153875 -0.01228 -0.3930211 2048 3.145846 -0.00425 -0.1924812 4096 3.13816 0.003432 0.21966513 8192 3.142739 -0.00115 -0.1037614 16384 3.146292 -0.0047 -0.601515 32768 3.14402 -0.00243 -0.4393216 65536 3.140908 0.000685 0.17526517 131072 3.143268 -0.00168 -0.6067218 262144 3.141268 0.000324 0.16612219 524288 3.141802 -0.00021 -0.1517320 1048576 3.140781 0.000811 0.83085

Using either method we are getting errors which decrease proportional to 1/ n . The advantage is that we can add points at will, check for convergence, and quit anytime we like.

Yet using uniformly spaced points we get errors that scale like 1/n. The disadvantage is that once committed to a specific value of n we must compute all the n integrand values.

There is a compromise: instead of using psuedo – random numbers we use quasi – random numbers.

These have the quality that they sample in interval [a, b] without somewhat ‘randomly’ but clustering or leaving voids. We can stop sampling anytime we like, and the convergence is proportional to (ln Nn n , where N is the number of dimensions (N = 1 in the examples so far).

) /

Trying this on 2( ) 4 (1 )f x x= − gives

0.000001

npts

0.00001

0.0001

0.001

0.01

0.1

11 10 100 1000 10000 100000 1000000 10000000

pseudo-quasi-uniform-

For higher dimensions things are even better.

Importance Sampling

When integrating x by Monte Carlo, or for that matter any other method, it really helps if the integrand is as smooth and level as possible. In fact, it is best if the integrand is a constant, for which any method will give the exact answer!

( )b

af x d∫

The technical reason for this is that the Monte Carlo method is based on

22VfdV V f f f

n≈ ± −∫

where

1

1 ( )n

ii

f f xn =

≡ ∑ and 2 2

1

1 ( )n

ii

f f xn =

≡ ∑

Hence if f is constant, f equals that constant and 2f f− is zero.

So let’s introduce p(x) positive on [a, b] and such that ( ) 1

b

ap x dx =∫

and rewrite our integral as

( ) ( )( )

b

a

f x p x dxp x

⎡ ⎤⎢ ⎥⎣ ⎦

∫

picking p(x) to look as much as possible like f(x). Then if we can sample random numbers xi from the probability distribution ( )p x dx we can estimate the integral as

1

1 ( )( )

n

ni

f xFn p x

=

= ∑

Note that if 1( )p xb a

=−

this reduces to sampling from a uniform distribution.

It is fairly easy to show that the optimum choice is ( ) ( )p x f x∼ but of course we still have to evaluate ( ) ( )

b b

a ap x dx f x dx=∫ ∫

and if we could do that why would we be bothering with Monte Carlo? So instead we must pick some p(x) that we can handle and which will still make f(x)/p(x) as level as possible.

How do we sample random numbers from an arbitrary distribution? The best way is the transformation method, if you can do it. Suppose we can generate a uniform deviate x and compute some function of it, y(x). Then the probability distribution of y, p(y)dy, is

( ) ( ) dxp y p xdy

=

For example, suppose that )( ) ln(y x x− and that p(x) is uniform. Then =

( ) ydxp y dy dy e dydy

−= =

Note that to get this we have to solve the differential equation

( )dx f ydy

=

whose solution is ( ) ( )y

x F y f y dy= = ∫ and we must then invert this to get

1( ) ( )y x F x−=

If we cannot do the indefinite integral and the function inversion another method is acceptance-rejection.

Here is a set of worked examples: http://www.eng.buffalo.edu/~kofke/applets/MCApplet2.html

All of the above about Monte Carlo has been in 1–d for simplicity, but the real power of MC is for higher values of dimension d. The Curse of Dimensionality Let’s consider that 6–d integral we saw above, and think about evaluating it using some n–point method. Even if we could get by with just 10 points per integral we would still have 106 evaluations of the integrand. But something else happens: if our method for 1–d gives an error that goes like n α− . then in d dimensions the error will go like / dn α− . So when the dimension gets large we have to do a lot of work with decreasing return on our investment. Monte Carlo, on the other hand, gives an error that goes at worst as 1/ 2n− for any dimension, and with a few tricks such as importance sampling, quasi-random numbers, etc we can often do even better.