lecture 3 introduction to numerical methods for differential and

Lecture 3Introduction to Numerical Methods for Differential

and Differential Algebraic Equations

Dr. Abebe Geletu

Ilmenau University of TechnologyDepartment of Simulation and Optimal Processes (SOP)

Winter Semester 2011/12

Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

TopicsNewton’s Law:mx = −F

l xmy = mgFl yConservation of mechanicalenergy: x2 + y2 = l2

(DAE) x1 = x3

x2 = x4

x3 = − F

m lx1

x4 = gF

lx2

0 = x2 + y2 − l2.1

2 Numerical Methods of Ordinary Differential Equations1 Initial Value Problems (IVPs)

Single Step MethodsMulti-step Methods

2 Boundary Value ProblemsFinite Difference MethodsSingle- and Multiple-Shooting Methods

3 Numerical Methods of Differential Algebraic Equations (DAEs)DAE models in engineering applicationsPeculiarities of DAEsIndex notions for DAEsBackward Difference Formula (BDF)Implicit Rung Kutta (IRK) MethodCollocation on finite elements


TU Ilmenau

Initial Value Problems

General form of an initial value problem (IVP) is

x(t) = f(t, x(t)), t0 ≤ t ≤ tf , (1)

x(t0) = x0 (given), (2)

where x and f can be vector functions:

x(t) =

x1(t)x2(t)

...xn(t)

, f(t, x(t)) =

f1(t, x(t))f2(t, x(t))

...fn(t, x(t))

.

If f is (explicitly) time independent; i.e. f(t, x(t)) = f(x(t)), thenthe system (1) - (2) is said to be autonomous.


TU Ilmenau

x(t0) = x0 is the initial condition (start time of the motion)

The expression x(t) represents the first order derivative of x(t)with respect to time; i.e. x(t) = d

dtx(t). Thus, the IVP (1) - (2)represents a system of first order differential equations.

If f(t, x(t)) = A(t)x(t), then we havex(t) = A(t)x(t), x(t0) = x0 is a first order linear IVP. In theautonomous case we have A(t) = A.

Example: Equation of motion of a simple pendulum:

θ = ω

ω = − gLsin(θ)

⇒(θω

)=

(ω

− gLsin(θ)

),θ(0) = π

4ω(0) = 0


TU Ilmenau

Commonly, numerical methods are developed for systems of firstorder differential equations.⇒ any higher order differential equation should be written as asystem of first order differential equations.

NB: The Matlab ODE Toolbox works only with systems of first orderdifferential equations.Strategy: Convert any n-th order initial value problem into a system

of first-order IVP by introduction new variables.

dnx(t)

dtn= f

(t,dx

dt,d2x

dt2, . . . ,

dn−1x

dtn−1

)x(t0) = a1, x(t0) = a2,

dx(t)

dt

∣∣∣∣t=t0

= a3, . . . ,dn−1x(t)

dtn−1

∣∣∣∣t=t0

= an.

Introduce new variables:x1 = x, x2 = x, x3 = d2x(t)

dt2, . . . , xn = dn−1x(t)

dtn−1 .


TU Ilmenau

Then the n-th order IVP reduces to

x1 = x2, x1(t0) = a1

x2 = x3, x2(t0) = a2...

xn = f(t, x1, x2, . . . , xn), xn(t0) = an.

Example: The equation of motion of a simple pendulum (neglectingair resistance) with mass m is a 2nd-order equation

mLθ +mg sin(θ(t)) = 0, (3)

θ(t0) = θ0, θ = ω0. (4)

Introduce ω = θ. Then

θ = ω, θ(t0) = θ0

ω = − gL

sin(θ), ω(t0) = ω0.


TU Ilmenau

Objective:To determine a continuous function x∗(t) that satisfies the dynamicequation (1); i.e. x∗(t) = f(t, x∗(t)) and the initial condition (2); i.e.x∗(t0) = x0.

!!! In general, the solution function x∗(t) of the IVP is not easy to beanalytically determined.


TU Ilmenau

Basic idea of numerical solution of differential equations:

Select a set of equally spaced time points t1, t2, . . . , tN from theinterval [t0, tf ] (tf freely chosen for IVPs). That is, t1 = t0 + h ,t2 = t1 +h, and so on; so that h = tk+1− tk, k = 0, 1, . . . , N − 1.

Approximate the continuous solution x∗(t) of (1) & (2) by valuesat the discrete time points t1, t2, . . . , tN ; i.e.x(t1) = x1, x(t2) = x2, . . . , x(tN ) = xN .

determine the values x1, x2, . . . , xN through the IVP (1) - (2).

Note:more discretization points from [t0, tf ] implies

better accuracy of the approximation; but

demands more computation effort,

depending on the algorithm used, it may be necessary to solve asystem of linear or nonlinear equations.


TU Ilmenau

Euler’s MethodMotivation: for a single IVP x(t) = f(t, x(t)), x(t0) = x0; i.e.x(t) ∈ R, 0 ≤ t ≤ tf .

Given the point (t0, x0) the value x(t0) = dx(t)dt

∣∣∣t=t0

is the slope of

the tangent line to the graph of x(t) at the point (t0, x0).

Figure: Euler’s Method


TU Ilmenau

Since the function x(t) is not yet known, the derivative (slope) can beapproximated by

x(t0) ≈x(t0 + h)− x(t0)

hThus, using x(t0) = f(t0, x0), we have

x(t0 + h)− x(t0)

h= f(t0, x(t0))

This impliesx(t0 + h) = x(t0) + hf(t0, x(t0))

⇒x1 = x(t1) = x(t0 + h) = x(t0) + hf(t0, x(t0))

From this we obtain the value of x1 = x(t1). By the same procedurex2 can be also determined.In general, once xk is known, xk+1 can be found from

xk+1 = xk + hf(tk, xk), k = 0, 1, . . . ,m− 1.


TU Ilmenau

Euler’s AlgorithmSelect N .Set h =

(tf−t0)N ; x0 = x(t0)

FOR k = 0 : (n− 1)xk+1 = xk + hf(tk, xk);tk+1 = tk + h;ENDFOR.

Matlab:function [t,x]=myEuler(@f,x0,t0,tf,N)

h=(tf-tn)/N;

t=[t0:h:tf]; %discrete time points between, and including, t0 and tf

x(1)=x0;

for k=2:N

x(k)=x(k-1) + h*feval(f,t(k-1),x(k-1));

end;


TU Ilmenau

Simple Pendulum with Euler’s Method

function time=MyPenEuler(N,tf)

t=cputime;

m=1;

L=9.81;

g=9.81;

beta=0;

t0=0;

theta0=(1/10)*pi;

omega0=0;

h=(tf-t0)/N;

tt=[t0:h:tf];

ttheta=zeros(size(tt));

oomega=zeros(size(tt));

ttheta(1)=theta0;

oomega(1)=omega0;Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

Simple Pendulum with Euler’s Method...

for k=1:N

oomega(k+1)=oomega(k)+h*...

(-beta/m/L*oomega(k)-g/L*sin(ttheta(k)));

ttheta(k+1)=ttheta(k)+h*oomega(k);

end

time=cputime-t;

plot(tt,oomega,tt,ttheta)

title(’Simple pendulum using Euler’’s method’)


TU Ilmenau

Advantages/Disadvantages of Euler’s Method

Advantages:

Euler’s Method is simple and direct

Can be used for nonlinear IVPs

Disadvantages:

it is less accurate and numerically unstable.

Approximation error is proportional to the step size h. Hence,good approximation is obtained with a very small h. Thisrequires a large number of time discretization leading to a largercomputation time.

Usually applicable to explicit differential equations.


TU Ilmenau

Improvements on Euler’s Method(A) The Mid-Point MethodLet k1 = hf(ti, xi).Note that: k1 = hf(ti, xi) = xi+1 − xi. But to compute k1, it is notnecessary to know the value of xi+1.

Figure: Use also the mid point


TU Ilmenau

Improvements on Euler’s Method...

• The point (ti + 12 ti, xi + 1

2k1) is the mid-point of the segment thatconnects the points (ti, xi) and (ti+1, xi+1).

• Now, instead of the single slope at (ti, xi), we use the slopes at(ti, xi) and at (ti + 1

2 ti, xi + 12k1) to approximate xi+1.

⇒ This leads to the Mid-Point Runge-Kutta Algorithm:

The mid-point Runge-Kutta Method

k1 = hf(ti, xi)

k2 = hf(ti +1

2h, xi +

1

2k1)

xi+1 = xi + k2.


TU Ilmenau

Improvements on Euler’s Method...(B) Methods that use Intermediate PointsQuestion: Can we improve the mid-point RK method by freelychoosing any point from the segment connecting (ti, xi) and(ti+1, xi+1)?

Figure: Choose freely any intermediate point


TU Ilmenau

• A weighted sum of f(ti, xi) and f(ti + ch, xi +mk1) providedsmore slope information than f(ti, xi) alone.Scheme:

(RK2) x(t+ h) = x(t) + hf(t, x)

where f(t, x) = [w1f(t, x) + w2f(t+ ch, x+mk1)] .How to determine w1, w2 , c and h?We use:(i) Taylor approximation of f(t+ ch, x+mk1):

f(t+ ch, x+mk1) = f(t, x) + chft(t, x) +mk1fx(t, x)x′ (∗)

Put this into (*) to obtain

x(t+h) = x(t)+h [w1f(t, x) + w2{f(t, x) + chft(t, x) +mk1fx(t, x)})] (∗)

⇒ (with k1 = hf(t, x))

x(t+h) = x(t)+h (w1 + w2) f(t, x)+w2ch2ft(t, x)+mw2h

2fx(t, x)f(t, x)


TU Ilmenau

Improvements on Euler’s Method...(ii) 2nd- order Taylor approximation of x(t+ h):

x(t+ h) = x(t) + hx′(t) +h2

2x′′(t) (∗∗)

x(t+ h) = x(t) + hx′(t) +h2

2x′′(t) (∗∗)

Recall x′(t) = x(t) = f(t, x)⇒ x′′(t) = ft(t, x) + fx(t, x)f(t, x).Putting these into (**) we obtain

x(t+ h) = x(t) + hf(x, t) +h2

2ft(t, x) +

h2

2fx(t, x)f(t, x)) (∗∗)

Comparing (*) and (**) we find w1 + w2 = 1, cw2 = 12 ,mw2 = 1

2 .

A system of three equations with four unknowns ⇒ This system hasinfinitely many solutions.


TU Ilmenau

Improvements on Euler’s Method...• By fixing one of the unknowns w1, w2, c and m, we can solve thesystem of equations for the other three.⇒ Hence, we can construct several RK2 methods.

Example:• If we fix w1 = 0, then w2 = 1, c = 1

2 and m = 12 . Hence, (RK2)

reduces to the mid-point RK2 method.

w1 w2 c m Method

0 1 12

12 Modified Euler Method

12

12 1 1 Heun’s Method

13

23

34

34 Ralston’s Method

Heun’s Method:k1 = f(ti, xi)xi+1 = xi + h

[12f(ti, xi) + 1

2f(ti + h, xi + k1)]


TU Ilmenau

Runge-Kutta MethodsThe above schemes are known as second-order Runge-Kuttamethods, since x(t+ h) has been approximated by a second degreeTaylor polynomial.I For an accurate approximation use more intermediate pointsbetween (ti, xi) and (ti + h, xi + k1) and include more slopeinformation.I Requires higher degree Taylor approximations of x(t+ h).

Figure: Use several intermediate pointsLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

Runge-Kutta Methods...

General Scheme:

[RK(m)] xi+1 = xi + hf(ti, xi);

wheref(ti, xi) = w1f(ti, xi) + w2f(ti + α1h, xi + β1k1) + . . .+ wmf(ti + αmh, xi + βmkm).

• The values k1, k2, . . . , km are found at the intermediate points.• w1 + w2 + . . .+ wm = 1.• The relation between ω1, . . . , ωm, α1, . . . , αm, β1, . . . , βm isdetermined by comparing the m-th order Talyor approximation ofx(t+ h) and the scheme RK(m)


TU Ilmenau

I Hence, the resulting method is called an m-th order Runge-Kuttamethod.I There are third-order Runge-Kutta (RK3), fourth-orderRunge-Kutta methods (RK4), etc.The classical 4-th order Runge-Kutta Method has the form:

(RK4s) k1 = f(ti, xi)

k2 = hf(ti +1

2h+ xi +

1

2k1)

k3 = hf(ti +1

2h+ xi +

1

2k2)

k4 = hf(ti + h, xi + k3)

xi+1 = xi +1

6k1 +

1

3k2 +

1

3k3 +

1

6k4


TU Ilmenau

The classical 4th order Runge-Kutta Method

function myRK4(N,tf)

m=5; %Mass

L=9.81; %length of the string

g=9.81; %gravitaional acceleration

beta=0; %

t0=0;

theta0=(1/10)*pi;

omega0=0;

h=(tf-t0)/N

tt=[t0:h:tf];

ttheta=zeros(size(tt));

oomega=zeros(size(tt));

ttheta(1)=theta0;

oomega(1)=omega0;


TU Ilmenau

%The classic 4th order Runge Kutta method

for k=1:N

k1=h*(-beta/m/L*oomega(k)-g/L*sin(ttheta(k)));

m1=h*oomega(k);

k2=h*(-beta/m/L*(oomega(k)+k1/2)-g/L*(sin(ttheta(k)+m1/2)));

m2=h*(oomega(k)+m1/2);


m3=h*(oomega(k)+k2/2);


m4=h*(oomega(k)+k3);

oomega(k+1)=oomega(k)+1/6*(k1+2*k2+2*k3+k4);

ttheta(k+1)=ttheta(k)+1/6*(m1+2*m2+2*m3+m4);

end

% Plot the solutions.

plot(tt,oomega,tt,ttheta)

title(’Simple pendulum solved with 4th order Runge Kutta’)


TU Ilmenau

In general the 4-th order Runge-Kutta Methods take the form:

(RK4) k1 = f(ti, xi)

k2 = hf(ti + c2h, xi + a21k1)

k3 = hf(ti + c3h, xi + a31k1 + a32k2)

k4 = hf(ti + c4h, xi + a41k1 + a42k2 + a43k3)

xi+1 = xi + ω1k1 + ω2k2 + ω3k3 + ω4k4

where the unknown parameters are organized as

0c2 a21c3 a31 a32c4 a41 a42 a43

ω1 ω2 ω3 ω4

known as Butcher’s table. As a result it is possible to constructvarious 4th order Runge-Kutta methods.


TU Ilmenau

Question: When do RK methods fail?Example:

x1 = −100x1 (5)

x2 = −0.1x2. (6)

The state x1(t) = x1(0)e−100t and x2(t) = x2(0) = e−0.1t. x2(t)decays faster than x1(t). So it is not reasonable to compute suchsystems with the same step length h.


TU Ilmenau

Advantages and Disadvantages of Simple RKMethods

Advantages

they are easy to implement

the are stable

Disadvantages

they require relatively large computer time

error estimation are not easy to be done

the above simple RK methods do not work well for stiffdifferential equations (eg. linear differential equations with widelyspread eigenvalues)

In particular, they are not good for systems of differentialequations with a mix of fast and slow state dynamics.


TU Ilmenau

Adaptive Runge-Kutta MethodsThe choice of a good step size h is crucial. In particular, this isimportant for stiff differential equations:• too large h may lead to inaccurate approximation of the solution;• too small h forces the algorithm use large computation resources.⇒ The selection of the step length h is a compromise betweennumerical accuracy and speed of computation.I Adaptive adjustment of the step size h reduces (local) truncationerror for approximating x(ti+1) by xi+1:

‖x(ti+1)− xi+1‖Recall the Taylor approximation:

x(ti + h) = x(ti) + hx(ti) +h2

2x(ti) +

h3

3!

...x (ti) +

h4

4!

....x (ti) + . . .

= xi + C1h+ C2h2 + C3h

3 + C4h4 + . . .


TU Ilmenau

Adaptive Runge-Kutta Methods...

• Adaptive strategies adjust the step-length from iteration to iterationby testing the accuracy iterates.• Usually this test of accuracy is done by comparing lower and higherorder RK algorithms.⇒ Hence, adaptive RK methods combine lower and higher orderRK methods to adjust the step-size h.

(RK3) k1 = f(ti, xi)

k2 = hf(ti +1

2h+ xi +

1

2k1)

k3 = hf(ti + h, xi − k1 + 2k2)

xRK3i+1 = xi +

h

6(k1 + 4k2 + k3) .


TU Ilmenau

(RK4s) k1 = f(ti, xi)

k2 = hf(ti +1

2h+ xi +

1

2k1)

k3 = hf(ti +1

2h+ xi +

1

2k2)

k4 = hf(ti + h, xi + k3)

xRK4i+1 = xi +

h

6(k1 + 2k2 + 2k3 + k4)

• Both xRK4i+1 and xRK4

i+1 are approximations for x(ti+1).Thus the local truncation errors (using the Taylor approximation ofx(t+ h) with ti+1 = ti + h):

from (RK3) is x(ti+1)− xRK3i+1 (ti+1) = C3h

3 + . . .from (RK4) x(ti+1)− xRK4

i+1 = C4h4 + . . .

⇒ (subtracting the second from the first)

xRK4i+1 − xRK3

i+1 ≈ C3h3


TU Ilmenau

I Hence, the truncation error can be estimated in two useful ways:(i) on the hand, the local truncation error is can be approximated by

ErrTrunc = |xRK4i+1 − xRK3

i+1 | ≈ ‖C1‖h3 (7)

In the next iteration step we would like to choose a new step-sizehnew so that the truncation error below a give tolerance:

ErrTruncnew ≈ ‖C1‖h3new < ErrTol. (8)

I The new discretization step-size hnew should be chosen in such away that the resulting truncation ErrTruncnew lies below a giventolerance level ErrTol.I The step-size should be adjusted relative to the truncation errorcommitted in the previous step and the given error tolerance for thenext step (taking ratios of terms in equations (7) and (8))

‖C1‖h3new‖C1‖h3

<ErrTol

ErrTrunc.


TU Ilmenau

Adaptive Runge-Kutta Methods...⇒

hnew < h

(ErrTol

ErrTrunc

)1/3

In particular the new h can be computed adaptively by

hnew = γh

(ErrTol

ErrTrunc

)1/3

for a fixed γ such that 0 < γ < 1.

(ii) on the other hand, comparing the algorithms (RK3) and (EK4s)s,

ErrTrunc = |xRK4i+1 − xRK3

i+1 | =h

6[−2k2 + k3 + k4]

These lead to an adaptive 3-4 Runge-Kutta algorithm.


TU Ilmenau

Adaptive Runge-Kutta Methods...an RK34

• Set t0,x0, initial step size h, ErrorTol, tf and γ,while t < tf do• Compute

k1 = f(t, x)k2 = hf(ti + 1

2h+ xi + 12k1)

k3 = hf(ti + h, xi + 12k2)

k4 = hf(ti + h, xi + k3)• Set ErrTrunc← h

6‖2k2 − k3 − k4‖.if ErrTrunc ≤ ErrorTol then• Set xi+1 ← xi + h

6 (k1 + 2k2 + 2k3 + k4)• Set t← t+ h

end if• New step-size: h← γh

(ErrorTolErrTrunc

)1/3end while


TU Ilmenau

Common adaptive methods are the Runge-Kutta-Fehlberg(RKF)formulas.• RKF23 (a combination of 2nd and 3rd order RKs), RKF45, etc.Matlab ode functions:I ode23 is an adaptive Runge-Kutta methods and a combination of a2nd and 3rd order Runge-Kutta methods.I ode45 is an adaptive RK known as the Dormand-Prince method.Example: Mass-springer dumper system:

Mx+ Cx+ kx = F

with mass M = 5kg,dumper valueC = 10Ns/m, springconstant k = 1000N/m,F = sin(20t)N withforcing frequency ω = 20.


TU Ilmenau

Mass-springer dumper system...MatlabTo solve 5x+ 10x+ 100x = sin(20t), with x(0) = 0, x(0) = 0.⇒ x1 = x2, x2 = −2x2 − 20x1 + 0.2 ∗ sin(20t).

function springDumper(N,tf)

t0=0;

h=(tf-t0)/N;

x10=0;

x20=0;

tt=t0:h:tf;

[t,x]= ode23(@diffEq,tt,[x10,x20]);

subplot(2,1,1), plot(t,x(:,1));

title(’Result from Ode23’)

[t,x]= ode45(@diffEq,tt,[x10,x20]);

subplot(2,1,2), plot(t,x(:,1),’r’);

title(’Result from Ode45’)

end


TU Ilmenau

Mass-springer dumper system...Matlab...

function dxdt=diffEq(t,x)

dx1dt = x(2);

dx2dt = -2*x(2)-20*x(1)+0.2*sin(20*t);

dxdt=[dx1dt;dx2dt];

end


TU Ilmenau

Adaptive Runge-KuttaMethods...Advantages/Disadvantages

Advantages:• provide better precision in the numerical approximation statetrajectories;• step-sizes can be automatically adapted to the dynamics of thestates;⇒ process dynamics with stiff-differential equation models can beefficiently solved

Disadvantages:• such algorithms may take enormous computation time;• simulation of dynamic systems with several state equations can becomputationally demanding;


TU Ilmenau

Multi-step and Implicit MethodsEuler, Runge-Kutta and adaptive RK methods are single-step andexplicit methods.

• Multi-step methods: use a set of earlier iterates up to the currentiterate xi to compute the next iterate xi+1.⇒ Earlier iterates xi, xi−1, . . . , xi−m are used to compute xi+1.For example:

a two-step method uses: xi and xi−1 to compute xi+1

a three-step method uses: xi, xi−1 and xi−2 to compute xi+1

• Implicit methods:

xi+1 = f(ti+1, xi+1),

where xi+1 = x(t)|t=ti+1 . Hence, xi+1 appears implcitily inf(ti+1, xi+1).


TU Ilmenau

Multi-step and Implicit Methods...I BDF is a well known efficient multi-step method:BDF = Backward Differentiation FormulaI While IRK is a well known implicit method:IRK = Implicit Runge-Kutta MethodI Both BDF and IRK are widely used for the direct discretization ofcontinuous dynamic optimization problems.I Particularly, when the IVP involves a stiff-differential equations,BDF and IRK are methods of choice.Stiff differential equations: A differential equation

x = f(t, x)

is stiff if the (Jacobi) matrix ∂f∂x has eigenvalues with widely spread

negative real parts.⇒ some states change fast while others are slower.Examples: mass-spring systems with large spring constants,(chemical) process engineering models, etc.


TU Ilmenau

Multi-step and Implicit Methods...BDFI Adaptive RK methods can solve some stiff problems, but they arein general not efficient for a system with several states.(A) BDF: Given xi, xi−1, . . . , xi+1−m to compute xi+1. Find an m-thdegree polynomial P that interpolates the m+ 1 points

(ti+1, xi+1), (ti, xi), (ti−1, xi−1), . . . , (ti+1−m, xi+1−m),

A good choice for the interpolating polynomials P is

P (t) =

m∑j=0

xi+1−jLj(t)

with the Lagrange polynomial

Lj(t) =

m∏l = 0l 6= j

[t− ti+1−l

ti+1−j − ti+1−l

], j = 0, 1, . . . ,m.


TU Ilmenau

BDF...Observe that

P (ti+1−j) = xi+1−j , j = 0, 1, . . . ,m.

In particular P (ti+1) = xi+1.Thus, replace xi+1 by P (ti+1) to obtain

P (ti+1) = f(ti+1, xi+1). (∗)

But

P (ti+1) =

m∑j=0

xi+1−jLj(ti+1) = xi+1L0(ti+1) +

m∑j=1

xi+1−jLj(t)

Putting this into (*) we get:

xi+1 = −m∑j=1

xi+1−jLj(ti+1)

L0(ti+1)+

1

L0(ti+1)f(ti+1, xi+1)


TU Ilmenau

Define

aj =Lj(ti+1)

L0(ti+1), bm =

1

hL0(ti+1).

An m-step BDF Algorithm (BDF m):Given a1, . . . , am,bm

xi+1 = −m∑j=1

ajxi+1−j + bmhf(ti+1, xi+1)

Example:(a) A BDF 2 algorithm

xi+1 =

(4

3xi −

1

3xi−1

)+

2

3hf(ti+1, xi+1)

(b) A BDF 3 algorithm

xi+1 =

(4

3xi −

9

11xi−1 +

2

11xi−2

)+

6

11f(ti+1, xi+1)


TU Ilmenau

BDF...• If f is a nonlinear function, all BDF algorithms require the solutionof nonlinear equaitons.=⇒ BDF is used in conjuction with a Newton method.

I Matlab function: ode15s uses a BDF algorithm.

Advantages:• are suitable for the solution of stiff differential equations• can be easily generalized to solve DAEs• BDF 2 up to BDF 6 are known to be stable• widely used in circuit simulationDisadvantages:• may require the solution of nonlinear equations• faces difficulties if the step-size h is constant (i.e. requiresadaptation of the step size).


TU Ilmenau

Multi-step and Implicit Methods...IRK

(B) IRK:I If xi is an approximation of x(ti), how to determine theapproximation xi+1 of x(ti+1)?Idea:Approximate x(ti+1) by xi+1 = P (ti+1), where P is a polynomialsatisfying (the requirements):

P (ti) = xi (9)

P (ti + τlh) = f(ti + τlh, P (ti + τlh)), l = 1, . . . ,m, (10)

where τ1, τ2, . . . , τm ∈ (0, 1) and h = ti+1 − ti the steps size.• Note that since P is a polynomial, P is also a polynomial.


TU Ilmenau


TU Ilmenau

IRK..Now define P as

P (ti + τh) =

m∑l=1

f(ti + τlh, P (ti + τlh))Ll(τ),

with

Ll(τ) =

m∏j = 1l 6= j

[τ − τjτl − τj

](11)

• This special polynomial satisfies equation (10).• Integrating equation (9) on both sides∫ τj

0P (ti + τh) =

m∑l=1

f(ti + τlh, P (ti + τlh))

∫ τj

0Ll(τ), j = 1, . . . ,m.


TU Ilmenau

IRK..It follows that

(i) P (ti+τjh) = xi+h

m∑l=1

f(ti+τlh, P (ti+τlh))

∫ τj

0Ll(τ), j = 1, . . . ,m.

and for τj=1

(ii) P (ti + h) = xi + h

m∑l=1

f(ti + τlh, P (ti + τlh))

∫ 1

0Ll(τ)dτ.

Now, define

ajl =

∫ τj

0Ll(τ) (12)

bl =

∫ 1

0Ll(τ) (13)


TU Ilmenau

IRK..An m-stage IRK Algorithm:Given the points τ1, τ2, . . . , τm ∈ [0, 1):

kl = xi + h

m∑j=1

aljf(ti + τjh, kj), l = 1, . . . ,m (14)

xi+1 = xi + h

m∑l=1

blf(ti + τlh, kl) (15)

• If τ1, . . . , τm are known the vector b = (b1, . . . , bm) and the matrixA = (alj) are easy to determine from equations (14) and (15).Butcher Tabeleau:

τ A

b>


TU Ilmenau

The difference in IRK methods lies:• on the selection of the parameters τ1, . . . , τm• on special adjustments to reduce the size of the matrix A.• IRK methods are also known as collocation methods.Some examples:(a) Gauss-Legendre τ1 = 1/2−

√15/10, τ2 = 1/2 and

τ3 = 1/2 +√

15/10 (are roots of the 3rd degree LegendrePolynomila):

1/2−√

15/10 536

29 −

√1515

536 −

√1530

1/2 536

√1524

29

536 −

√1524

1/2 +√

15/10 536 +

√1530

29 +

√1515

536

518

49

518

(b) Radau IIA(c) Radau 5(d) Lobatto IIIA(e) Diagonally implicit RK methods


TU Ilmenau

IRK..I Matlab function: ode23tb is an implementation of an IRKalgorithm.

Example: A continually stirred homoiothermal reactor

A continuously stirred reactor (CSTR) with feed A and product B.Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

• Concentration of A in the reactor: CA• The feed concentration: CAf• Mixture temperature in the CSTR: T• Bulk volume of the mixture in the CSTR V• Density and specific heat constants: ρ, Cp• k0 reaction constant

Constants:k0 = 3.493x107h−1, E = 11843kcal/kmol,(−∆H) = 5960kcal/kmol, ρCp = 500kcal/m3/KUA = 150kcal/h/K,R = 1.987kcal/kmol/K, V =1m3, q = 1m3/h,CAi = 10kmol/m3, Ti = 298K,Tj = 298K


TU Ilmenau

dCAdt

=q

V(CAi − CA) k0exp(−E/RT )CA

dT

dt=

q

V(Ti − T ) +

(−∆H)k0exp(−E/RT )CAρCp

+UAρCpV

(Tj − T )

Objective: Study the dynamics of the CSTR.

xo2 =[6,399];

df=@(t,x)cstr(x);

[t2,x2]=ode15s(df,[0,100],xo2,[]);

subplot(3,1,1);

[t2,x2]=ode23tb(df,[0,100],xo2,[]);

subplot(3,1,2);

plotyy(t2,x2(:,1),t2,x2(:,2));

subplot(3,1,3);

[t2,x2]=ode45(df,[0,100],xo2,[]);

plotyy(t2,x2(:,1),t2,x2(:,2));


TU Ilmenau

IRK...Advantages/Disadvantages

Advantages:• can be used to solve stiff differential equations and DAEs• widely used for the discretization of optimal control problems

Disadvantages:• to guarantee stability of higher-order IRK some of the τl’s need tobe chosen outside [0, 1] (J. C. Butcher)• may require the solution of nonlinear equations implying intensivecomputations• adaptive selection of the step-size h can be resource intensive• multi-step IRK is still a research topic⇒ The method of ”orthogonal collocation on finite elements” is ageneralizations of IRK methods.


TU Ilmenau

Boundary Value Problems

Note that: The solution x∗(t) of an IVP depends on the initial valuest0 and x0. If we change one or both of these values, the solution x∗(t)also changes. This dependence is indicated by writing x∗(t; t0, x0).

In general, a boundary value problems has the form

(BV P ) x = f(t, x), x(t0) = A, x(tf ) = B.

• The function f(t, x) is assumed continuous and differentiable.• The solution x(t) is required to satisfy the two condones x(t0) = Aand x(tf ) = B known as boundary conditions.

Objective: to compute a function x(t) that satisfies the BVP.


TU Ilmenau

Boundary Value Problems (BVPs)Difficulties with BVPS:

A BVP (in contrast to IVPs) can have several solutions (i.e.,there can be several trajectories (paths) to arrive at x(tf ) = Bstarting from x(t0) = A.)


TU Ilmenau

Single-ShootingStrategy: BVPs are solved by solving a sequence of IVPsHence, any one of: Euler, Runge-Kuta, BDF, IRK, etc. can be used inthe solution strategy of BVPs.Single-ShootingIdea of single-shooting:

from the initial position x(t0) shoot with an angle to hit thetarget at x(tf ).


TU Ilmenau

Single-Shooting...

• Question: What is the shooting angle?Note that: the shooting (aiming) angle is related with the slopex(t0) of the trajectory x(t) (since tanα = x(t0)).• Nevertheless, x(t0) is not given (unknown). That is, x(t0) =?• So we make a guess and take

x(t0) = u.

(But, with this guess (trial shooting angle) we may not hit the target).• Now, drop the end-time condition x(tf ) = B and consider the initialvalue problem

x = f(t, x), x(t0) = A, x(t0) = u.


TU Ilmenau

• Use the transformation

x1 = x, x2 = x1 ⇒ x2 = x1 = x

to write the problem as a system of first-order IVPs

x1 = x2, x1(t0) = A (16)

x2 = ft(t, x1) + fx(t, x1)x2, x2(t0) = u. (17)

• Solve this IVP using any one of the methods for IVPs to obtain a

solution x∗(t) =

(x∗1(t)x∗2(t)

).

• If x∗1(t) satisfies the end-point boundary condition at t = tf ; i.e. if

x∗1(tf ) = B

then we are done: x∗1(t) solves the (BVP).


TU Ilmenau

Otherwise,x∗1(tf ) 6= B.

• Our earlier guess is wrong and we have to adjust our shooting(aiming) angle, i.e. take another guess for u.

• Question: How to make a good guess for u?

• We known that the solution x∗(t) depends on the initial values ofthe IVP, so we write x∗(t;u).• The solution of the system of equations

F (u) = x∗1(tf , u)−B = 0

provides a good guess for u. In general, this can be a system ofnonlinear equations.• We use the Newton’s method to solve such system of nonlinearequations.


TU Ilmenau

A Newton Algorithm

Initialize: Set u0 and tolerance εwhile ‖F (u(k))‖ > ε do• Compute the vector F (u(k)) and the Jacobian matrix F ′(u(k)).• Solve the system of linear equations

F ′(u(k))d = −F (u(k))

to obtain dk as a solution.• Set u(k+1) = u(k) + dk

end while

• To solve the system of nonlinear equations we need the values ofF (u(k)) and F ′(u(k)).• The value of F (u(k)) is found as F (uk) = x∗(tf , uk)−B.• It remains to compute the derivative F ′(u(k)).


TU Ilmenau

We us the forward-difference numerical approximation for F ′(uk):

F ′(u(k)) =F (u(k) + ∆)− F (u(k))

∆

for a given ∆. Note that F (u(k) + ∆) = x∗1(tf , u(k) + ∆)−B.

• Since x∗1(tf , u(k) + ∆) is unknown, we need to re-solve the problem

(16) & (17) using x2(t0) = u(k) + ∆

x1 = x2, x1(t0) = A (18)

x2 = ft(t, x1) + fx(t, x1)x2, x2(t0) = u(k) + ∆. (19)


TU Ilmenau

Single Shooting ...A Single-Shooting Algorithm

Select u0, ∆ > 0 and tolerance ε > 0while ‖F (u(k))‖ > ε do• Compute inital value problems (16) & (17) with x2(tf ) = u(k)

and (18) & (19) with x2(tf ) = u(k) + ∆• Compute the vector F (u(k)) and the Jacobian matrix F ′(u(k)):

F ′(u(k)) =F (u(k) + ∆)− F (u(k))

∆• Solve the system of linear equations

F ′(u(k))d = −F (u(k))

to obtain dk as a solution.• Set k ← k + 1• Set u(k) ← u(k) + dk

end whileLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

Single Shooting ... Advantages/Disadvantages

Advantages:• Simple to implement• effective when the interval [t0, tf ] is short.Disadvantages:• for some problems the algorithm can become unstable• requires good initial guesses for the IVPs• the solution strategy is highly influenced by the length of theshooting (distance) interval [t0, tf ]. Hence, large numerber ofiterations may be needed.


TU Ilmenau

Multiple-Shooting Method

• Consider again the problem

x = f(t, x), x(t0) = A, x(tf ) = B. (20)

• In the multiple-shooting method, the interval [t0, tf ] is divided intosubintervals [t0, t1], [t1, t2], . . . , [tm−1, tm] by using time-pointst0 = t1, t2, . . . , tm−1, tm = tf .• On each subinterval tj ≤ t ≤ tj+1, j = 0, 1, . . . ,m− 1, solve theinitial value problems

x = f(t, x), x(tj) = uj , tj ≤ t ≤ tj+1, j = 0, 1, . . . ,m− 1, (21)

to get a solution x(t; tj , uj) .


TU Ilmenau

Multiple-Shooting Method...


TU Ilmenau

Multiple-Shooting Method...

• Test if all of the following equations hold true at the end points ofeach sub-interval

x(t1; t0, u0) = u1 (22)

x(t2; t1, u1) = u2 (23)... (24)

x(tm−1; tm−2, um−2) = um−1 (25)

x(tm; tm−1, um−1) = um = B. (26)

• If yes, we are done. The solution obtained satisfies the (BVP).• Otherwise, make a new and a better guess of the u’s.


TU Ilmenau

• To find a better guess for the u′s, solve the system of (possiblynonlinear) equations:

F1(u) = x(t1; t0, A)− u1 = 0 (27)

F2(u) = x(t2; t1, u1)− u2 = 0 (28)... (29)

Fm−1(u) = x(tm−1; tm−2, um−2)− um−1 = 0 (30)

Fm(u) = x(tm; tm−1, um−1)−B = 0. (31)

• Put all the guess values into a variable vector u as

u =

u1u2...

um−1

• Note that, each of the uj ’s can be themselves vectors.


TU Ilmenau

• Hence, solve the system of equations

F (u) =

F1(u)F2(u)

...Fm(u)

= 0

using the Newton’s Algorithm to find a better guess for u.• For a given u(k), the vector F (u(k)) can be computed usingequations (27)-(31). Thus, we have

F (u(k)) =

F1(u

(k))

F2(u(k))

...

Fm(u(k))

=

x(t1; t0, A)− u(k)1

x(t2; t1, u(k)1 )− u(k)2...

x(tm−1; tm−2, u(k)m−2)− u

(k)m−1

x(tm; tm−1, u(k)m−1)−B


TU Ilmenau

• The elements of the Jacobian matrix F ′(u(k)) can be computedusing finite differences:

∂Fj∂ul

=Fj(u

(k) + ∆el)− F (u(k))

∆

where el is the l-th unit vector in RK andK = length(u1) + . . .+ length(um).• The value of Fj(u

(k) + ∆el) is obtained by solve the IVP

x = f(t, x), x(tj) = u(k)j + ∆elj , tj ≤ t ≤ tj+1, j = 0, . . . ,m− 1.

• Once the values F (u(k)) and F ′(u(k)) are available, solve thesystem of linear equaitons

F ′(u(k))d = −F (u(k))

to obtain a solution d(k), and update the guess vector asu(k+1) = u(k) + d(k).


TU Ilmenau

• In fact, the Jacobian F ′(u(k)) has the special structure

F ′(u(k)) =

−IX2 −I

X3 −I. . .

. . .

Xm−1 −IXm

,

where (the following are sensitivities w.r.t. the initial values)

X2(t2; t1, u1) =∂x(t2; t1, u1)

∂u

∣∣∣∣u=u(k)

(32)

... (33)

Xm−1(tm−1; tm−2, um−2) =∂x(tm−1; tm−2, um−2)

∂u

∣∣∣∣u=u(k)

(34)

Xm(tm; tm−1, um−1) =∂x(tm; tm−1, um−1)

∂u

∣∣∣∣u=u(k)

(35)Lecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

A Multiple-Shooting Algorithm

• Select shooting-points t0 < t1 < . . . , tm = tf , initial guess

(u(0)1 , u

(0)2 , . . . , u

(0)m ), ∆ > 0 and tolerance ε > 0

while ‖F (u(k))‖ > ε do

• Solve the IVPs in equation (21) twice to obtain x(tj+1; tj , u(k)j )

and X(tj+1; tj , u(k)j ), j = 0, 1, . . . ,m− 1

• Construct the vector the vector F (u(k)) and the Jacobianmatrix F ′(u(k)).• Solve the system of linear equations

F ′(u(k))d = −F (u(k))

to obtain dk as a solution.• Set k ← k + 1• Set u(k) ← u(k) + dk

end whileLecture 3 Introduction to Numerical Methods for Differential and Differential Algebraic Equations

TU Ilmenau

Multiple-Shooting ... Remarks

Remark (Parallel Computation)

I The most time-consuming step in the multiple-shooting method isthe computation of the values of the vector F (u) and Jacobian matrixF ′(u) for each Newton iteration.I The special structure of the Jacobian F ′(u) (also of F (u))indicates that:• each j-th block row of the Jacobian can be obtained through thesolution of the IVP only on the j-th shooting interval.⇒ the computation of the Jacobian can be done by solving smallerIVPs in parallel, correspond to the block rows of the Jacobian.

Choosing the shooting points

I The optimal choice of the shooting points t1, t2, . . . , tm−1 can beproblem dependent. Hence, they can be considered as variables to bedetermined through some computational scheme.


TU Ilmenau

Multiple-Shooting ... Advantages/Disadvantages

Advantages:• it is efficient when the time-interval [t0, tf ] too long;• it is easy to parallelize and, hence, applicable to problems of largedimensions;• accuracy of computations can be improved by considering additionalintermediate points on the interval [tj , tj+1];• it is applicable to solve stiff differential equations and DAEsDisadvantage:• For large-number of sub-intervals it can be computationallyintensive;• The Newton-step may require the solution of a large set of linearequations requiring special linear-algebra strategies,• The choice of the shooting time points t1, t2, . . . , tm−1 has animpact on the quality of the solution.


TU Ilmenau

References and Toolboxes

References:• U. M. Ascherm, L. R. Petzold: Computer methods for ordinarydifferential equations and differential algebraic equations. SISM 1998.• L. F. Shampine and M. W. Reichelt: The Matlab ODE Suite, SIAMJ. Sci. Comput., V. 18, pp. 1-22, 1997.• R. Ashino, M. Nagase: Behind and beyond the Matlab ODEToolbox. Comput. Math. Appl. V. 40, pp. 491 – 512, 2000.• L. F. Shampine, J. Kierzenak:. Solving boundary value problems forordinary differential equations in Matlab bvp4c.• W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery:Numerical recipes in C++ (3r ed.). Cambridge University Press, 2007.


TU Ilmenau

References and Toolboxes...

Toolboxes:

The Matlab ODE-Toolbox found under the MATrix LABoratory(MATLAB) software - The Math Works.

SUite of Nonlinear and DIfferential/ALgebraic equation Solvers(Sundials) :https://computation.llnl.gov/casc/sundials/main.html

Modelica : https://modelica.org/

Scialb: http://www.scilab.org/

EMSO (Environment for Modelling, Simulation andOptimisation): http://www.vrtech.com.br/rps/emso.html


TU Ilmenau

lecture 3 introduction to numerical methods for differential and

Documents