theory and applications of optimal control problems … · theory and applications of optimal...

Theory and Applications of Optimal ControlProblems with Time-Delays

Helmut Maurer

University of MunsterInstitute of Computational and Applied Mathematics

South Pacific Optimization Meeting (SPOM)

Newcastle, CARMA, 9–12 February 2013

Helmut Maurer [2mm] University of Munster Institute of Computational and Applied Mathematics [-4mm]Optimal Control Problems with Time-Delays

EXPECT DELAYS


Challenges for delayed optimal control problems

Theory and Numerics for non-delayed optimal control problemswith control and state constraints are rather complete:

1 Necessary and sufficient conditions,

2 Stability and sensitivity analysis,

3 Numerical methods:Boundary value methods, Discretization and NLP,Semismooth Newton methods,

4 Real-time control techniques for perturbed extremals.

Challenge: establish similar theoretical and numerical methodsfor delayed (retarded) optimal control problems.


Book on SSC for regular and bang-bang controls

This book is devoted to the theory and applications of second-order necessary and sufficient optimality conditions in the calculus of variations and optimal control. The authors develop theory for a control problem with ordinary differential equations subject to boundary conditions of both the equality and inequality type and for mixed state-control constraints of the equality type. The book is distinctive in that

• necessary and sufficient conditions are given in the form of no-gap conditions,• the theory covers broken extremals where the control has finitely many points

of discontinuity, and• a number of numerical examples in various application areas are fully solved.

This book is suitable for researchers in calculus of variations and optimal control and researchers and engineers in optimal control applications in mechanics; mechatronics; physics; chemical, electrical, and biological engineering; and economics.

Nikolai P. Osmolovskii is a Professor in the Department of Informatics and Applied Mathematics, Moscow State Civil Engineering University; the Institute of Mathematics and Physics, University of Siedlce, Poland; the Systems Research Institute, Polish Academy of Science; the University of Technology and Humanities in Radom, Poland;

and of the Faculty of Mechanics and Mathematics, Moscow State University. He was an Invited Professor in the Department of Applied Mathematics, University of Bayreuth, Germany (2000), and at the Centre de Mathématiques Appliquées, École Polytechnique, France (2007). His fields of research are functional analysis, calculus of variations, and optimal control theory. He has written fifty papers and four monographs.

Helmut Maurer was a Professor of Applied Mathematics at the Universität Münster, Germany (retired 2010) and has conducted research in Austria, France, Poland, Australia, and the United States. His fields of research in optimal control are control and state constraints, numerical methods, second-order sufficient conditions, sensitivity analysis, real-time control techniques, and various applications in mechanics, mechatronics, physics, biomedical and chemical engineering, and economics.

For more information about SIAM books, journals, conferences, memberships, or activities, contact:

Society for Industrial and Applied Mathematics 3600 Market Street, 6th Floor

Philadelphia, PA 19104-2688 USA +1-215-382-9800 • Fax +1-215-386-7999

[email protected] • www.siam.org

DC24

DC24

Nikolai P. O

smolovskii

Helm

ut Maurer

ISBN 978-1-611972-35-1

Applications to Regular and Bang-Bang ControlSecond-Order Necessary and Sufficient Optimality Conditions in Calculus of Variations and Optimal Control

Applications to R

egular and Bang-B

ang Control

Second-Order N

ecessary and Sufficient Optim

ality C

onditions in Calculus of V

ariations and Optim

al Control

Nikolai P. OsmolovskiiHelmut Maurer

N. P

. Osm

olov

skii

H. M

aure

r

DC24_Osmolovskii-Maurer_cover.indd 1 10/2/2012 9:58:56 AM


Overview

1 Formulation of optimal control problems withstate and control delays.

2 Example: Two-stage continuous stirred tankreactor (CSTR).

3 Minimum Principle.

4 NLP methods: discretize and optimize.

5 Optimal control of the innate immune response.


Delayed Optimal Control Problem with State Constraints

State x(t) ∈ Rn, Control u(t) ∈ Rm, Delays dx , du ≥ 0.

Dynamics and Boundary Conditions

x(t) = f (x(t), x(t − dx), u(t), u(t − du)), a.e. t ∈ [0, tf ] ,

x(t) = x0(t), t ∈ [−dx , 0] ,

u(t) = u0(t), t ∈ [−du, 0),

ψ(x(tf )) = 0

Control and State Constraints

u(t) ∈ U ⊂ Rm, S(x(t)) ≤ 0, t ∈ [0, tf ] (S : Rn → Rk) .

Minimize

J(u, x) = g(x(tf )) +

∫ tf

0f0(x(t), x(t − dx), u(t), u(t − du)) dt

All functions are assumed to be sufficiently smooth.Helmut Maurer [2mm] University of Munster Institute of Computational and Applied Mathematics [-4mm]Optimal Control Problems with Time-Delays

Two-Stage Continuous Stirred Tank Reactor (CSTR)

Time delays are caused by transport between the two tanks.


Two Stage CSTR

Dadebo S. and Luus R. Optimal control of time-delay systemsby dynamic programming, Optimal Control Applications andMethods 13, pp. 29–41 (1992).

A chemical reaction A⇒ B is processed in two tanks.

State and control variables:

Tank 1 : x1(t) : (scaled) concentration

x2(t) : (scaled) temperature

u1(t) : temperature control

Tank 2 : x3(t) : (scaled) concentration

x4(t) : (scaled) temperature

u2(t) : temperature control


Dynamics of the Two-Stage CSTR

Reaction term in Tank 1 : R1(x1, x2) = (x1 + 0.5) exp(

25x21+x2

)Reaction term in Tank 2 : R2(x3, x4) = (x3 + 0.25) exp

(25x41+x4

)Dynamics:

x1(t) =−0.5− x1(t)− R1(t),

x2(t) =−(x2(t) + 0.25)− u1(t)(x2(t) + 0.25) + R1(t),

x3(t) = x1(t − d)− x3(t)− R2(t) + 0.25,

x4(t) = x2(t − d)− 2x4(t)− u2(t)(x4(t) + 0.25) + R2(t)− 0.25.

Initial conditions:

x1(t) = 0.15, x2(t) = −0.03, −d ≤ t ≤ 0,

x3(0) = 0.1, x4(0) = 0.

Delays d = 0.1, d = 0.2, d = 0.4 in the state variables x1, x2 .


Optimal control problem for the Two-Stage CSTR

Minimizetf∫0

( x21 + x22 + x23 + x24 + 0.1u21 + 0.1u22 ) dt (tf = 2) .

Hamiltonian with yk(t) = xk(t − d), k = 1, 2 :

H(x , y , λ, u) = f0(x , u) + λ1x1+λ2(−(x2 + 0.25)− u1(x2 + 0.25) + R1(x1, x2) )

+λ3(y1 − x3 − R2(x3, x4) + 0.25)

+λ4(y2 − 2x4 − u2(x4 + 0.25) + R2(x3, x4) + 0.25)

Advanced adjoint equations:

λ1(t) = −Hx1(t)− χ [ 0,tf−d ] λ3(t + d),

λ2(t) = −Hx2(t)− χ [ 0,tf−d ] λ4(t + d),

λk(t) = −Hxk (t) (k = 3, 4).

The minimum condition yields Hu = 0 and thus

u1 = 5λ2(x2 + 0.25), u2 = 5λ4(x4 + 0.25).


Two-Stage CSTR with free x(tf ) : x1, x2, x3, x4

-0.02 0

0.02 0.04

0.06 0.08

0.1 0.12

0.14 0.16

0 0.5 1 1.5 2

concentration x1

d=0.1d=0.2d=0.4

-0.03

-0.02

-0.01

0

0.01

0.02

0 0.5 1 1.5 2

temperature x2

d=0.1d=0.2d=0.4

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0 0.5 1 1.5 2

concentration x3

d=0.1d=0.2d=0.4

0 0.005

0.01 0.015

0.02 0.025

0.03 0.035

0.04 0.045

0 0.5 1 1.5 2

temperature x4

d=0.1d=0.2d=0.4

Delays d = 0.1, d = 0.2, d = 0.4.


Two-Stage CSTR with free x(tf ) : u1, u2, λ1, λ2

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u1

d=0.1d=0.2d=0.4

0

0.05

0.1

0.15

0.2

0.25

0.3

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u2

d=0.1d=0.2d=0.4

-0.02 0

0.02 0.04

0.06 0.08

0.1 0.12

0.14 0.16

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

adjoint variable λ1

d=0.1d=0.2d=0.4

-0.4

-0.3

-0.2

-0.1

0

0.1

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2


d=0.1d=0.2d=0.4

Delays d = 0.1, d = 0.2, d = 0.4.


Two-Stage CSTR with x(tf ) = 0 : x1, x2, x3, x4

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0 0.5 1 1.5 2

concentration x1

d=0.1d=0.2d=0.4

-0.03

-0.02

-0.01

0

0.01

0.02

0 0.5 1 1.5 2

temperature x2

d=0.1d=0.2d=0.4

-0.02

0

0.02

0.04

0.06

0.08

0.1

0 0.5 1 1.5 2

concentration x3

d=0.1d=0.2d=0.4

0

0.005

0.01

0.015

0.02

0.025

0.03

0 0.5 1 1.5 2

temperature x4

d=0.1d=0.2d=0.4

Delays d = 0.1, d = 0.2, d = 0.4.


Two-Stage CSTR with x(tf ) = 0 : u1, u2, λ1, λ2

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u1

d=0.1d=0.2d=0.4

0

0.05

0.1

0.15

0.2

0.25

0.3

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u2

d=0.1d=0.2d=0.4

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2


d=0.1d=0.2d=0.4

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2


d=0.1d=0.2d=0.4

Delays d = 0.1, d = 0.2, d = 0.4.


Two-Stage CSTR with x(tf ) = 0 and x4(t) ≤ 0.01

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0 0.5 1 1.5 2

temperature x4

d=0.1d=0.2d=0.4

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u1

d=0.1d=0.2d=0.4

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 0.5 1 1.5 2

multiplier µ for x4 <= 0.01

d=0.1d=0.2d=0.4

0.05

0.1

0.15

0.2

0.25

0.3

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

control u2

d=0.1d=0.2d=0.4

Delays d = 0.1, d = 0.2, d = 0.4.


Delayed Optimal Control Problem with State Constraints

State x(t) ∈ Rn, Control u(t) ∈ Rm, Delays dx , du ≥ 0.

Dynamics and Boundary Conditions

x(t) = f (x(t), x(t − dx), u(t), u(t − du)), a.e. t ∈ [0, tf ] ,

x(t) = x0(t), t ∈ [−dx , 0] ,

u(t) = u0(t), t ∈ [−du, 0),

ψ(x(tf )) = 0

Control and State Constraints

u(t) ∈ U ⊂ Rm, S(x(t)) ≤ 0, t ∈ [0, tf ] (S : Rn → Rk) .

Minimize

J(u, x) = g(x(tf )) +

∫ tf

0f0(x(t), x(t − dx), u(t), u(t − du)) dt

All functions are assumed to be sufficiently smooth.Helmut Maurer [2mm] University of Munster Institute of Computational and Applied Mathematics [-4mm]Optimal Control Problems with Time-Delays

Literature on optimal control with time-delays

State delays and pure control constraints:

Kharatishvili (1961), Oguztoreli (1966), Banks (1968),Halanay (1968), Soliman, Ray (1970, chemical engineering),Warga (1972, abstract theory, optimization in Banach spaces),Guinn (1976, transform delayed problems to standard problems),Colonius, Hinrichsen (1978), Clarke, Wolenski (1991),Dadebo, Luus (1992), Mordukhovich, Wang (2003–).

State delays and pure state constraints: Angell, Kirsch (1990).

State and control delays and mixed control–state constraints:

Gollmann, Kern, Maurer (OCAM 2009),

Gollmann, Maurer: ”Theory and applications of optimal controlproblems with multiple time-delays,” to appear in JIMO 2013.


Optimal control problems with state constraints

Use the transformation method of Guinn (1976) and transform anoptimal control problem with delays and state constraints to astandard non–delayed optimal control problem with stateconstraints. Then apply the

necessary conditions for non-delayed problems:

Jacobson, Lele, Speyer (1975): KKT conditions in Banachspaces.

Maurer (1979) : Regularity of multipliers for state constraints.

Hartl, Sethi, Thomsen (SIAM Review 1995): Survey onMaximum Principles.

Vinter (2000): (Nonsmooth) Optimal Control


Hamiltonian

Hamiltonian (Pontryagin) Function

H(x , y , λ, u, v) :=λ0f0(t, x , y , u, v) + λf (t, x , y , u, v)

y variable with y(t) = x(t − dx)

v variable with v(t) = u(t − du)

λ ∈ Rn, λ0 ∈ R adjoint (costate) variable

Let (u, x) ∈ L∞([0, tf ],Rm)×W1,∞([0, tf ],Rn) be a

locally optimal pair of functions. Then there exist

an adjoint function λ ∈ BV([0, tf ],Rn) and λ0 ≥ 0,

a multiplier ρ ∈ Rq (associated with terminal conditions),

a multiplier function (measure) µ ∈ BV([0, tf ],Rk),

such that the following conditions are satisfied for a.e. t ∈ [0, tf ] :Helmut Maurer [2mm] University of Munster Institute of Computational and Applied Mathematics [-4mm]Optimal Control Problems with Time-Delays

Minimum Principle

(i) Advanced adjoint equation and transversality condition:

λ(t) =tf∫t

(Hx(s) + χ[0,tf−dx ](t)Hy (s + dx) ) ds +tf∫tSx(x(s)) dµ(s)

+ (λ0g + ρψ)x(x(tf )) ( if S(x(tf )) < 0 ),

where Hx(t) and Hy (t + dx) denote evaluations along the optimaltrajectory and χ[0,tf−dx ] is the characteristic function.

(ii) Minimum Condition:

H(t) + χ[0,tf−du ](t)H(t + du)

= min w∈U [ H(x(t), y(t), λ(t),w , v(t))

+χ[0,tf−du ](t)H(t + du)H(x(t + du), y(t), λ(t + du), u(t + du),w) ]

(iii) Multiplier condition and complementarity condition:

dµ(t) ≥ 0,

tf∫0

S(x(t)) dµ(t) = 0


Regularity conditions for dµ(t) = η(t)dt if du = 0

Boundary arc : S(x(t)) = 0 for t1 ≤ t ≤ t2.

Assumption : u(t) ∈ int(U) for t1 < t < t2 .

Under certain regularity conditions we have dµ(t) = η(t) dtwith a smooth multiplier η(t) for all t1 < t < t2 .

Adjoint equation and jump conditions

λ(t) = −Hx(t)− χ[0,tf−dx ]Hy (t + dx)− η(t)Sx(x(t))

λ(tk+) = λ(tk−)− νkSx(x(tk)) , νk ≥ 0

at each contact or junction time tk , νk = µ(tk+)− µ(tk−)

Minimum condition on the boundary

Hu(t) = 0 .

This condition allows to compute the multiplier η = η(x , λ).


Model of the immune response

Dynamic model of the immune response:

Asachenko A, Marchuk G, Mohler R, Zuev S,Disease Dynamics, Birkhauser, Boston, 1994.

Optimal control:

Stengel RF, Ghigliazza R, Kulkarni N, Laplace O,Optimal control of innate immune response,Optimal Control Applications and Methods 23, 91–104 (2002),

Lisa Poppe, Julia Meskauskas: Diploma theses, UniversitatMunster (2006,2008).

L. Gollmann, H. Maurer: Optimal control problems with multipletime-delays, to appear in JIMO 2013.


Innate Immune Response: state and control variablesState variables:

x1(t) : concentration of pathogen(=concentration of associated antigen)

x2(t) : concentration of plasma cells,which are carriers and producers of antibodies

x3(t) : concentration of antibodies, which kill the pathogen(=concentration of immunoglobulins)

x4(t) : relative characteristic of a damaged organ( 0 = healthy, 1 = dead )

Control variables:

u1(t) : pathogen killer

u2(t) : plasma cell enhancer

u3(t) : antibody enhancer

u4(t) : organ healing factor


Generic dynamical model of the immune response

x1(t) = (1− x3(t))x1(t)− u1(t),

x2(t) = 3A(x4(t))x1(t − d)x3(t − d)− (x2(t)− 2) + u2(t) ,

x3(t) = x2(t)− (1.5 + 0.5x1(t))x3(t) + u3(t) ,

x4(t) = x1(t)− x4(t)− u4(t) .

Immune deficiency function triggered by target organ damage

A(x4) =

{cos(πx4) , 0 ≤ x4 ≤ 0.50 0.5 ≤ x4

}.

For 0.5 ≤ x4(t) the production of plasma cells stops.

State delay d ≥ 0 in variables x1 and x3

Initial conditions (d = 0) : x2(0) = 2, x3(0) = 4/3, x4(0) = 0

Case 1 : x1(0) = 1.5, decay, requires no therapy (control)Case 2 : x1(0) = 2.0, slower decay, requires no therapyCase 3 : x1(0) = 3.0, diverges without control (lethal case)


Optimal control model: cost functional

State x = (x1, x2, x3, x4) ∈ R4, Control u = (u1, u2, u3, u4) ∈ R4

L2-functional quadratic in control: Stengel et al.

Minimize J2(x , u) = x1(tf )2 + x4(tf )2

+tf∫0

( x21 + x24 + u21 + u22 + u23 + u24 ) dt

L1-functional linear in control

Minimize J1(x , u) = x1(tf )2 + x4(tf )2

+tf∫0

( x21 + x24 + u1 + u2 + u3 + u4 ) dt

Control constraints: 0 ≤ ui (t) ≤ umax, i = 1, .., 4

Final time: tf = 10Helmut Maurer [2mm] University of Munster Institute of Computational and Applied Mathematics [-4mm]Optimal Control Problems with Time-Delays

L2–functional, d = 0 : optimal state and control variables

State variables x1, x2, x3, x4 and optimal controls u1, u2, u3, u4 :

second-order sufficient conditions via matrix Riccati equation


L2–functional, d = 0 : state constraint x4(t) ≤ 0.2

State and control variables for state constraint x4(t) ≤ 0.2 .Boundary arc x4(t) ≡ 0.2 for t1 = 0.398 ≤ t ≤ t2 = 1.35


L2–functional, multiplier η(t) for constraint x4(t) ≤ 0.2

Compute multiplier η as function of (x , λ) :

η(x , λ) = λ2 3π sin(πx4)x1x3 − λ1 + 2λ4 − 2x3x1 + 2x4

Scaled multiplier 0.1 η(t) and boundary arc x4(t) = 0.2


L2–functional, delay d > 0 , constraint x4(t) ≤ α

Dynamics with state delay d > 0

x1(t) = (1− x3(t))x1(t)− u1(t),

x2(t) = 3 cos(πx4) x1(t − d)x3(t − d)− (x2(t)− 2) + u2(t) ,

x3(t) = x2(t)− (1.5 + 0.5x1(t))x3(t) + u3(t) ,

x4(t) = x1(t)− x4(t)− u4(t)

x4(t) ≤ α ≤ 0.5

Initial conditions

x1(t) = 0 , −d ≤ t < 0, x1(0) = 3,x3(t) = 4/3 , −d ≤ t ≤ 0,x2(0) = 2, x4(0) = 0.


L2–functional : delay d = 1 and x4(t) ≤ 0.2

State variables for d = 0 and d = 1


L2–functional : delay d = 1 and x4(t) ≤ 0.2

Optimal controls for d = 0 and d = 1


L2–functional, d = 1 : multiplier η(t) for x4(t) ≤ 0.2

Compute multiplier η as function of (x , λ) :

η(x , y , λ) = λ2 3π sin(πx4)y1y3 − λ1 + 2λ4 − 2x3x1 + 2x4

Scaled multiplier 0.1 η(t) and boundary arc x4(t) = 0.2 ;

η(t) is discontinuous at t = d = 1


L1–functional and state delay d ≥ 0

Minimize

J1(x , u) = p11x1(tf )2 + p44x4(tf )2

+tf∫0

( p11x21 + p44x

24 + q1 u1 + q2 u2 + q3 u3 + q4 u4 ) dt

Dynamics with delay d and control constraints

x1(t) = (1− x3(t))x1(t)− u1(t),

x2(t) = 3A(x4(t))x1(t − r)x3(t − r)− (x2(t)− 2) + u2(t) ,

x3(t) = x2(t)− (1.5 + 0.5x1(t))x3(t) + u3(t) ,

x4(t) = x1(t)− x4(t)− u4(t) ,

0 ≤ ui (t) ≤ umax , 0 ≤ t ≤ tf (i = 1, .., 4)


L1–functional : non-delayed problem d = 0 : umax = 2


L1–functional : delayed problem d = 1 : umax = 2


L1–functional : non-delayed, time–optimal control forx1(tf ) = x4(tf ) = 0, x3(tf ) = 4/3

umax = 1: minimal time tf = 2.2151, singular arc for u4(t)


Further applications and future work

1 Optimal oil extraction and exploration : state delay (Bruns,Maurer, Semmler)

2 Biomedical applications: optimal protocols in cancertreatment and immunology

3 Vintage control problems

4 Delayed control problems with free final time

5 Optimal control problems with state-dependent delays

6 Verifiable sufficient conditions


Thank you for your attention !


theory and applications of optimal control problems … · theory and applications of optimal...

Documents