thesis tuan

VIETNAM NATIONAL UNIVERSITY, HANOI

HANOI UNIVERSITY OF SCIENCE

FACULTY OF MATHEMATICS, MECHANICS AND INFORMATICS

Tuan Anh Nguyen

AN OPTIMAL CONTROL APPROACH

FOR THE TREATMENT

OF HEPATITIS C PATIENTS

Undergraduate Thesis

Undergraduate Honors Program in Mathematics

Hanoi - December, 2013

VIETNAM NATIONAL UNIVERSITY, HANOI

HANOI UNIVERSITY OF SCIENCE

FACULTY OF MATHEMATICS, MECHANICS AND INFORMATICS

Tuan Anh Nguyen

AN OPTIMAL CONTROL APPROACH

FOR THE TREATMENT

OF HEPATITIS C PATIENTS

Undergraduate Thesis

Undergraduate Honors Program in Mathematics

Thesis advisor:Prof. Hien Trong Tran

North Carolina State University

Hanoi - December, 2013

ACKNOWLEDGEMENTS

From bottom of my heart, I would like to thank my advisor, Professor Hien T.Tran for his guidance, encouragement and support during the time of this project.Truthly, there were so many difficulties associated with doing research over two con-tinents. Nonetheless, his enthusiasm and patience have contributed to the steadyprogress and eventually the completion of this research.

Moreover, it is also my great pleasure to express my gratitude to all faculties andstaff members of Faculty of Mathematics, Mechanics and Informatics, Hanoi Uni-versity of Science for their contribution and kind help to my mathematical educa-tion.

Furthermore, I also would like to thank all my trusted friends as well as my class-mates of K54 Advanced and Honors Mathematics class for their wonderful friend-ship and fervent help. We had joy, we had fun, we had season in the sun. Bestwishes for us to achieve our dreams after graduation!

Finally, deep inside myself, I would like to thank my parents for their love andbelief in anything I pursue.

iii

INTRODUCTION

Viral diseases are major causes of human morbidity and mortality worldwide. Globalhealth promotion, therefore, requires the establishment of evidence-based interven-tions at the individual and population levels aiming at the prevention, treatmentand control of viral infections. The design, implementation and impact of suchactions depend on a broad understanding of not only host-pathogen interactionsin vivo, but also of the intertwining environmental, social and cultural cofactorsthat may contribute to viral disease development. Multidisciplinary research is thusneeded for an effective response to be achieved against these diseases.

Hepatitis C virus (HCV) infection stands out among human viral infections for itscurrent importance in global public health. HCV is spread primarily by blood-to-blood contact and affects the liver. The infection is often asymptomatic, but chronicinfection can lead to scarring of the liver and ultimately to cirrhosis. Up to 20%of those with cirrhosis will develop liver cancer.11 The World Health Organization(WHO) estimates that 180 million people worldwide are affected by the hepatitis Cvirus. It is globally the most common cause of liver cirrhosis, hepatocellular carci-noma, end-stage liver disease and liver transplantation.

The current standard treatment for hepatitis C is a 48-week program of weekly treat-ments with pegylated interferon-alpha and ribavirin that is reviewed after 12 weeksand stopped if the patient is not responding. The results of this approach are mixed.Some patients are cured (sustained virology response (SVR)) , some appear to becured as the end of treatment but the virus returns once the treatment has stopped(relapse) while for others the treatment is not effective (partial virology response(PVR)). This highlights the shortcomings of the current approach of the pharma-ceutical industry that could be described as product focused. New products arebrought to the market and used as a standard treatment for all patients. However,there is a growing realization in the industry that there is a need to move to a patientfocused approach in which the treatment is tailored to the individual patient. Thearea of personalized medicine has, indeed, been the focus of intense current interestand vigorous research efforts. In essence, personalized medicine seeks to make opti-mal treatment decisions for an individual patient based on all information available

iv

for that patient, including not only genetic and genomic characteristics but also alldemographic, physiological, and other clinical factors, thereby allowing the righttreatment for the right patient.

A precondition for getting better diagnostic procedures and follow-up treatmentplans for HCV patients is to obtain better knowledge of the pathophysiologicalstates. One approach to achieve this objective is via patient specific mathematicalmodeling where knowledge of viral kinetic, pharmacokinetic of the drugs, and var-ious aspects of immune responses is integrated to provide a more comprehensivepicture of the biology underlying changes in HCV RNA during therapy. However,for the mathematical models to be clinically relevant, they must be developed inclose collaboration with medical doctors and be validated with clinical and labo-ratory data. In addition, analysis can be performed on the patient specific mathe-matical models to determine which data gives more information about the systemdynamics. Such information can be used for designing better diagnostic and treat-ment techniques, which in turn have potential to be used for analyzing effects of agiven treatment for a given patient. Finally, mathematical models can also be usedto predict what elements or biological mechanisms should be studied in more detailand what adjustments of behavior or medical treatment can bring forward positiveclinical results.

The primary objective of HCV therapy is permanent eradication of the virus. Opti-mal control theory, a mathematical theory for modifying the behaviour of nonlineardynamical through control of system inputs, is a promising approach to suggestingadaptive HCV treatment strategies. In this research, we will study the feasibility ofusing optimal control theory to develop control theoretic methods for personalizedtreatment of HCV patients. The organization of this thesis is as follows:

Chapter 1 reviews some basic definitions, notations, and theories from the calculusof variations that are used frequently in the subsequent chapters. More precisely,this introduces a variational approach in finding an optimal trajectory for a givenfunctional.

Chapter 2 applies the variational approach that was introduced in the previouschapter to the problem of finding optimal solutions to the optimal control problems.There is a very important class of optimal control problems that will be discussedhere is the linear regulator problems.

Chapter 3 contains the main results of this project. Here, we consider the applicationof optimal control theory to obtain adaptive HCV treatment strategies. A potentialproblem to consider in this area is whether there exists a drug treatment schedule

v

that can sustain a low viral load and a healthy liver while minimizing the amountof drugs used. In particular, it will be illustrated how optimal control methodologycan produce a drug dosing strategy and how this treatment strategy has featuresof Structured Treatment Interruption (STI). In STI type of treatment, the drug cy-cles from on treatment to off treatment (drug holiday). The STI type treatment hasbeen considered as an alternative type of treatment for human immunodeficiencyvirus (HIV). For HIV, there are several reasons to consider an STI type of treatmentstrategy. First, this type of treatment could potentially stimulate the bodys ownvirus specific immune response. An additional argument for treatment interruptionis with individuals with drug resistance strains. A treatment interruption has thepotential to allow the drug-sensitive strains to out compete the resistance strains.5

vi

Contents

1 The Calculus of Variations 11.1 Fundamental concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 The Fundamental Theorem of the Calculus of Variations . . . . . . . 51.3 Functionals of a single function . . . . . . . . . . . . . . . . . . . . . . 6

1.3.1 The simplest variational problem . . . . . . . . . . . . . . . . . 61.3.2 Final-time specified, final-state free . . . . . . . . . . . . . . . 101.3.3 Final-time free, final-state specified . . . . . . . . . . . . . . . 111.3.4 Problem with both the final-time and final-state free . . . . . . 14

2 An Introduction to Optimal Control Theory 182.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.1.1 Mathematical Model . . . . . . . . . . . . . . . . . . . . . . . . 182.1.2 The performance measure . . . . . . . . . . . . . . . . . . . . . 202.1.3 The Optimal Control Problem . . . . . . . . . . . . . . . . . . . 202.1.4 Forms of the Optimal Control . . . . . . . . . . . . . . . . . . . 212.1.5 System Classifications . . . . . . . . . . . . . . . . . . . . . . . 222.1.6 Output Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.2 Necessary Conditions for Optimal Control . . . . . . . . . . . . . . . 232.3 Linear Regulator Problems . . . . . . . . . . . . . . . . . . . . . . . . . 272.4 Bounded Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.5 Numerical Solutions to the Optimal Control Problems . . . . . . . . . 30

3 An Optimal Control Approach for Treatment of Hepatitis C Patients 363.1 Mathematical Modeling of HCV Infection . . . . . . . . . . . . . . . . 363.2 An Optimal Treatment Formulation Problem . . . . . . . . . . . . . . 40

Outline 52

vii

Chapter 1

THE CALCULUS OF VARIATIONS

Calculus of variations is a branch of mathematics that is extremely useful in solvingoptimization problems. Precisely speaking, it is a part of mathematical analysis thatdeals with maximizing or minimizing functionals, which are mappings functionsfrom a certain class to real numbers. Beside its mathematical applications and itslinks with other branches of mathematics, such as geometry or differential equa-tions, it is widely used in physics, engineering, economics and biology. In this chap-ter, the foundation of the calculus of variations will be briefly introduced. The refer-ence of this chapter is Chapter 4 of the book: Optimal Control Theory: An Introductionby Donald E. Kirk.8

1.1 Fundamental concepts

To begin we define the vector x(t) Rn, that is, x(t) has n components:

x (t) ,

x1 (t)x2 (t)

xn (t)

.

In control theory, x is called the state vector. For the remainder of this thesis, we willuse x to denote vector functions and x to denote scalar functions.

Next, some well-known definitions should be recalled:

Definition 1.1 (see [8, page 109]). A functional J is a rule of correspondence that assigns

1

to each function x in a certain class a unique real number:

J () : Rx 7 J (x) ;

Here, is said to be the domain of the functional J. Notice that functions in might be defined in multi-dimensional vector spaces, R2, R3, Rn or even Cn. Theset of real numbers associated with the functions in is called the range of thegiven functional. It should be noted that since the domain of a functional is a classof functions, intuitively it might say that a functional is a function of function.

Example 1.1 Suppose that x is a continuous function of t defined in the interval[t0, t f

].

Let us consider

J (x) =

t ft0

|x(t)|dt.

Obviously, J assigns the area under the curve |x(t)|. Hence, J() is a functional.Definition 1.2 (see [8, page 111]) J is a linear functional of x if and only if it satisfies theprinciple of homogeneity

J (x) = J (x)

for all x and for all real number such that x , and the principle of additivity

J(x(1) + x(2)) = J(x(1)) + J(x(2))

for all x(1), x(2) and x(1) + x(2) in .

Definition 1.3 (see [8, page 113]) The norm of a function is a rule of correspondencethat assigns to each function x , defined for t [t0, t f ], a real number. The norm of x,denoted by x, satisfies the following properties:

1. x 0 and x = 0 if and only if x(t) = 0 for all t [t0, t f ].2. x = || x for all real numbers and function x .

3.x(1) + x(2) x(1)+ x(2) for all x(1) and x(2) .

Since all norms are equivalent in any finite dimensional vector space, one can asso-ciate the norm with functions with the max-norm

x = x = max{tdom(xi)} {|x1(t)|, ..., |xn(t)|} .

2

Definition 1.4 (see [8, page 114]) If x and (x + x) are functions for which the functionalJ is defined, then the increment of J, denoted by J, is the quantity

J , J (x + x) J (x) . (1.1)In order to see it more explicitly, J is often written as J (x, x) to illustrate that theincrement depends on the functions x and x. Here x is called the variation of the functionx.

Definition 1.5 (The Variation of a Functional) (see [8, page 117]) In the case that theincrement of a given functional J can be written as

J (x, x) = J (x, x) + g (x, x) x, (1.2)where J(x, x) is linear in x and its derivatives, and

limx0

{g (x, x)} = 0,

then J is said to be differentiable on x and J (x, x) is called the variation of J evaluated forthe function x.

Example 1.2 (see [8, page 117]) Let x = x(t) be a continuous function defined for t [0, 1]. Determine the variation of the functional

J(x) = J(x(t)) =1

0

[x2(t) + 2x(t)

]dt. (1.3)

First of all, calculate the increment of J and combine its components with respect to theorders,

J (x, x) = J (x + x) J (x)

=

10

{[x (t) + x (t)]2 + 2 [x (t) + x (t)]

}dt

10

[x2 (t) + 2x (t)

]dt

=

10

{[2x (t) + 2] x (t) + [x (t)]2

}dt

=

10

{[2x (t) + 2] x (t)}dt +1

0

[x (t)]2dt.

3

Obviously, the first term is linear in term of x(t). Let us consider the second term:

10

[x (t)]2dt =xx

10

[x (t)]2dt = x1

0

[x (t)]2

x dt.

Denote

g (x, x) =1

0

[x (t)]x

2

dt

Here, it is not very hard to see that

10

[x (t)]2

x dt =1

0

|x (t)| . |x (t)|x dt

10

|x (t)|dt 0 as x 0.

Hence,limx0

{g (x, x)} = 0.

Consequently, J (x, x) can be written as the expression (1.2). As a result, the variation ofJ is

J (x, x) =1

0

{[2x (t) + 2] x (t)}dt.

Definition 1.6 (see [8, page 120]) A functional J with domain has a relative extremumat x if there is a sufficient small e > 0 such that for all functions x in which satisfiesx x < e, the increment of J has the same sign. In particular, if

J = J(x) J(x) 0, (1.4)J(x) is a relative minimum. If

J = J(x) J(x) 0, (1.5)J(x) is a relative maximum. If (1.4) satisfies for arbitrary large e R, then J(x) is aglobal, or absolute, minimum. Similarly, If (1.5) satisfies for arbitrary large e, then J(x)is a global, or absolute, maximum. x is called an extremal, and J(x) is referred to as anextremum.

It is clear that global extremum is relative extremum, but the converse is not true ingeneral.

4

1.2 The Fundamental Theorem of the Calculus of Vari-ations

Recall that the fundamental theorem of calculus is used to find extremal values offunctions. That is, a necessary condition is that the derivatives vanish at an extremalpoint (except extrema at the boundaries of closed regions). Similarly, the fundamen-tal theorem of the calculus of variations is that the variation must be zero on an ex-tremal curve (provided that there are no bounds imposed on the curves). We nowformally state the theorem and its proof.

Theorem 1.1 (see [8, page 121]) Let x be a vector function of t in the class , and J (x) bea differentiable functional of x. Assume that the functions in the class are not constrainedby any boundaries. The fundamental theorem of the calculus of variations states that:

If x is an extremal, the variation of J must vanish on x, i.e,

J(x, x) = 0 (1.6)

for all admissible x.

Proof This theorem can be easily proved by contradiction. Indeed, assume that xbe an extremal and J (x, x) 6= 0. Let us show that these assumptions imply thatthe increment J can be made to change sign in a small neighborhood of x. In orderto do that, first of all, compute the increment

J (x, x) = J (x + x) J (x) = J (x, x) + g (x, x) x , (1.7)where g(x, x) 0 as x 0. Hence, x has an e-neighborhood of 0, i.e, x 0 and x(1) < e. Suppose

without losing of generality that

J(

x, x(1))< 0.

Since J is a linear functional of x as well as x(1), the principle of homogeneitygives that

J(

x, x(1))= J

(x, x(1)

)< 0.

5

The signs of J and J are the same for x < e. Then,

J(

x, x(1))< 0.

Next, we consider the variation

x = x(1)

where x(1) has been already defined above. Clearly, x(1) < e implies that x(1) < e. Therefore, the sign of J(x, x(1)) is the same as the sign of

J(x, x(1)). Again using the principle of homogeneity, we obtain:

J(x, x(1)) = J(x, x(1)).Therefore, since J(x, x(1)) < 0, J(x, x(1)) > 0. It implies

J(x, x(1)) > 0.To recapitulate, we have shown that if J(x, x) 6= 0, then in an arbitrarily smallneighborhood of x

J(x, x(1)) < 0and

J(x, x(1)) > 0,thus contradicts the assumption that x be an extremal. Consequently, if x be anextremal, it is necessary that

J(x, x) = 0 (1.8)

for arbitrary x. The assumption that the functions in are not bounded guaranteesthat x(1) and x(1) are both admissible variations.

1.3 Functionals of a single function

In this section we are going to apply the fundamental theorem of the calculus ofvariations to determine extrema of functionals depending on a single function.

1.3.1 The simplest variational problem

Let x = x (t) be a scalar function in the class of all real functions with continuousfirst derivatives. Consider the following problem:

6

Problem 1: (see [8, page 123]) It is desired to find a function x (t) for which the func-tional

J (x) =

t ft0

g (x (t) , x (t) , t)dt (1.9)

has a relative extremum. The notation J (x) means that J is a functional of the func-tion x. The integrand, g (x (t) , x (t) , t), on the other hand, is a function that assignsa real number to the point (x (t) , x (t) , t). It is assumed that the integrand g (, , )has continuous first and second partial derivatives with respect to any argument.Times t0 and t f are fixed, and the end points of the curve are specified as x0 and x f .

We would like to find such a curve (if any exists) that optimizes J (x). The searchbegins by finding a curve that satisfies the fundamental theorem of the calculus ofvariations. Let x be any curve in , and determine the variation J (x, x) from theincrement:

J (x, x) = J (x + x) J (x)

=

t ft0

g (x (t) + x (t) , x (t) + x (t) , t)dtt f

t0

g (x (t) , x (t) , t)dt

=

t ft0

[g (x (t) + x (t) , x (t) + x (t) , t) g (x (t) , x (t) , t)]dt. (1.10)

Note that x (t) = ddt [x (t)] and x (t) =ddt [x(t)].

Expanding the integrand of (1.10) in a Taylor series at the points x (t) and x (t):

J =

t ft0

{g (x (t) , x (t) , t) +

[gx

(x (t) , x (t) , t)]x (t) +

[gx

(x (t) , x (t) , t)]x (t)

+12

[[2gx2

(x (t) , x (t) , t)][x (t)]2 +

[2gx2

(x (t) , x (t) , t)][x (t)]2

+2[2gxx

(x (t) , x (t) , t)]x (t) x (t)

]+o([x (t)]2[x (t)]2

) g (x (t) , x (t) , t)

}dt.

7

Next, extract the terms in J that are linear in x (t) and x (t) to obtain the variation

J (x, x) =

t ft0

{[gx

(x (t) , x (t) , t)]x (t) +

[gx

(x (t) , x (t) , t)]x (t)

}dt.

x(t) and x(t) are related by

x(t) =t

t0

x(t)dt + x(t0).

Hence, the variation of the given functional J(x) can be rewritten as follows:

J(x, x) =[gx(x(t), x(t), t)

]x(t)

t ft0

(1.11)

+

t ft0

{[gx(x(t), x(t), t)

] d

dt

[gx(x(t), x(t), t)

]}x(t)dt.

Since x (t0) and x(t f)

are specified, all admissible curves have to pass through thesepoints. Therefore, x (t0) = x

(t f)= 0. Consequently, the terms outside the in-

tegral vanish. If we now consider an extremal curve, applying the fundamentaltheorem of the calculus of variations yields

J (x, x) = 0 (1.12)

=

t ft0

{[gx

(x (t) , x (t) , t)] d

dt

[gx

(x (t) , x (t) , t)]}

x (t) dt

for any x(t).

Lemma 1.1 (Anh-Tuan Nguyen, 2013) Let g(t) C1 [t0, t f ], i.e, g(t) be a continuous,differentiable functions on

[t0, t f

]. Then, if

t ft0

g(t)h(t)dt = 0

for any h(t) C1 [t0, t f ]. Then g(t) = 0 for any t [t0, t f ]. Simply writing, g 0.8

Proof We can prove by contradiction. Suppose to the contrary that there exists t1 [t0, t f

]so that g(t1) 6= 0. We begin by assuming that t1 6= t0 and t1 6= t f (the case that

t1 is any end point will be considered later) and that g(t1) > 0. Hence, there existsan open e-neighborhood N (t1, e) = (t1 e, t1 + e) of t1 in

[t0, t f

]so that g(t) > 0

for any t N (t1, e). Let us consider the function h(t) defined as follows:

h (t) ={ [t (t1 e)] [t (t1 + e)] i f t N(t1, e)

0 otherwise.

It is obvious thatt f

t0

g (t) h (t)dt = t1+e

t1eg (t) (t t1 + e) (t t1 e)dt > 0

since g(t) and h(t) are positive on N(t1, e). This contradicts with the assumption

thatt ft0

g (t) h (t)dt = 0. Hence,

g 0. (1.13)For cases t1 t0, say g(t1) > 0. There will be an open interval N(t1, e) so thatthe sign of g(t) doesnt change in this interval. Here choose N(t1, e) = (t1, t1 + e).Therefore, let us consider the function h(t) defined as follows:

h (t) ={ [t (t1 + e)] (t t1) i f t N(t1, e)

0 otherwise.

It is obvious thatt f

t0

g (t) h (t)dt = t1+et1

g (t) (t t1 e) (t t1)dt > 0

since g(t) and h(t) are positive on N(t1, e). Hence,

g 0. (1.14)Similar arguments can also be shown for the case t1 t f . This completes the proof.Using the above lemma, we obtain from (1.12),

gx

(x (t) , x (t) , t) ddt

[gx

(x (t) , x (t) , t)]= 0 (1.15)

for all t [t0, t f ]. The above equation is called the Euler-Lagrange equation. Since itinvolves the derivative ddt of a function of x (t) in the second term, the Euler-Lagrangeequation is generally of second order and nonlinear. They are, in general, hard tosolve analytically.

9

1.3.2 Final-time specified, final-state free

Problem 2: (see [8, page 130]) Find a necessary condition for a function to be an ex-tremal for the functional

J (x) =

t ft0

g (x (t) , x (t) , t)dt,

where t0, x (t0) and t f are specified, but x(t f)

is free. To use the fundamental theo-rem of the calculus of variations, we begin by finding the variation as in Problem 1.From (1.11):

J(x, x) =[gx(x(t), x(t), t)

]x(t)

t ft0

(1.16)

+

t ft0

{[gx(x(t), x(t), t)

] d

dt

[gx(x(t), x(t), t)

]}x(t)dt.

Now x (t0) = 0 for all admissible curves since t0 is fixed, but x(t f)

is arbitrarysince x

(t f)

is free.

For an extremal curve x, we know that J(x, x) must be zero. Let us show thatthe integral in (1.16) must be zero on an extremal.

The value of x(t f ) is x f . Now consider a fixed end point problem with the samefunctional, the same initial and final times, and with specified end points x(t0) = x0and x(t f ) = x f that are the same as for the extremal x in the free end point prob-lem. Therefore, x must be a solution of the Euler-Lagrange equation (1.15), and theintegral term must be zero on an extremal. In other words, an extremal for a freeend point problem is also an extremal for the fixed end point problem with the sameend points and the same functional. Hence, regardless of the boundary conditions,the Euler-Lagrange equation must be satisfied.

SinceJ (x, x) = 0,

andgx

(x (t) , x (t) , t) ddt

[gx

(x (t) , x (t) , t)]= 0

10

for all t [t0, t f ], from (1.16) we have[gx(x(t f)

, x(t f)

, t f)]

x(t f)= 0.

But since x(t f ) is free, then x(t f ) is arbitrary. Therefore, it is necessary that

gx(x(t f)

, x(t f)

, t f)= 0. (1.17)

The Euler-Lagrange equation is second order, and (1.17) provides the second re-quired boundary condition. We will call the equation (1.17) to be the natural boundarycondition.

1.3.3 Final-time free, final-state specified

In Problem 2 we considered the situation where x(t f ) was free, but the final time t fwas specified. Let us now investigate problems in which x(t f ) is specified, but t f isfree.

Problem 3 (see [8, page 134]): Find a necessary condition that must be satisfied byan extremal of the functional

J (x) =

t ft0

(x (t) , x (t) , t) dt,

where t0, x (t0) and x(t f)= x f are specified, but t f is free.

This problem is more complicated. First, it is noted that x (t) = [x (t) x (t)]has meaning only in the interval

[t0, t f

]. Hence, x

(t f)= x

(t f) x (t f ).

11

Computing the increment

J =

t f+t ft0

g (x (t) , x (t) , t) dtt f

t0

g (x (t) , x (t) , t) dt

=

t ft0

{g (x (t) , x (t) , t) g (x (t) , x (t) , t)} dt

+

t f+t ft f

g (x (t) , x (t) , t) dt

=

t ft0

{g (x (t) + x (t) , x (t) + x (t) , t) g (x (t) , x (t) , t)} dt

+

t f+t ft f

g (x (t) , x (t) , t) dt. (1.18)

The first integral can be expanded in Taylor series as follows:

J =

t ft0

{[gx

(x (t) , x (t) , t)]x (t) +

[gx

(x (t) , x (t) , t)]x (t)

}dt (1.19)

+ o (x (t) , x (t)) +

t f+t ft f

g (x (t) , x (t) , t) dt.

The second integral can be written as

t f+t ft f

g(x(t), x(t), t)dt =[g(x(t f ), x(t f ), t f )

]t f + o(t f ). (1.20)

12

Integrating by parts the first integral term in (1.19) and substituting (1.20) for thesecond integral term yields

J =[gx(x(t f ), x(t f ), t f )

]x(t f ) +

[g(x(t f ), x(t f ), t f )

]t f (1.21)

+

t ft0

{gx(x(t), x(t), t) d

dt

[gx(x(t), x(t), t)

]}x(t)dt + o(),

where we have also used the fact that x(t0) = 0. Next, we write g(x(t f ), x(t f ), t f )in term of g(x(t f ), x(t f ), t f ) by the expansion

g(x(t f ), x(t f ), t f ) =g(x(t f ), x(t f ), t f ) (1.22)

+

[gx(x(t f ), x(t f ), t f )

]x(t f )

+

[gx(x(t f ), x(t f ), t f )

]x(t f ) + o().

Substituting this expression in (1.21) yields

J =[gx(x(t f ), x(t f ), t f )

]x(t f ) +

[g(x(t f ), x(t f ), t f )

]t f (1.23)

+

t ft0


dt

[gx(x(t), x(t), t)

]}x(t)dt + o().

x(t f ) is neither zero nor free but depends on t f . The variation of J, J, consists ofthe first-order terms in the increment J. Therefore, the dependence of x(t f ) on t fmust be linearly approximated. By inspection of [8, Figure 4-12, page 135], we have

x(t f ) + x(t f )t f.= 0

orx(t f )

.= x(t f )t f . (1.24)

13

Substituting (1.24) into (1.23), and retaining only first-order terms, we have the vari-ation

J(x, x) =0 (1.25)

=

{[gx(x(t f ), x(t f ), t f )

]x(t f ) + g(x(t f ), x(t f ), t f )

}t f

+

t ft0


dt

[gx(x(t), x(t), t)

]}x(t)dt.

Notice that the integral term represents the partial variation of J caused by x(t),t [t0, t f ], and the term involving t f is the partial variation of J caused by thedifference in end points. Together, these partial variations make up the general (ortotal) variation.

As in Problem 2, we argue that the extremal for this free end point problem is alsoan extremal for a particular fixed end point problem. Therefore, the Euler-Lagrangeequation (1.15) must be satisfied, and the integral is zero. t f is arbitrary, so itscoefficient must be zero, and the required boundary condition at t f is

g(x(t f ), x(t f ), t f )[gx(x(t f ), x(t f ), t f )

]x(t f ) = 0. (1.26)

1.3.4 Problem with both the final-time and final-state free

Now we are ready to consider problems having both t f and x(t f)

unspecified. Notsurprisingly, we shall find that the necessary conditions of Problems 2 and Problem 3are included as special cases of the following problem:

Problem 4 (see [8, page 138]): Find a necessary condition that must be satisfied byan extremal for a functional of the form

J (x) =

t ft0

g (x (t) , x (t) , t) dt,

where t0 and x (t0) = x0 are specified, but t f and x(t f)

are free.

It is necessary to distinguish the following notions in order that there will be no con-fusion. Notice that x

(t f)

is the difference in ordinates at t = t f and x f is the differ-ence in ordinates of the end points of the two curves, i.e, x

(t f)=[x(t f) x (t f )],

14

and x f is the difference of the end points of x and x, i.e, x f = x(t f + t f

) x(t f). It is important to keep in mind that x

(t f) 6= x f .

In order to use the fundamental theorem of the calculus of variations, we must firstdetermine the variation by computing the increment. This is accomplished in simi-lar manner as in Problem 3:

J =[gx(x(t f ), x(t f ), t f )

]x(t f ) +

[g(x(t f ), x(t f ), t f

)]t f (1.27)

+

t ft0

{gx

(x(t), x(t), t) ddt

[gx

(x(t), x(t), t)]}

x(t)dt + o().

Next, we must relate x(t f)

to t f and x f . From the construction in [8, page 138]we have

x f.= x(t f ) + x(t f )t f , (1.28)

or

x(t f ) = x f x(t f )t f . (1.29)Substituting this in (1.23) and collecting terms, we obtain as the variation

J (x, x) = 0 =[gx(x(t f)

, x(t f)

, t f)]

x f (1.30)

+

[g(x(t f)

, x(t f)

, t f) [g

x(x(t f)

, x(t f)

, t f)]

x(t f)]

t f

+

t ft0

{gx

(x (t) , x (t) , t) ddt

[gx

(x (t) , x (t) , t)]}

x (t) dt.

As before, we have already proved that the Euler-Lagrange equation must be sat-isfied. Therefore, the integral is zero. Here, there may be a variety of end pointconditions in practice. However, for the moment we are going to consider only twopossibilities:

1. t f and x(t f ) are unrelated. In this case, x f and t f are independent of oneanother and arbitrary, so their coefficients must each be zero. From (1.30), then

gx(x(t f)

, x(t f)

, t f)= 0, (1.31)

15

and

g(x(t f)

, x(t f)

, t f) [g

x(x(t f)

, x(t f)

, t f)]

x(t f)= 0, (1.32)

which together imply that

g(x(t f)

, x(t f)

, t f)= 0. (1.33)

2. t f and x(t f ) are related. For example, the final value of x may be constrainedto lie on a specified moving points, (t), that is

x(t f)=

(t f)

. (1.34)

In this case the difference in end point x f is related to t f by

x f.=

ddt(t f)t f . (1.35)

The geometric interpretation of this relationship is shown in the figure below.The distance a is a linear approximation to x f . That is

a =[

ddt(t f)]

t f .

Substituting (1.35) into (1.30) and collecting terms, it gives[gx(x(t f ), x(t f ), t f

)] [ddt(t f) x (t f )]+ g (x(t f ), x(t f ), t f ) = 0

(1.36)because t f is arbitrary. This equation is said to be the transversality condition.

16

In any one of two cases considered, integrating the Euler-Lagrange equation willgive a solution x(c1, c2, t) where c1 and c2 are some constants. c1, c2 and the un-known value of t f can then be determined from x(c1, c2, t0) = x0, (1.31) and (1.33)if x(t f ) and t f are unrelated, or (1.34) and (1.36) if x(t f ) and t f are related.

So far, we have been considered the calculus of variations with the main approachbeing to identify the variations. This method is very important in solving optimalcontrol problems, which we will discuss in the next chapter.

17

Chapter 2

AN INTRODUCTION TO OPTIMALCONTROL THEORY

2.1 Introduction

The objective of optimal control theory is to determine the control signals that will causea process to satisfy the physical constraints and at the same time minimize (or maximize)some performance criterion. In this chapter, we are going to apply variational methodsthat have been discussed in previous chapter to optimal control problems. First ofall, assuming that the admissible controls are not bounded, then we derive neces-sary conditions for optimal controls. These necessary conditions are then employedto find the optimal control law for the important class of problems known as the lin-ear regulator problems. We then consider the case in which the admissible controlsare bounded. This is an important case in most applications including the optimalcontrol of hepatitis C virus that is considered in this research. Finally, optimal con-trol problems are in general nonlinear and therefore, generally do not have analyticsolutions (except in the simplest cases such as the linear regulator problems). Tothis end, we will describe a general open-source MATLAB optimal control softwarecalled GPOPS (General Pseudo-Spectral Optimal Control Software).

2.1.1 Mathematical Model

A nontrivial part of any optimal control problem is modeling the process. The ob-jective is to obtain the simplest mathematical description that predicts the responseof the physical/biological system to the inputs. Our discussion will be restricted tothe system described by ordinary differential equations (in state space descriptionform). We denote

x1(t), x2(t), ......, xn(t)

18

to be the state variables (or simply the states) of the process at the time t, and

u1(t), u2(t), ......, un(t)

the control inputs to the process at the time t. Hence, the system maybe described byn first-order differential equations

x1 (t) = a1 (x1 (t) , x2 (t) , ..., xn (t) , u1 (t) , u2 (t) , ..., um (t) , t)x2 (t) = a2 (x1 (t) , x2 (t) , ..., xn (t) , u1 (t) , u2 (t) , ..., um (t) , t)

.

. (2.1)

.xn (t) = an (x1 (t) , x2 (t) , ..., xn (t) , u1 (t) , u2 (t) , ..., um (t) , t) .

We will define

x (t) ,

x1 (t)x2 (t)

xn (t)

as the state vector and

u (t) ,

u1 (t)u2 (t)

un (t)

as the control vector. The state equation can then be written as follows:

x (t) = a (x (t) , u (t) , t) , (2.2)

where the definition of a (with the components a1(t), a2(t), ...., an(t)) is apparent byconparision with (2.1).

Definition 2.1 (see [8, page 6]) A history of control input values during the interval[t0, t f

]is denoted by u and is called a control history, or simply a control.

19

Definition 2.2 (see [8, page 6]) A history of state values in the interval[t0, t f

]is called a

state trajectory and is denoted by x.

Definition 2.3 (see [8, page 8]) A state trajectory which satisfies the state variable con-straints during the entire time interval

[t0, t f

]is called an admissible trajectory.

2.1.2 The performance measure

In order to evaluate the performance of a system quantitatively, the control designerselects a performance measure. An optimal control is defined as one that minimizes(or maximizes) the performance measure. In certain cases the problem statementmay clearly indicate what to select for a performance measure, whereas in otherproblems the selection is a subjective matter. For example, the statement, Transferthe system from point A to point B as quickly as possible, clearly indicates thatelapsed time is the performance measure to be minimized. On the other hand, thestatement, Maintain the position and velocity of the system near zero with a smallexpenditure of control energy, does not instantly suggest a unique performancemeasure. In such problems the designer may be required to try several performancemeasures before selecting one which yields what he considers to be optimal per-formance. This performance measure is very important to design optimal cost func-tional for HCV optimal control problem, which will be discussed in the next chapter.

In most cases, it will be assumed that the performance measure of a system is eval-uated by a measure of the form

J (u) = h(x(t f ), t f

)+

t ft0

g (x(t), u(t), t)dt, (2.3)

where t0 and t f are the initial and final time, h and g are scalar functions. t f may bespecified or free, depending on the problem formulation.

Starting from the initial state x(t0) = x0 and applying a control signal u(t) fort [t0, t f ] cause the system to follow some state trajectory. The performance mea-sure assigns a unique real number to each trajectory of the system.

2.1.3 The Optimal Control Problem

The problem is represented (see [8, page 10]) as follows:

Find an admissible control u which causes the system

x (t) = a (x (t) , u (t) , t) , (2.4)

20

to follow an admissible trajectory x that minimizes the performance measure

J = h(x(t f ), t f

)+

t ft0

g (x(t), u(t), t)dt. (2.5)

u and x are said to be optimal control and optimal trajectory, respectively. There arevery natural questions. First, we may not know in advance that an optimal controlexists. That is, it may be impossible to find a control which (a) is admissible and(b) causes the system to follow an admissible trajectory. Since existence theoremsare difficult to establish in general, in most cases we will attempt to find an optimalcontrol rather than try to prove that one exists.

Second, even if an optimal control exists, it may not be unique. A non-unique op-timal control may complicate computational procedures, but they allow the possi-bility of choosing among several controller configurations. This is certainly helpfulto the designer, because he/she can then consider other factor, such as cost, size,...which may both have been included in the performance measure.

Third, when we say that u causes the performance measure to be minimized, wemean that

J , h(x(t f ), t f

)+

t ft0

g (x(t), u(t), t)dt

h (x(t f ), t f )+t f

t0

g (x(t), u(t), t)dt

for all u U and x X. The above inequality states that an optimal control and itstrajectory cause the performance measure to have a value smaller than or equal tothe performance measure of any other admissible control and trajectory. Hence, weare seeking the absolute or global minimum of J, not mere local minima. Of course, oneway to find the global minimum is to determine all of the local minima and thensimply find the smallest value for the performane measure.

2.1.4 Forms of the Optimal Control

Definition 2.4 (see [8, page 14]) If a functional relationship of the form

u(t) = f (x(t), t) , (2.6)

21

can be found for the optimal control at the time t, then the function f is said to be the optimalcontrol law, or the optimal policy.

This form of the control is also known as the feedback control or closed-loop con-trol. Notice that (2.6) implies that f is a rule which determines the optimal controlat the time t for any admissible state value at time t.

Definition 2.5 (see [8, page 15]) If the optimal control is determined as a function of timefor a specified initial state value, that is,

u(t) = e (x(t0), t) , (2.7)

then the optimal control is said to be in open-loop form.

Thus the optimal open-loop control is optimal only for a particular initial statevalue, whereas, if the optimal control law is known, the optimal control history start-ing from any state value can be generated.

2.1.5 System Classifications

Systems are described by the term linear, nonlinear, time-invariant and time-varying.We shall classify according to the form of their state equations. For example, if asystem is nonlinear and time-varying, the state equations are written

x(t) = a(x(t), u(t), t). (2.8)

Nonlinear, time-invariant systems are represented by state equations of the form

x(t) = a(x(t), u(t)). (2.9)

If a system is linear and time-varying, its state equations are

x(t) = A(t)x(t) + B(t)u(t), (2.10)

where A(t) and B(t) are n n and nm matrices with time-varying elements. Stateequations for linear, time-invariant systems have the form

x(t) = Ax(t) + Bu(t), (2.11)

where A and B are constant matrices.

22

2.1.6 Output Equations

The physical quantities that can be measured are called the outputs and are denotedby y1(t), y2(t), ...., yq(t). If the outputs are nonlinear, time-varying function of thestates and controls, we write the output equations

y(t) = c (x(t), u(t), t) . (2.12)

If the output is related to the states and controls by a linear, time-invariant relation-ship, then

y(t) = Cx(t) + Du(t), (2.13)

where C and D are q n and qm constant matrices.

2.2 Necessary Conditions for Optimal Control

This section will apply the techniques introduced in the previous chapter to deter-mine necessary conditions for optimal control (see [8, page 184-218]). Our problemis to find an admissible control u(t) that causes the system

x(t) = a (x(t), u(t), t) (2.14)

to follow an admissible trajectory x(t) that minimizes the performance measure

J (u) = h(x(t f ), t f

)+

t ft0

g (x(t), u(t), t)dt. (2.15)

We shall initially assume that the admissible state and control regions are not bounded,and that the initial conditions x (t0) = x0 and the initial time t0 are specified. As of-ten, x is the n 1 state vector and u is the m 1 vector of control inputs.

In the terminology of Chapter 1, we have a problem involving n+m functions whichmust satisfy the n differential equation constraints (2.14). The m control inputs arethe independent functions.

The only difference between (2.15) and the functionals considered in Chapter 1 isthe term containing control u(t).

Now, assuming that h is a differentiable function, we can write

h(x(t f ), t f

)=

t ft0

ddt[h (x(t), t)]dt + h (x(t0), t0) , (2.16)

23

so that the performance measure can be expressed as

J (u) =

t ft0

{g (x(t), u(t), t) +

ddt[h (x(t), t)]

}dt + h (x(t0), t0) . (2.17)

Since x(t0) and t0 are fixed, the minimization does not affect the h (x(t0), t0) term,so we need to consider only the functional

J (u) =

t ft0

{g (x(t), u(t), t) +

ddt[h (x(t), t)]

}dt. (2.18)

Applying the chain rule of differentiation, we obtain

J (u) =

t ft0

{g (x (t) , u (t) , t) +

[hx

(x (t) , t)]T

x (t) +ht

(x (t) , t)

}dt. (2.19)

To include the differential equation constraints, we form the augmented functionalas follow

Ja (u) =

t ft0

{g (x (t) , u (t) , t) +

[hx

(x (t) , t)]T

x (t)

+

[ht

(x (t) , t)]+ pT (t) [a (x (t) , u (t) , t) x (t)]

}dt (2.20)

by introducing the Lagrange multipliers p1(t), p2(t), ..., pn(t). Let us define

ga(x(t), x(t), u(t), p(t), t) ,g(x(t), u(t), t) + pT(t) [a(x(t), u(t), t) x(t)]

+

[hx(x(t), t)

]Tx(t) +

ht(x(t), t)

so that

Ja (u) =

t ft0

{ga (x(t), x(t), u(t), p(t), t)} dt (2.21)

24

We shall assume that the end point at t = t f can be specified or free. To determinethe variation of Ja, we introduce the variations x, x, u, p and t f . From theresults of Problem 1 in Chapter 1, this gives

Ja (u) = 0 =[gax(x(t f)

, x(t f)

, u(t f)

, p(t f)

, t f)]T

x f

+

[ga(x(t f)

, x(t f)

, u(t f)

, p(t f)

, t f)

[gax(x(t f)

, x(t f)

, u(t f)

, p(t f)

, t f)]T

x(t f)]

t f

+

t ft0

{[[gax

(x (t) , x (t) , u (t) , p (t) , t)]T

(2.22)

ddt

[gax

(x (t) , x (t) , u (t) , p (t) , t)]T]

x (t)

+

[gau

(x (t) , x (t) , u (t) , p (t) , t)]T

u (t)

+

[gap

(x (t) , x (t) , u (t) , p (t) , t)]T

p (t)

}dt.

Notice that the above result is obtained because u (t) and p (t) do not appear in ga.Next, let us consider only those terms inside the integral which involves the functionh. These terms contain

x

[[hx(x(t), t)

]Tx(t) + h

t(x(t), t)

] d

dt

{hx

[[hx(x(t), t)

]Tx(t)

]}.

(2.23)Writing out the indicated partial derivatives, it gives[

2hx2

(x(t), t)]

x(t) +[2htx

(x(t), t)] d

dt

[hx(x(t), t)

]. (2.24)

So, if we apply the chain rule to the last term,[2hx2

(x (t) , t)]

x (t) +[2htx

(x (t) , t)]

[2hx2

(x (t) , t)]

x (t)[2htx

(x (t) , t)]= 0.

25

Therefore, in the integral term we get

t ft0

{[[gx

(x (t) , u (t) , t)]T

+ [p (t)]T[ax

(x (t) , u (t) , t)]

ddt[p (t)]T

]x (t) +

[[gu

(x (t) , u (t) , t)]T

+ [p (t)]T[au

(x (t) , u (t) , t)]]

u (t) (2.25)

+[[a (x (t) , u (t) , t) x (t)]T

]p (t)

}dt.

This integral must vanish on an extremal regardless of the boundary conditions. Wefirst observe that the constraints

x (t) = a (x (t) , u (t) , t) (2.26)

must be satisfied by an extremal so that the coefficient of p(t) is zero. The Lagrangemultipliers are arbitrary, so let us select them to make the coefficient of x(t) equalto zero, that is

p (t) = [ax

(x (t) , u (t) , t)]T

p (t) gx

(x (t) , u (t) , t) . (2.27)

We shall call (2.27) the costate equations and p(t) the costate. The remaining variationu(t) is independent, so its coefficient must be zero. Hence,

0 =gu

(x (t) , u (t) , t) +[au

(x (t) , u (t) , t)]T

p (t) . (2.28)

There are still the terms outside the integral to deal with. Since the variation mustbe zero, we have[

hx(x(t f)

, t f) p (t f )]Tx f + [g (x (t f ) , u (t f ) , t f )+ ht (x (t f ) , t f )

+[p(t f)]T [a (x (t f ) , u (t f ) , t f )] ] t f = 0. (2.29)

In writing (2.29), we have used the fact that x(t f)= a

(x(t f)

, u(t f)

, t f). Equa-

tion (2.26), (2.27) and (2.28) are the necessary conditions consisting of a set of 2n,

26

first order differential equations. The solution of the state and costate equations willcontain 2n constants of integration. To evaluate these constants we use the n equa-tion x(t0) = x0 and an additional set of n or (n + 1) relationships - depending onwhether or not t f is specified - from (2.29). Notice that, as expected, we are againconfronted by a two-points boundary value problem.

In the following we shall find it convenient to use the function H, called the Hamil-tonian, defined as

H (x (t) , u (t) , p (t) , t) = g (x (t) , u (t) , t) + pT (t) [a (x (t) , u (t) , t)] . (2.30)

Using this notation, we can write the necessary condition (2.26), (2.27) and (2.28) asfollows:

x (t) =Hp

(x (t) , u (t) , p (t) , t)

p (t) = Hx

(x (t) , u (t) , p (t) , t)

0 =Hu

(x (t) , u (t) , p (t) , t)

for all t [t0, t f ] and[H(x(t f)

, u(t f)

, p(t f)

, t f)+

ht(x(t f)

, t f)]

t f

+

[hx(x(t f)

, t f) p (t f )]Tx f = 0. (2.31)

In a particular problem, either g or h may be missing. In this case, we simple strikeout the terms involving the missing function. To determine the boundary conditionsis a matter of making the appropriate substitutions in (2.31). In general, it will beassumed that we have the n equations x(t0) = x0.

2.3 Linear Regulator Problems

In this section, we are going to consider a very important class of optimal controlproblems: Linear regulator systems. We will illustrate that for linear regulator prob-lems, the optimal control law can be found as a linear time-varying function of the

27

system states (i.e, in feedback form). The process is described by linear state equa-tions:

x (t) = A (t) x (t) + B (t) u (t) , (2.32)

which may have time-varying coefficients. The performance measure to be mini-mized is

J =12

xT(t f)

Hx(t f)+

12

t ft0

[xT (t)Q (t) x (t) + uT(t)R(t)u(t)

]dt. (2.33)

Here, the final time t f is fixed, H and Q are real symmetric positive semi-definitematrices, and R is a real symmetric positive definite matrix. It is assumed that thestates and controls are not bounded, and x(t f ) is free. We attach the following phys-ical interpretation to this performance measure: It is desired to maintain the statevector close to the origin without an excessive expending of control effort. TheHamiltonian is

H (x (t) , u (t) , p (t) , t) =12

xT (t)Q (t) x (t) +12

uT (t)R (t) u (t)

+ pT (t)A (t) x (t) + pT (t)B (t) u (t) , (2.34)

and necessary conditions for optimality are

x (t) = A (t) x (t) + B (t) u (t) (2.35)

p (t) = Hx

= Q (t) x (t)AT (t) p (t) (2.36)

0 =Hu

= R(t)u(t) + BT(t)p(t) (2.37)

The Equation (2.37) can be solved for u(t) to give

u (t) = R1 (t)BT (t) p (t) ; (2.38)Note that R1 always exists since R is a positive definite matrix. Substituting yields

x(t) = A(t)x(t) B(t)R1(t)BT(t)p(t); (2.39)Hence, we have the set of 2n linear homogeneous differential equations x(t)

p(t)

= A(t) | B(t)R1(t)BT(t) | Q(t) | AT(t)

x(t)p(t)

. (2.40)28

The solution of these equations in terms of the state transition matrices has the form x(t f )p(t f )

= 11 (t f , t) | 12(t f , t) |

21(t f , t) | 22(t f , t)

x(t)p(t)

, (2.41)where 11, 12, 21 and 22 are n n state transition matrices. From the boundary-condition equations, we find that

p(t f ) = Hx(t f ). (2.42)

Substituting this for p(t f ) in (2.41), it gives x(t f )Hx(t f )

= 11 (t f , t) | 12(t f , t) |

21(t f , t) | 22(t f , t)

x(t)p(t)

. (2.43)Therefore,

H11(t f , t)x(t) + H12(t f , t)p(t) = 21(t f , t)x(t) + 22(t f , t)p(t). (2.44)

Hence,

p(t) =[22(t f , t)H12(t f , t)

]1 [H11(t f , t) 21(t f , t)] x(t). (2.45)As a result,

p(t) , K(t)x(t), (2.46)which means that p(t) is a linear function of the states of the system. K is an n nmatrix. Actually, K depends on t f also, but t f is specified. Substituting in (2.38), weobtain:

u(t) = R1(t)BT(t)K(t)x(t) , F(t)x(t) (2.47)which indicates that the optimal control law is a linear, albeit time varying, combi-nation of the system states. Notice that even if the plant is fixed, the feedback gainmatrix F is time-varying. In addition, measurements of all of the states variablesmust be available to implement the optimal control law.

To determine the feedback gain matrix F, we need the state transition matrices forthe system. If all the involved matrices (A, B, R, Q) are time-invariant, the requiredtransition matrix can be found by evaluating the inverse Laplace transform of thematrix sI

A | BR1BT | Q | AT

1

.

29

Unfortunately, when the order of the system is large this becomes a tedious andtime-consuming task. If any of the matrices in (2.40) is time-varying, we must gen-erally resort to a numerical procedure for evaluating (t f , t). Fortunately, it can beshown that the matrix K satisfies the Riccati equation:

K(t) = K(t)A(t)AT(t)K(t)Q(t) + K(t)B(t)R1(t)BT(t)K(t), (2.48)with the boundary condition K(t f ) = H. For t f , i.e, the infinite time horizonproblem, it can be shown that [7, Kalman] K(t) K (a constant matrix) as t f . To determine the K matrix for an infinite-time process, we either integrate theRiccati equation backward in time until a steady-state solution is obtained or solvethe nonlinear algebraic equations

0 = KAATKQ + KBR1BTK, (2.49)obtained by setting K(t) = 0 in (2.48).

2.4 Bounded Controls

If the control u(t) is constrained to lie in an admissible region, u U, (for example,in the case of optimal control of HCV, the controls are bounded above by 1 andbelow by 0), the stationary condition

Hu

= 0

is replaced with the more general condition

H (x (t) , u (t) , p (t) , t) H (x (t) , u (t) , p (t) , t) .This is the well known Pontryagins Maximum (or Minimum) Principle.

2.5 Numerical Solutions to the Optimal Control Prob-lems

Optimal control problems are in general nonlinear and therefore, generally do nothave analytic solutions (e.g, like the linear-quadratic optimal control problem). Asa consequence, it is necessary to utilize numerical methods to solve optimal con-trol problems. In principle, there are three approaches for numerically solving theoptimal control problem:10 (i) solve the two-point boundary value problem as de-rived from the necessary conditions; (ii) completely discretize the necessary con-ditions and solve these equations as a nonlinear equation; or (iii) approximate the

30

original optimal control problem by a discrete optimization problem. As our ad-missible control set is bounded, we approach this problem as an optimization prob-lem. In particular, we employed GPOPS (General Pseudo-spectral Optimal Con-trol Software), open-source MATLAB optimal control software that implements theGauss and Radau HP-adaptive pseudospectral methods. These methods approxi-mate the state using a basis of Lagrange polynomials and collocate the dynamics atthe Legendre-Gauss-Radau points. The continuous-time optimal control problem isthen transcribed to a finite-dimensional nonlinear programming problem (NLP) andthe NLP is solved using well-known software tools.14 This approach is known asthe direct transcription method and often referred to in the literature as discretizethen optimize. GPOPS has been shown to converge exponentially fast for problemswhose solutions are sufficiently smooth.

In GPOPS, the cost functional that need to be minimized is of the following form:

J =P

p=1

J(p)P

p=1

[(p)(x(p)(t0), t0, x(p)(t f ), t f , q(p)) + L(p)(x(p)(t), u(p)(t), t, q(p))dt

](2.50)

subject to the dynamic constraint

x(p) = f (p)(x(p), u(p), t, q(p)), (p = 1, 2, ...., P), (2.51)

the inequality path constraints

C(p)min C(p)(x(p)(t), u(p)(t), t; q(p)) C(p)max, (p = 1, ...., P), (2.52)and the phase continuity (linkage) constraints

P(s)(

x(psl )(t f)

, t(psl )

f ; q(psl ), x(p

su) (t0) , t

(psu)0 ; q

(psu))= 0, (pl, pu [1, ..., P] , s = 1, ...., L)

(2.53)where x(p)(t) Rnp , u(p)(t) Rmp , q(p) Rqp , and t R are, respectively, thestate, control, static parameters, and time in phase p [1, ...., P], L is the number ofphases to be linked, psl [1, ...., P], (s = 1, ...., L) are the left phase numbers, andpsu [1, ...., P], (s = 1, ...., L) are the right phase numbers. While much of the timea user may want to solve a problem consisting of multiple phases, it is important tonote that the phases need not to be sequential. To the contrary, any two phases maybe linked provided that the independent variable does not change direction.

In order to verify that we can implement an optimal control problem in GPOPS cor-rectly, we now consider the following optimal control example from [8, page 216])in which the exact optimal solutions are known.

31

In particular, we consider the second-order differential equations system

x1 (t) = x2 (t)x2 (t) = 2x1 (t) x2 (t) + u (t)

with initial conditions x1(0) = 4 and x2(0) = 4. The system is to be controlled tominimize the cost functional

J (u) =T

0

[x21 (t) +

12

x22 (t) +14

u2 (t)]

dt.

The code that sets up the optimal control problem in GPOPS is given below:

%------------------------------------------------------------------%

% BEGIN: script Example522KirkMain.m %

%------------------------------------------------------------------%

% This example is taken from the following reference: %

% Donald E. Kirk, Optimal Control Theory: An Introduction %

% Example 5.2-2, page 216.

%------------------------------------------------------------------%

clear all

clc

t0 = 0;

tf = 10;

x10 = -4.0;

x20 = 4.0;

x1min = -10;

x1max = 10;

x2min = -10;

x2max = 10;

umin = -10;

umax = 10;

32

iphase = 1;

limits(iphase).time.min = [t0 tf];

limits(iphase).time.max = [t0 tf];

limits(iphase).state.min(1,:) = [x10 x1min x1min];

limits(iphase).state.max(1,:) = [x10 x1max x1max];



limits(iphase).control.min = umin;

limits(iphase).control.max = umax;

limits(iphase).parameter.min = [];

limits(iphase).parameter.max = [];

limits(iphase).path.min = [];

limits(iphase).path.max = [];

limits(iphase).event.min = [];

limits(iphase).event.max = [];

guess(iphase).time = [t0; tf];

guess(iphase).state(:,1) = [x10; x10];


guess(iphase).control = [0; 0];

guess(iphase).parameter = [];

setup.name = Example522Kirk;

setup.funcs.cost = Example522KirkCost;

setup.funcs.dae = Example522KirkDae;

setup.linkages = [];

setup.limits = limits;

setup.guess = guess;

setup.derivatives = finite-difference;

setup.checkDerivatives = 0;

setup.autoscale = off;

setup.mesh.tolerance = 1e-6;

setup.mesh.iteration = 20;

setup.mesh.nodesPerInterval.min = 4;

setup.mesh.nodesPerInterval.max = 10;

[output,gpopsHistory] = gpops(setup);

%----------------------------------%

% END: script Example522KirkMain.m %

%----------------------------------%

33

%-------------------------------------%

% BEGIN: function Example522KirkDae.m %

%-------------------------------------%

function [dae] = Example522KirkDae(sol);

t = sol.time;

x = sol.state;

u = sol.control;

p = sol.parameter;

x1dot = x(:,2);

x2dot = 2*x(:,1)-x(:,2)+u;

dae = [x1dot x2dot];

%-------------------------------------%

% END: function Example522KirkDae.m %

%-------------------------------------%

%--------------------------------------%

% BEGIN: function Example522KirkCost.m %

%--------------------------------------%

function [Mayer,Lagrange] = Example522KirkCost(sol);

t0 = sol.initial.time;

x0 = sol.initial.state;

tf = sol.terminal.time;

xf = sol.terminal.state;

t = sol.time;

x = sol.state;

u = sol.control;

p = sol.parameter;

Mayer = zeros(size(t0));

Lagrange = x(:,1).^2+x(:,2).^2+0.25*u.^2;

%--------------------------------------%

% END: function Example522KirkCost.m %

%--------------------------------------%

The numerical optimal solutions obtained with GPOPS are depicted in the fol-

34

lowing figure:

The approximated optimal solutions are, indeed, closely matched with the exactoptimal solutions given in [8, page 218].

35

Chapter 3

AN OPTIMAL CONTROL APPROACHFOR TREATMENT OF HEPATITIS CPATIENTS

3.1 Mathematical Modeling of HCV Infection

One of the first mathematical model of hepatitis C viral dynamics was the modelby Neumann in 1998 [9, page 104]. Neumann developed the model to study theantiviral effects of interferon- (IFN). The mathematical model is given by

dTdt

= s dT (1 ) VT (3.1)dIdt

= (1 ) VT I (3.2)dVdt

= (1 e) pI cV (3.3)

where T and I are the concentrations of healthy and infected hepatocytes, respec-tively, and V is the concentration of viral load. Healthy hepatocytes are produced atrate s and die with death rate constant d. Cells become infected with de novo infec-tion rate constant and, once, infected, die with rate constant . Hepatitis C virionsare produced by infected cells per day and are cleared with clearance rate constantc. The possible effects of IFN in this model are to reduce either the production ofvirions from infected cells by a fraction (1 e) or the de novo rate of infection by afraction (1 ). Before IFN therapy, e = = 0. Once therapy is initiated, e > 0 or > 0 or both. A major contribution of Neumann et al. was their determination thatinterferon acts through the e mechanism rather than the mechanism, i.e, IFN actsby inhibiting the production of visions.1

36

Standard care of HCV has since evolved to include, in addition to IFN, the an-tiviral ribavirin, which acts by rendering some visions noninfectious. To study theresponse of hepatitis C to this combined treatment, Snoeck et al. [15, page 711] ex-tended the original model of Newmann et al. to include a noninfectious virionsvariable. The model is given by

dTdt

= s + rT(

1 T + ITmax

) dT VIT

dIdt

= VIT + rI(

1 T + ITmax

) I

dVIdt

= (1 ) (1 e) pI cVIdVNI

dt= (1 e) pI cVNI

Here, VI represents the concentration of infectious visions and VNI describes theconcentration of noninfectious virions. Another difference is the addition of termswith coefficient r to model the livers regenerative mechanism. The dynamics ofthis model after the end of treatment are of interest, so Snoeck et al. include expo-nential decay of the drug efficacies e and , which correspond to IFN and ribavirin,respectively.

e = eek(ttend)+

and = ek(ttend)+ ,

where tend marks the end of treatment, and

(a)+ =

{a if a 00 otherwise

.

37

The steady states of the system were obtained by Ransley (see [13, page 25]) as fol-lows:

Tu =Tmax

2r

(r d +

(r d)2 + 4rs

Tmax

)Iu = VIu = VNIu = 0

Ti =12

r2DA2

+

(r2DA2

)2+

4rsTmaxA2

Ii = Ti

(Ar 1)+ Tmax Tmaxr

VIi = epc

Ii

VNIi = epc

Ii,

where A = epTmax

c , D =Tmax

r2 [A (r ) + r (d )] , = 1 , e = 1 e.Here, Tu, Iu, VIu, and VNIu denote the uninfected steady states and Ti, Ii, VIi, VNIirepresents the infected steady states. The Snoeck et al. model was validated againstclinical data.15, 1 In the following, we will simulate the partial virologic response(PVR) in which the patient responds to the treatment initially and then stops re-sponding while still on treatment. The parameters that correspond to PVR are:

Const Description Unit Typical valueTmax Total number of hepatocytes per ml H/ml 18.5 10+6

s Hepatocyte production rate H/ml/day 61.7 10+3d Hepatocyte death rate constant H/day 0.003r Hepatocyte proliferation rate constant H/day 0.00562p Virion production rate IU/day 25.1k Antiviral-effect decay constant day 1 0.0238 Infection rate constant H/IU/day 4.1684 109 Infected cell death rate constant H/day 1.2110 101c Virion elimination rate constant IU/day 2.7018e PEG drug-efficacy Scalar 6.1382 101 RVB drug-efficacy Scalar 1.2216 101

The initial conditions for our system are not known and are difficult to determineclinically. We followed13 and took the initial conditions to be equal to the infectedsteady states of the model. The PVR simulation results are depicted in the next 3

38

figures.

39

3.2 An Optimal Treatment Formulation Problem

We formulate the problem of finding an effective multi drug therapy as an optimalcontrol problem. Let us consider the optimal control problem:

Minimize

J(x, u) = 224

0

[V2 + I2 T2 + e2 + 2

]dt

subject to

dTdt

= s + rT(

1 T + ITmax

) dT VIT

dIdt

= VIT + rI(

1 T + ITmax

) I

dVIdt

= (1 ) (1 e) pI cVIdVNI

dt= (1 e) pI cVNI

Here, the viral load V = VI +VNI and the control variables 0 e 1 and 0 1represent the efficacies of the drugs IFN and ribavirin, respectively. The goals

40

of the controls are to maximize the healthy hepatocytes while minimizing the viralload, the infected hepatocytes, and the drug usages subject to the HCV dynamicsgiven by the equation (??). It should be noted that the parameters for the HCVmodel correspond to the PVR case and are given in Section 3.1.

The GPOPS code that implemented the optimal control problem for the HCV modelis given below.

%------------------------------------------------------------------%

% BEGIN: HCVModelMain.m %

%------------------------------------------------------------------%

clear all

clc

t0 = 0;

tf = 224;

% tf = 336;

x10 = 1.0e+6*2.0175;

x20 = 1.0e+6*0.4879;

x30 = 1.0e+6*2.7035;

x40 = 0;

x1min = 0;

x1max = 1.e+10;

x2min = 0;

x2max = 1.e+10;

x3min = 0;

x3max = 1.e+10;

x4min = 0;

x4max = 1.e+10;

u1min = 0;

u1max = 1;

u2min = 0;

u2max = 1;

iphase = 1;

limits(iphase).time.min = [t0 tf];

limits(iphase).time.max = [t0 tf];









limits(iphase).control.min(1,:) = u1min;

limits(iphase).control.max(1,:) = u1max;

limits(iphase).control.min(2,:) = u2min;

limits(iphase).control.max(2,:) = u2max;

41

limits(iphase).parameter.min = [];

limits(iphase).parameter.max = [];

limits(iphase).path.min = [];

limits(iphase).path.max = [];

limits(iphase).event.min = [];

limits(iphase).event.max = [];

guess(iphase).time = [t0; tf];





guess(iphase).control(:,1) = [1.2216e-01; 1.2216e-01];

guess(iphase).control(:,2) = [6.1382e-01; 6.1382e-01];

guess(iphase).parameter = [];

setup.name = HCVModelMain;

setup.funcs.cost = HCVModelCost;

setup.funcs.dae = HCVModelDae;

setup.linkages = [];

setup.limits = limits;

setup.guess = guess;

setup.derivatives = finite-difference;

setup.checkDerivatives = 0;

setup.autoscale = off;

setup.mesh.tolerance = 1e-6;

%limits(iphase).nodesPerInterval = tf*20;

%setup.mesh.iteration = 0;

setup.mesh.iteration = 5;

setup.mesh.nodesPerInterval.min = 4

setup.mesh.nodesPerInterval.max = 12;

[output,gpopsHistory] = gpops(setup);

%----------------------------------%

% END: script HCVModelMain.m %

%----------------------------------%

%-------------------------------------%

% BEGIN: function HCVModelDae.m %

%-------------------------------------%

function [dae] = HCVModelDae(sol);

% PVR

beta = 4.1684e-9;

delta = 1.2110e-1;

p = 25.1;

c = 2.7018e+0;

r = 0.00562;

Tmax = 18.5e+6;

s = 61.7e+3;

d = 0.003;

% Relapse

% beta = 1.2e-8;

% delta = 1.0382e-01;

% p = 25.1;

% c = 2.2236e+00;

% r = 0.00562;

% Tmax = 18.5e+6;

% s = 61.7e+3;

42

% d = 0.003;

% SVR

% beta = 1.2e-8;

% delta = 5.8411e-01;

% p = 25.1;

% c = 4.7268e-1;

% r = 0.00562;

% Tmax = 18.5e+6;

% s = 61.7e+3;

% d = 0.003;

t = sol.time;

x = sol.state;

u = sol.control;

par = sol.parameter;

x1dot = (s+r*x(:,1).*(1-(x(:,1)+x(:,2))/Tmax)-d*x(:,1)-beta*x(:,3).*x(:,1));

x2dot = (beta*x(:,3).*x(:,1)+r*x(:,2).*(1-(x(:,1)+x(:,2))/Tmax)-delta*x(:,2));

x3dot = (p*x(:,2).*(1-u(:,1)).*(1-u(:,2))-c*x(:,3));

x4dot = (p*x(:,2).*u(:,1).*(1-u(:,2))-c*x(:,4));

dae = [x1dot x2dot x3dot x4dot];

%-------------------------------------%

% END: function HCVModelDae.m %

%-------------------------------------%

%--------------------------------------%

% BEGIN: function HCVModelCost.m %

%--------------------------------------%

function [Mayer,Lagrange]=HCVModelCost(sol);

q = 1;

r1 = 1;

r2 = 1;

t0 = sol.initial.time;

x0 = sol.initial.state;

tf = sol.terminal.time;

xf = sol.terminal.state;

t = sol.time;

x = sol.state;

u = sol.control;

p = sol.parameter;

VL = (x(:,3)+x(:,4));

HC = (x(:,1));

IC = (x(:,2));

Mayer = zeros(size(t0));

Lagrange = q*VL.^2+1.0*IC.^2-HC.^2+r1*u(:,1).^2+r2*u(:,2).^2;

%--------------------------------------%

% END: function HCVModelCost.m %

%--------------------------------------%

figure(1);

plot(output.solution.time,output.solution.control(:,1));

xlabel(Time (days));

43

ylabel(rho);

title(Optimal Control Solution);

figure(2);

plot(output.solution.time,output.solution.control(:,2));


ylabel(epsilon);

title(Optimal Control Solution);

figure(3);

plot(output.solution.time,output.solution.state(:,1));


ylabel(T);

title(Optimal Solution (Healthy Hepatocytes));

figure(4);

plot(output.solution.time,output.solution.state(:,2));


ylabel(I);

title(Optimal Solution (Infected Hepatocytes));

figure(5)

plot(output.solution.time,output.solution.state(:,3)+output.solution.state(:,4));


ylabel(V);

title(Optimal Solution (Viral Load));

The treatment is during 32 weeks (224 days). The optimal solutions depicted inthe next five figures showed that the optimal control solutions successfully treatedthe patient, i.e, sustained virology response (SVR), using a treatment protocol thatfollows the pattern of structured treatment interruption (that is, on-off-type of treat-ment). The viral load goes to zero very fast while the concentration of healthy hep-atocytes steadily increases and the concentration of infected hepatocytes decreasesto zero. As shown in Section 3.1, this same patient when treated with a continuousdosage (e = 6.1382 101, = 1.2216 101) of drugs IFN and ribavirin for 32

44

weeks only responded to the treatment partially (PVR).

45

To make sure that the patients remain cured after the treatment has stopped, wesimulated the HCV model after the treatment has stopped. The simulated results,which are depicted in the next three figures, showed that viral load and the infectedhepatocytes stay below the detected levels while the healthy hepatocytes continued

47

to grow.

48

Conclusions

We presented and used methodologies from optimal control theories to design andsynthesize an open-loop control based treatment regimen for HCV dynamics. Themathematical model for HCV progression includes compartments for healthy hepa-tocytes, infected hepatocytes, infectious virions and noninfectious virions. The HCVmodel is subjected to multiple (IFN and ribavirin) drug treatments as controllers.We demonstrated through numerical simulations that by using a target trackingapproach, an optimal treatment strategies can be designed to successfully cure apatient with HCV (SVR). The optimal treatment strategy followed the pattern of on-off schedule known in the HIV community as structured treatment interruption. Itshould be emphasized that this same patient when subjected to a continuous treat-ment regimen only showed partial virologic response (PVR). The potential for anoptimal treatment approach for HCV patients is very exciting, but more work isnecessary before it can be implemented in clinical settings. For example, more re-search is needed to incorporate the effects of drug resistance and mutation as wellas more accurate mathematical models of HCV dynamics that include the immuneresponse.

50

Bibliography

[1] ARTHUR, J. G., TRAN, H. T. Feasibility of Parameter Estimation in Hepatitis CViral Dynamics Models. Pre-print submitted to Applied Mathematics Letters.

[2] BEELER, S. C., TRAN, H. T., BANKS, H. T. Feedback Control Methodologiesfor Nonlinear Systems. Journal of Optimization, Theory and Applications 107, 1,(2000), pp. 1 - 33.

[3] BRIAN R. HUNT, RONALD L. LIPSMAN, JONATHAN M. ROSENBERG WITHKEVIN R. COOMBES, JOHN E. OSBORN, GARRETT J. STUCK. A Guide to MAT-LAB for Beginners and Experienced Users. Cambridge University Press, (2001).

[4] BANKS, H. T., TRAN, H. T. Mathematical and Experimental Modelling for Physicaland Biological Processes. Chapman & Hall/CRC Press, (2009).

[5] DAVID, J. A. Optimal Control, Estimation, and Shape Design: Analysis and Appli-cations. Ph.D Thesis, Department of Mathematics, North Carolina State Univer-sity, Raleigh, North Carolina, (2007).

[6] DIXIT, N. M., LAYDEN-ALMER, J. E., LAYDEN, T. J., PERELSON, A. S. Mod-elling how ibavirin improves interferon response rates in Hepatitis C Virus in-fection. Nature 432, (2004), pp. 922 - 924.

[7] KALMAN, R. E. Contribution to the Theory of Optimal Control. Boletin SociedadMatematica Mexicana (1960), pp. 102-119.

[8] KIRK, D. E. Optimal Control Theory: An Introduction. Dover Publication, Inc,(2004).

[9] NEUMANN, A. U., LAM, N. P., DAHARI, H., GRETCH, D. R., WILEY, T. E.,LAYDEN, T. J., PERELSON, A. S. Hepatitis C Viral Dynamics in Vivo and theAntiviral Efficacy of Interferon- Therapy. Science 282, 5386, (1998), pp. 103 -107.

[10] SARGENT, R. W. H. Optimal Control. J. Comput. Appl. Math., 124, (2000), pp.361-371.

51

[11] Hepatitis C. Wikipedia: http://en.wikipedia.org/wiki/Hepatitis C.

[12] Hepatitis A, B, and C Center: Symptoms, Causes, Tests, Transmission, and Treat-ments. WebMD.

[13] RANSLEY, M. MMath Project: Mathematical Analysis of the HCV Model.http://www.ucl.ac.uk/~ucbpran/MMath.pdf (2011).

[14] RAO, A. V., BENSON, D., DARBY, C. L., MAHON, H., FRANCOLIN, C., PAT-TERSON, M., SANDERS, I., HUNTINGTON, G. T. Users Manual for GPOPS Ver-sion 5.0: A MATLAB Software for Solving Multiple-Phase Optimal Control ProblemsUsing hp-Adaptive Pseudospectral Methods, (2009).

[15] SNOECK, E., CHANU, P., LAVIELLE, M., JACQMIN, P., JONSSON, E. N., JORGA,K., GOGGIN, T., GRIPPO, J., JUMBE, N. L., FREY, N. A Comprehensive Hepati-tis C Viral Kinetic Model Explaining Cure. Clinical Phamacology and Therapeutics87, 6, (2010), pp. 706 - 713.

52

The Calculus of VariationsFundamental conceptsThe Fundamental Theorem of the Calculus of VariationsFunctionals of a single functionThe simplest variational problemFinal-time specified, final-state freeFinal-time free, final-state specifiedProblem with both the final-time and final-state free

An Introduction to Optimal Control TheoryIntroductionMathematical ModelThe performance measureThe Optimal Control ProblemForms of the Optimal ControlSystem ClassificationsOutput Equations

Necessary Conditions for Optimal ControlLinear Regulator ProblemsBounded ControlsNumerical Solutions to the Optimal Control Problems

An Optimal Control Approach for Treatment of Hepatitis C PatientsMathematical Modeling of HCV InfectionAn Optimal Treatment Formulation Problem

Outline

thesis tuan

Documents

liver transplantation

liver cancer

honors mathematics class

hepatitis cvirus

vietnam national university

endstage liver disease

human viral infections

viral disease development