Chapter 2Hamilton–Jacobi–Bellman equations

In this chapter we present recent developments in the theory of Hamilton–Jacobi–Bellman (HJB) equations as well as applications. The intention of this chapter is toexhibit novel methods and techniques introduced few years ago in order to solvelong-standing questions in nonlinear optimal control theory of Ordinary DifferentialEquations (ODEs).

The HJB approach we are concerned with is a technique used to study optimalcontrol problems, and it is based on a functional equation known as the DynamicProgramming Principle, introduced by Richard Bellman in the 1950s. This func-tional equation holds under very mild hypotheses and it is basically an optimalitycondition that suggests that some quantity remains constant all along optimal tra-jectories of the dynamic optimization problem at hand.

The main advantage of this method is that, in principle, the value function of asuitable optimal control problem is the unique mapping that verifies the DynamicProgramming Principle and therefore, the idea is to find an equivalent formulationof the functional equation in terms of a Partial Differential Equation (PDE), theso-called HJB equation.

Hamilton-Jacobi (HJ) equations are fully nonlinear PDEs normally associatedwith classical mechanics problems. The HJB equation is a variant of the latter andit arises whenever a dynamical constraint affecting the velocity of the system ispresent. This constraint in turn, appears frequently in the form a control variable, aninput that allow us to change the output of the system in a well defined way.

The HJB equation, as mentioned earlier, can also be considered as a differentialexpression of the Dynamic Programming Principle. Under rather mild assumptionsand when no constraints affect directly the system, this fully nonlinear PDE of firstor second order has been shown to be well-posed in the context of viscosity solu-tions, which were introduced by Crandall and Lions in the 1980s. From the optimalcontrol point of view, the approach consists in calculating the value function asso-ciated with the control problem by solving the HJB equation, and then identify anoptimal control and the associated optimal trajectory. The method has the great ad-vantage of directly reaching the global optimum of the control problem, which is

119

particularly relevant when the problem is non convex, besides providing a construc-tive procedure for the synthesis of an optimal control in feedback form.

For an optimal control problem with n state variables, the application of the dy-namic programming principle leads to a HJB equation over a state space of at leastthe same dimension. It is clear then how a key point for the applicability of themethod, is to have effective tools and appropriate numerical techniques to deal witha complexity that grows exponentially with respect to dimension of the state space.

The numerical tools developed in an optimal control context can be successfullyadapted to different problems coming from applied purposes. This is the case ofthe Shape-from-Shading problem, front propagation and Mean Field theory. Thoseextensions are discussed in the devoted sections.

Moreover, due to physical or economical limitations, we may be forced to includestate constraints in the formulation of the optimal control problem. This fact yieldsto some technical difficulties, for example, the value function may not be continuousnor real-valued, not even for very regular data. Thus, some additional compatibilityassumptions involving the dynamics and the state constraints set are required for thecharacterization of the value function in terms of the HJB equation. This fact can beexplained by the lack of information on the boundary of the state constraints.

The chapter is organized as follows. In Section 1 we present a brief overviewinto some basic results on optimal control and viscosity solutions theory. Then, inSection 2 we discuss two recent theoretical results on state-constrained optimal con-trol problems. The first one deals with the well-posedness of the HJB equation ongeometrical complex domain, while the second is concerned with method that leadsto a practical way to compute the value function via an auxiliar optimal controlproblem without state constraints. In Section 3, we study the numerical approxi-mation of HJB equations via special high-order filtered schemes, and we also ex-hibit a collection of benchmark tests where the advantages and the drawbacks ofthe proposed method are studied. We solve in Section 4, applied problems in engi-neering and computer vision, by using the tools proposed in the previous sections.Finally, in Section 5 we present a survey on Mean Field Games, involving the cou-pling between HJB and transport equations. This recent subject of research has hada sustained development in last years, due to its wide range of applicability in in-dustrial economics, non-linear statistics, modeling of commodities price dynamics,microstructure and crowd management.

1 Introduction

In this section we recall useful basic definitions and theoretical results concerningHJB equations.

1.1 Preliminaries on Control Systems and Optimization problems

We begin by considering a parametrized dynamical system dwelling on Rn:y(s) = f (s,y(s),u(s)), for a.e. s ∈ (t,T )u(s) ∈U, for all s ∈ (t,T )y(t) = x.

(162)

The elements that define a control system are the following: T ∈ R∪+∞ is thefinal horizon, t ∈ (−∞,T ) is the initial time, x ∈Rn is the initial position, u : R→Uis the control function with values in the control space U and f : R×Rn×U → Rn

is the dynamics mapping. In this chapter we assume that

U ⊆ Rm is compact and nonempty. (HU )

For sake of simplicity, we introduce the set

U (a,b) = u : (a,b)→U measurable.

Unless otherwise stated, all along this chapter we are assuming that(i) f : R×Rn×U → Rn is continuous.(ii) ∃L f > 0,∀x,y ∈ Rn, ∀t,s ∈ R, ∀u ∈U

| f (t,x,u)− f (s,y,u)| ≤ L f (|t− s|+ |x− y|).(H f )

Under mild hypotheses (H f ), given a measurable control function u∈U (t,+∞),the control system (162) admits a unique solution which is an absolutely continuousarc defined on (t,+∞). To emphasize the dependence upon control and initial data,we reserve the notation yu

t,x(·) for such trajectory, which is called the state of thecontrol system. In the case of autonomous systems, we assume t = 0 and denote thetrajectory by yu

x(·).

Example 1.1 (Linear systems) Suppose that U ⊆Rm and let A(s) ∈Mn×n(R) andB(s) ∈Mn×m(R) for any s ∈ (t,T ). A linear control system has the structure

y(s) = A(s)y(s)+B(s)u(s) and u(s) ∈U, for a.e. s ∈ (t,T ).

Example 1.2 (Control-affine systems) Let f0, . . . , fm : R×Rn→Rn be given vec-tor fields and write u(s) = (u1(s), . . . ,um(s)). A dynamical system is called control-

affine provided (162) can be written as

y(s) = f0(s,y(s))+m

∑i=1

fi(s,y(s))ui(s) and u(s) ∈U, for a.e. s ∈ (t,T ).

An optimal control problem with fixed final horizon is an optimization problemthat aims at minimizing the following functional

U (t,T ) 3 u 7→∫ T

te−λ s`(s,yu

t,x(s),u(s))ds+ e−λTψ(yu

t,x(T )), (163)

where λ ≥ 0 is the discount factor, `(·) is the running cost and ψ(·) is the final cost.In this chapter the running cost is supposed to satisfy:

(i) ` : R×Rn×U → R is continuous.(ii) ∀R > 0, ∃LR

f > 0,∀x,y ∈ BR, ∀t,s ∈ (−R,R), ∀u ∈U|`(t,x,u)− `(s,y,u)| ≤ LR

` (|t− s|+ |x− y|).(iii) ∃c` > 0, λ` ≥ 1, ∀(x,u) ∈ Rn×U :

0≤ `(x,u)≤ c`(1+ |x|λ`).

(H`)

The problem takes different name according to the data. In particular, we areinterested in two particular problems: the Infinite Horizon problem (T = +∞ andψ ≡ 0) and the Bolza problem (T <+∞ and λ = 0).

Example 1.3 (Quadratic cost) Let Q(s) ∈Mn×n(R) and R(s) ∈Mm×m(R) for anys ∈ (t,T ), and P ∈Mn×n(R). A quadratic optimal control problem is of the form:for any s ∈ (t,T ), y ∈ Rn and u ∈U ⊆ Rm

`(s,y,u) = 〈Q(s)y,y〉+ 〈R(s)u,u〉 and ψ(y) = 〈Py,y〉.

The value function is the mapping that associates any initial time t and initialposition x with the optimal value of the problem (163). In the case of the infinitehorizon we are only considering the autonomous case. Hence the value function

v(x) = infu∈U (0,+∞)

∫∞

0e−λ s`(yu

x(s),u(s))ds. (164)

On the other hand, for the Bolza problem the value function is

v(t,x) = infu∈U (t,T )

∫ T

t`(s,yu

t,x(s),u(s))ds+ψ(yut,x(T ))

(165)

and in addition it satisfies the final condition:

v(T,x) = ψ(x), ∀x ∈ Rn. (166)

Furthermore, in the formulation of (163) we may also consider that the final hori-zon is not fixed, which leads to a more general class of optimal control processes.Among these, the most relevant for the exposition is the so-called Minimum timeproblem to reach a given target Θ ⊆ Rn. In this case we write the value function asTΘ (·) and name it the minimum time function, which is given by

TΘ (x) = infu∈U (0,+∞)

T ≥ 0 | yux(T ) ∈Θ .

This function satisfies by definition the condition TΘ (x) = 0 at any x ∈Θ . We re-mark that this case can be viewed as an Infinite Horizon problem with λ = 0 and`(y,u)≡ 1.

1.2 Dynamic Programming Principle and HJB equations

The HJB approach for optimal control problems is based in a functional equationknown as the Dynamic Programming Principle. This equation has different formsbased on the issue at hand:

• Infinite Horizon problem: for any x ∈ Rn and τ ∈ (0,+∞)

v(x) = infu∈U (0,τ)

∫τ

0e−λ s`(yu

x(s),u(s))ds+ e−λτ v(yux(τ))

. (167)

• Bolza problem: for any x ∈ Rn and τ ∈ (t,T )

v(t,x) = infu∈U (t,τ)

∫τ

t`(s,yu

t,x(s),u(s))ds+ v(τ,yut,x(τ))

. (168)

• Minimum time problem: for any x ∈ Rn and τ ∈ (0,TΘ (x))

TΘ (x) = infu∈U (0,τ)

τ +TΘ (yu

x(τ)). (169)

Let us assume that the value functions are continuously differentiable functions.Then, some standard calculations yield to the following HJB equation:

• Infinite Horizon problem:

λv(x)+H(x,∇v(x)) = 0, x ∈ Rn, (170)

with H(x, p) := sup−〈 f (x,u), p〉− `(x,u) | u ∈U.• Bolza problem:

−∂tv(t,x)+H(t,x,∇xv(t,x)) = 0, (t,x) ∈ (−∞,T )×Rn,

with H(t,x, p) := sup−〈 f (t,x,u), p〉− `(t,x,u) | u ∈U.

• Minimum time problem:

−1+H(x,∇TΘ (x)) = 0, x ∈ int(domTΘ ),

with H(x, p) := sup−〈 f (x,u), p〉 | u ∈U.However, the value function is rarely differentiable and so solutions to the

Hamilton-Jacobi-Bellman equations need to be understood in a weak sense. Themost suitable framework to deal with these equations is the Viscosity Solutions The-ory introduced by Crandall and Lions in 1983 in their famous paper [52].

We remark that for the finite horizon problem the HJB equation assumes the timedependent form.

Let us now introduce the notion of viscosity solution of the HJ equation

F(x,v(x),∇v(x)) = 0, x ∈Ω (171)

where Ω is an open domain of Rn and the Hamiltonian F = F(x,r, p) is a contin-uous, real valued function on Ω ×R×Rn. Later we will discuss further hypothesison the Hamiltonian. The notion of viscosity solution, allows us to obtain importantexistence and uniqueness results for some equations of the form (171).

Remark 1.1 The equation (171) can depend on time, describing the evolution of asystem. In that case it is defined in the space (t,T )×Ω with T ∈ R and it is

F(t,x,v(t,x),∂tv(t,x),∇xv(t,x)) = 0, t ∈ (0,T ),x ∈Ω

v(T,x) = v0(x), x ∈Ω .(172)

for a final time Dirichlet boundary condition v0(x).

1.3 Viscosity solutions

It is well know that equation (171) is in general not well-posed in the classicalsense. That is, it is possible to show several examples in which no continuously dif-ferentiable solution exists. Furthermore, it is possible to construct an infinite numberof almost everywhere differentiable solutions. For example, let us consider a simple1-dimensional Eikonal equation with a Dirichlet boundary condition, that is

|∇v(x)|= 1, x ∈ (−1,1)v(x) = 0, x =±1 (173)

This equation admits an infinite number of almost everywhere differentiable so-lutions (see Fig. 5). The theory of viscosity solutions was developed in order toovercome these problems. It gives a way to get uniqueness of the solution and insome cases also to select the solution that has a physical interpretation.

Fig. 5: Multiple a.e. differentiable solutions of the eikonal equation (173).

Definition 1. A continuous function v : Ω→R is a viscosity solution of the equation(171) if the following conditions are satisfied:

• for any test function φ ∈C1(Ω), if x0 ∈Ω is a local maximum point for v−φ ,then

F(x0,v(x0),∇φ(x0))≤ 0 (viscosity subsolution)

• for any test function φ ∈C1(Ω), if x0 ∈ Ω is a local minimum point for v−φ ,then

F(x0,v(x0),∇φ(x0))≥ 0 (viscosity supersolution)

Remark 1.2 The notion of viscosity solution can also be extended to the case oflower semicontinuous value functions; for instance if ψ is only lower semicontin-uous in the Bolza problem. In this case the solutions are usually called bilateralviscosity solutions; see for instance [18, Chapter 5.5]. We will develop this frame-work more deeply later.

The motivation for the terminology viscosity solutions is that this kind of solutioncan be recovered as the limit function v = limε→0+ vε where vε ∈ C2(Ω) is theclassical solution of the parabolic problem

−ε∆vε +F(x,vε ,∇vε) = 0, x ∈Ω . (174)

Under some technical assumptions vε exists and converges locally uniformly to theviscosity solution v. This method is named vanishing viscosity, and it is the originalidea behind this notion of solution proposed by Crandall and Lions in [52].

Remark 1. Despite we focus more on the time-independent case, the same definitionand results as in the following could be shown in the time-dependent framework ofthe form (172). To see that it is sufficient to make the standard transformation

y = (x, t) ∈Ω × [0,T ]⊆ Rn+1, F(y,r,q) = qn+1 +F(x,r,(q1, ...,qn)) (175)

where q = (q1, ...,qn+1) ∈ Rn+1.

We present some comparison results between viscosity sub- and supersolutions.As simple corollary, each comparison result produces a uniqueness theorem for theassociated Dirichlet problem. In the following of the section we assume F of theform F(x,r,q) = ar+H(x,q) where the positive constant a (possibly zero) will bespecified in each case.

Theorem 1. Let Ω be a bounded open subset of Rn. Assume that v1, v2 ∈C(Ω) are,respectively, viscosity sub- and supersolution of

v(x)+H(x,∇v(x)) = 0, x ∈Ω (176)

andv1 ≤ v2 on ∂Ω . (177)

Assume also that H satisfies

|H(x, p)−H(y, p)| ≤ ω1(|x− y|(1+ |p|)), (178)

for x,y ∈ Ω , p ∈ Rn, where ω1 is a modulus, that is ω1 : [0,+∞)→ [0,+∞) iscontinuous non decreasing with ω1(0) = 0. Then v1 ≤ v2 in Ω .

The proof of the previous theorem can be found in [18] Chapter II, Theorem 3.1.

Theorem 2. Assume that v1,v2 ∈ C(Rn)∩L∞(Rn) are, respectively, viscosity sub-and supersolution of

v(x)+H(x,∇v(x)) = 0, x ∈ Rn. (179)

Assume also that H satisfies (178) and

|H(x, p)−H(x,q)| ≤ ω2(|p−q|), for all x, p,q ∈ Rn. (180)

where ω2 is a modulus. Then v1 ≤ v2 in Rn.

Remark 2. Theorem 2 (for the proof we refer to [18] Chapter II, Theorem 3.5.)can be generalized to cover the case of a general unbounded open set Ω ⊂ Rn.Moreover, the assumptions v1,v2 ∈ C(Rn)∩ L∞(Rn) can be replaced by v1,v2 ∈UC(Rn) (uniformly continuous).

A comparison result for the more general case

H(x,∇v(x)) = 0, x ∈Ω (181)

can be stated if we assume the convexity of H with respect to the p variable. Thisassumption plays a key role in many theoretical results.

Theorem 3. Let Ω be a bounded open subset of Rn. Assume that v1,v2 ∈C(Ω) are,respectively, viscosity sub- and supersolution of (181) with v1 ≤ v2 on ∂Ω . Assumealso that H satisfies (178) and the two following conditions

• p→ H(x, p) is convex on Rn for each x ∈Ω ,• there exists φ ∈C(Ω)∩C1(Ω) such that φ ≤ v1 in Ω

and supx∈B H(x,Dφ(x))< 0 for all B⊂Ω .

Then v1 ≤ v2 in Ω .

The proof of this result can be found in [18] Chapter II, Theorem 3.9.

1.4 The Eikonal equation

The classical model problem for (181) is the Eikonal equation on geometric optics

c(x)|∇v(x)|= 1, x ∈Ω (182)

Theorem 3 applies to the eikonal equation (182) whenever c(x) ∈ Lip(Ω) and it isstrictly positive. In fact the second condition of Theorem 3 is satisfied by takingφ(x)≡min

Ωv1.

It is easy to prove that the distance function from an arbitrary set S ⊆ Rn, S 6= /0defined by

dS(x) = dist(x,S) := infz∈S|x− z|= min

z∈S|x− z| (183)

is continuous in Rn. Moreover, for smooth ∂S, it satisfies in the classical sense theequation (182) in Rn \S for c(x)≡ 1.

For a general set S, it can be shown that the function dS is the unique viscositysolution of

|∇v(x)|= 1, x ∈ Rn \Sv(x) = 0, x ∈ ∂S

(184)

where we impose Dirichlet conditions (distance is zero) on the boundary of the setS.

Remark 3. If we consider the eikonal equation in the form |∇v(x)| = g(x) whereg is a function vanishing at least in a single point in Ω , then the uniqueness re-sult does not hold. This situation is referred to as degenerate eikonal equation. Itcan be proved that in this case many viscosity or even classical solution may ap-pear. Consider for example the equation |v′| = 2|x| for x ∈ (−1,1) complementedby Dirichlet boundary condition v = 0 at x =±1. It is easy to see that v1(x) = x2−1and v2(x) = 1−x2 are both classical solutions. The case of degenerate eikonal equa-tions has been studied by Camilli and Siconolfi [36] and numerically by Camilli andGrune in [35].

The minimum time problem

Let us come back to the minimum time problem to reach a given closed targetΘ ⊂ Rn. Note that a priori nothing implies that the end-point constraint

yux(T ) ∈Θ

will be satisified for any x ∈ Rn and u ∈ U (0,T ). This implies that the minimumtime function may not be well defined in some regions of the space, which in math-ematical terms means that TΘ (x) = +∞.

Definition 2. The reachable set is RΘ := x ∈ Rn : TΘ (x) < +∞, i.e. it is the setof starting points from which it is possible to reach the target Θ .

Note that the reachable set depends on the target, the dynamics and on the set ofadmissible controls and it is not a datum in our problem. The importance of thereachable set is reflected by the following result

Proposition 1. If RΘ \Θ is open and TΘ ∈ C(RΘ \Θ), then TΘ is a viscositysolution of

maxu∈U− f (x,u) ·∇T (x)−1 = 0, x ∈RΘ \Θ (185)

A more detailed proof of the result above can be found in [18] Chapter IV, Proposi-tion 2.3. The uniqueness of the solution can be proved under an additional conditionof small time local controllability (for further details we refer to Section 2 and [18]Chapter IV, Theorem 2.6). Natural boundary conditions for (185) are

TΘ (x) = 0 x ∈ ∂Θ

limx→∂RΘ

TΘ (x) = +∞. (186)

In order to archive uniqueness of the viscosity solution of equation (185) is usefulan exponential transformation named Kruzkov transform

v(x) :=

1− e−TΘ (x) if T (x)<+∞

1 if T (x) = +∞(187)

It is easy to check (at least formally) that if TΘ is a solution of (185) then v is asolution of

v(x)+maxu∈U− f (x,u) ·∇v(x)−1 = 0, x ∈ Rn \Θ . (188)

This transformation has many advantages:

• The equation for v has the form (176) so that we can apply the uniqueness resultalready introduced in this chapter.

• v takes value in [0,1] whereas TΘ is generally unbounded (for example if fvanishes in some points) and this helps in the numerical approximation.

• The domain in which the equation has to be solved is no more unknown.• One can always reconstruct TΘ and RΘ from v by the relations

TΘ (x) =− ln(1− v(x)), RΘ = x : v(x)< 1 .

Optimal feedback and trajectories

Let us consider for simplicity the minimum time problem. As mentioned above, thefinal goal of every optimal control problem is to find the a control u∗ ∈U (0,TΘ (x))that minimizes

TΘ (x) = infτ > 0 | yu∗x (τ) ∈Θ (189)

The next theorem shows how to compute u∗ in feedback form, i.e. as a function ofthe state y(t). This form turns out to be more useful than open-loop optimal controlwhere u∗ depends only on time t. In fact, the feedback control leads the state to thetarget even in presence of perturbations and noise.

Theorem 4. Assume that a function TΘ ∈C1(RΘ \Θ) be the unique viscosity so-lution of (185) and suppose that the mapping κ(x) defined below is continuous

κ(x) := argmaxu∈U

− f (x,u) ·∇TΘ (x)

, x ∈RΘ \Θ . (190)

Let y∗(t) be the solution ofy∗(t) = f (y∗(t),κ(y∗(t))), t > 0y∗(0) = x (191)

Then, u∗(t) = κ(y∗(t)) an optimal control.

The result above is related to some regularity issues. The regularity of the valuefunction in the minimum time case is a delicate issue and it was discussed andstudied in several works. A detailed presentation of the problem can be found in theChapter IV of [18] and in [39, 17]. More recent works are [38], [70], [76].

1.5 Semi-Lagrangian approximation for Hamilton-Jacobiequations

We recall how to obtain a convergent numerical scheme for Hamilton-Jacobi equa-tions. As a model we will consider the infinite horizon problem. In Section 3 it isconsidered instead the time dependent case, with the description of a finite differ-ences filtered scheme. In the current section we introduce a scheme for the station-ary case in semi-Lagrangian form. In such an approach the numerical approximationis based on a time-discretization of the original control problem via a discrete ver-sion of the Dynamical Programming Principle. Then, the functional equation for thetime-discrete problem is “projected” on a grid to derive a finite dimensional fixedpoint problem. We also show how to obtain the same numerical scheme by a directdiscretization of the directional derivatives in the continuous equation. Note that thescheme we study is different to that obtained by Finite Difference approximation.In particular, our scheme has a built-in up-wind correction.

Semi-discrete scheme

The aim of this section is to build a numerical scheme for equation (170). In orderto do this, we first make a discretization of the autonomous version of the originalcontrol problem (162) introducing a time step h = ∆ t > 0.

We obtain a discrete dynamical system associated to (162) just using any one-step scheme for the Cauchy problem. A well known example is the explicit Eulerscheme which corresponds to the following discrete dynamical system

yn+1 = yn +h f (yn,un), n = 1,2, ...y0 = x (192)

where yn = y(tn) and tn = nh. We will denote by yx(n;un) the state at time nhof the discrete time trajectory verifying (192). We also replace the cost functional(163) by its discretization by a quadrature formula (e.g. the rectangle rule). In thisway we get a new control problem in discrete time. The value function vh for thisproblem (the analogous of (164)) satisfies the following proposition

Proposition 2 (Discrete Dynamical Programming Principle). We assume that

∃M > 0 : |`(x,u)| ≤M for all x ∈ Rn,u ∈U (193)

then vh satisfies

vh(x) = minu∈U(1−λh)vh(x+h f (x,u))+ `(x,u), x ∈ Rn. (194)

This characterization leads us to an approximation scheme, at this time, discreteonly on the temporal variable.

Under the usual assumptions of regularity on f and ` (Lipschitz continuity,boundedness on uniform norm) and for λ > L f as in (H f ), the family of functionsvh is equibounded and equicontinuous, then, by the Ascoli-Arzela theorem we canpass to the limit and prove that it converges locally uniformly to v, value functionof the continuous problem, for h going to 0. Moreover, the following estimate holds(cf. i.e. [57])

||v− vh||∞ ≤Ch12 . (195)

Fully discrete scheme

In order to compute an approximate value function and solve (194) we have to makea further step: a discretization in space. We need to project equation (194) on a finitegrid. First of all, we restrict our problem to a compact subdomain Ω ⊂Rn such that,for h sufficiently small

x+h f (x,u) ∈Ω ∀x ∈Ω ∀u ∈U. (196)

We build a regular triangulation of Ω denoting by X the set of its nodes xi, i ∈ I :=1, ...,N and by S the set of simplices S j, j ∈ J := 1, ...,L. Let us denote by k thesize of the mesh i.e. k = ∆x := max jdiam(S j). Note that one can always decideto build a structured grid (e.g. uniform rectangular meshes) for Ω as it is usualfor Finite Difference scheme, although for dynamic programming/semi-Lagrangianscheme is not an obligation. Main advantage of using structured grid is that one caneasly find the simplex containing the point xi +h f (xi,a) for every node xi and everycontrol a ∈ A and make interpolations.

Now we can define the fully discrete scheme simply writing (194) at every nodeof the grid. We look for a solution of

vkh(xi) = min

u∈U(1−λh)I[vk

h](xi +h f (xi,u))+h`(xi,u), i = 1, ...N (197)

I[vkh](x) = ∑

jλ j(x)vk

h(x j), 0≤ λ j(x)≤ 1, ∑j

λ j(x) = 1 x ∈Ω .

in the space of piecewise linear functions on Ω . Let us make a number of remarkson the scheme above:

i) The function u is extended on the whole space Ω in a unique way by linearinterpolation, i.e. as a convex combination of the values of vk

h(xi), i∈ I. It shouldbe noted that one can choose any interpolation operator. A study of variousresults of convergence under various interpolation operators are contained in[61].

ii) The existence of (at least) one control u∗ giving the minimum in (197) relies onthe continuity of the data and on the compactness of the set of controls.

iii) By construction, u belongs to the set

W k := w : Q→ [0,1] such that w ∈C(Q), Dw = costant in S j, j ∈ J (198)

of the piecewise linear functions.

We map all the values at the nodes onto a N-dimensional vector V = (V1, ...,VN) sothat we can rewrite (197) in a fixed point form

V = G(V ) (199)

where G : RN×RN is defined componentwise as follows

[G(V )]i := minu∈U

[(1−λh)∑

jλ j(xi +h f (xi,u))Vj+h`(xi,u)

]i

(200)

The proofs of the following results are rather direct with the use of the Banach’sfixed point theorem.

Theorem 5. For a λ > 0 and a h small enough to verify |1−λh|< 1, the operatorG defined in (200) has the following properties:

• G is monotone, i.e. U ≤V implies G(U)≤ G(V );• G is a contraction mapping in the uniform norm ||W ||∞ =maxi∈I |Wi|, β ∈ (0,1)

||G(U)−G(V )||∞ ≤ β ||U−V ||∞

Proposition 3. The scheme (197) has a unique solution in W k. Moreover, the solu-tion can be approximated by the fixed point sequence

V (n+1) = G(V (n)) (201)

starting from the initial guess V (0) ∈ RN .

There is a global estimate for the numerical solution ([18] Appendix A, Theorem1.3., see also [56, 20]). Other more recent results are [66, 98]

Theorem 6. Let v and vkh be the solutions of (170) and (197). Assume the Lipschitz

continuity and the boundness of f and `, moreover assume condition (196) and thatλ > L f , said L f , L` Lipschitz constant of the function f and `, then

||v− vkh||∞ ≤Ch

12 +

L`

λ (λ −L f )

kh. (202)

Time-optimal control

At now, we introduce a numerical approximation for the solution of the minimumtime problem.

After a discretization of the dynamics as in the previous section, let us define thediscrete analogue of admissible controls

U h :=unn∈N : un ∈U for all n

and that of the reachable set

RΘ ,h :=

x ∈ Rn : there exists un ∈U h and n ∈ N such that yx(n;un) ∈Θ

.

Let us also define

nh(x,un) :=

minn ∈ N : yx(n;un) ∈Θ x ∈RΘ ,h

+∞ x /∈RΘ ,h

andNh(x) := inf

un∈U hnh(x,un).

The discrete analogue of the minimum time function T (x) is Th(x) := hNh(x)

Proposition 4 (Discrete Dynamical Programming Principle). Let h> 0 fixed. Forall x ∈ Rn, 0≤ m < Nh(x) (so that x /∈Θ )

Nh(x) = infum∈U h

m+Nh(yx(m;um)) . (203)

The proof of the Proposition 4 can be found in [17]. Choosing m = 1 in (203) andmultiplying by h, we obtain the time-discrete Hamilton-Jacobi-Bellman equation

Th(x) = minu∈UTh(x+h f (x,u))+h. (204)

Note that we can obtain the equation (204) also by a direct discretization of equation(185):

0 = maxu∈U− f (x,u) ·∇T (x)−1≈max

u∈U

−Th(x+h f (x,u))−Th(x)

h

−1

and, multiplying by h,

−minu∈UTh(x+h f (x,u))−Th(x)−h =−min

u∈UTh(x+h f (x,u))+Th(x)−h = 0.

As in continuous problem, we apply the Kruzkov change of variable

vh(x) = 1− e−Th(x).

Note that, by definition, 0 ≤ vh ≤ 1 and vh has constant values on the set of initialpoints x which can be driven to Θ by the discrete dynamical system in the samenumber of steps (of constant width h). This shows that vh is a piecewise constantfunction. By (204) we easly obtain that vh satisfies

vh(x) = minu∈Uβvh(x+h f (x,u))+1−β .

where β = e−h and we have the following

Proposition 5. vh is the unique bounded solution ofvh(x) = min

u∈Uβvh(x+h f (x,u))+1−β x ∈ Rn \Θ

vh(x) = 0 x ∈ ∂Θ(205)

Note that the time step h we introduced for the discretization of the dynamical sys-tem is still present in the time-independent equation (205) and then it could be in-terpreted as a fictitious time step.

Definition 3. Assume ∂Θ smooth. We say that Small Time Local Controllability(STLC) assumption is verified if

for any x ∈ ∂Θ , there exists u ∈U such that f (x, u) ·η(x)< 0 (206)

where η(x) is the exterior normal to Θ at x.

We have the next important result (refer to [17] for the proof):

Theorem 7. Let Θ be compact with nonempty interior. Then under our assumptionson f and ST LC, vh converges to v locally uniformly in Rn for h→ 0+.

Just like in the previous case, we project equation (205) on a finite grid. First ofall, we restrict our problem to a compact subdomain Q containing Θ and we builda regular triangulation of Q with: X the nodes xi, i ∈ I := 1, ...,N, S the set ofsimplices S j, j ∈ J := 1, ...,L, k the size of the mesh.

We will divide the nodes into three subsets.

IΘ =i ∈ I : xi ∈ΘIin =i ∈ I \ IΘ : there exists u ∈U such that xi +h f (xi,u) ∈ Q

Iout =i ∈ I \ IΘ : xi +h f (xi,u) /∈ Q for all u ∈U

Now we can define the fully discrete scheme writing (205) on the grid adding theboundary condition on ∂Q

vkh(xi) = minu∈Uβ I[vk

h](xi +h f (xi,u))+1−β i ∈ Iinvk

h(xi) = 0 i ∈ IΘvk

h(xi) = 1 i ∈ Iout

(207)

I[vkh](x) = ∑

jλ j(x)vk

h(x j), 0≤ λ j(x)≤ 1, ∑j

λ j(x) = 1 x ∈ Q.

The condition on Iout assigns to those nodes a value greater than the maximum valueinside Q. It is like saying that once the trajectory leaves Q it will never come backto Θ (which is obliviously false). Nonetheless the condition is reasonable since wewill never get the information that the real trajectory (living in the whole space) canget back to the target unless we compute the solution in a larger domain containingQ. In general, the solution will be correct only in a subdomain of Q and it is greaterthan the real solution everywhere in Q. This means also that the solution we getstrictly depends on Q. Also in this case, by construction, vk

h belongs to the set

W k := w : Q→ [0,1] such that w ∈C(Q),Dw = constant in S j, j ∈ J (208)

of the piecewise linear functions.We map all the values at the nodes onto a N-dimensional vector V = (V1, ...,VN)

so that we can rewrite (207) in a fixed point form

V = G(V ) (209)

where G is defined componentwise as follows

[G(V )]i :=

minu∈Uβ ∑ j λ j(xi +h f (xi,u))Vj+1−β i ∈ Iin

0 i ∈ IΘ1 i ∈ Iout

(210)

It is possible to prove, using again for direct computation and for a fixed pointargument, the following results.

Theorem 8. The operator G defined in (210) has the following properties:

• G is monotone, i.e. U ≤V implies G(U)≤ G(V );• G : [0,1]N → [0,1]N;• G is a contraction mapping in the uniform norm ||W ||∞ = maxi∈I |Wi|, i.e. for a

constant β > 0||G(U)−G(V )||∞ ≤ β ||U−V ||∞

Proposition 6. The scheme (207) has a unique solution in W k. Moreover, the solu-tion can be approximated by the fixed point sequence

V (n+1) = G(V (n)) (211)

starting from the initial guess V (0) ∈ RN .

A typical choice for V (0) is

V (0)i =

0 xi ∈Θ

1 elsewhere (212)

which guarantees a monotone decreasing convergence to the fixed point V ∗.

2 HJB approach for State-Constrained Optimal ControlProblems

by Christopher Hermosilla1 and Athena Picarelli2

In this section, we present two recent approaches to study the value function ofstate-constrained optimal control problems. The first approach is based on [82]. Inthis work, the authors provide a characterization of the value function in terms of aninduced stratification of the state constraints set. The main contribution and noveltyof this study is that it provides a set of inequalities that completes the constrainedHJB equation introduced by Soner in 1986. Furthermore, this technique, in particu-lar the stratified structure behind the state-constraints set, can be also used to studyoptimal control problems on networks.

The second approach is based on [10]. This article is devoted to study the epi-graph of the value function via an exact penalization technique. The authors are ableto show that, under fairly general assumptions, it is always possible to compute thevalue function via an auxiliary augmented optimal control problem without stateconstraints. The main advantage of this approach is that it provides a way to com-pute the value function using the available methods designed for optimal controlproblem with unrestricted state spaces and Lipschitz value functions, which leads toa constructive way for determining the value function and to its numerical approxi-mation.

2.1 Preliminaries on state-constrained optimal control problems.

It is natural in models to include constraints on the state space to reflect the real-world restrictions. Consequently, we are interested in controlled trajectories that liein a closed set K (K ⊂Rn, K non empty), called the state constraints set. Hencewe require that

y(s) ∈K , ∀s ∈ [t,T ). (213)

For a given final horizon T ∈ R∪ +∞, initial time t ∈ (−∞,T ) and initialposition x ∈K , the set of controls which make a solution to (162) feasible on Kis called the set of admissible controls and is defined by

Ux(t,T ) :=

u ∈U (t,T ) | yut,x(s) ∈K , ∀s ∈ [t,T )

.

1 Department of Mathematics, Louisiana State University, United States.e-mail: chermosilla@lsu.edu2 Mathematical Institute, University of Oxford, United Kingdom.e-mail: picarelli@maths.ox.ac.uk

The main difference between the constrained (K is strictly contained in RN)and unconstrained case (K = RN), lies in the structure of the set of the admissiblecontrols Ux(t,T ). Indeed, for the unconstrained case this set is essentially U (t,T )and at any point on the domain and, by contrast, in the constrained case Ux(t,T )may vary from point to point in a very complicated way (possibly being empty atsome points)

This fact has two important consequences which make the study of the valuefunction for the state-constrained problems more delicate to treat.

• The value function may not be continuous nor real-valued even for very regulardata.• The value function is a viscosity solution of the HJB equation only on the inte-

rior of its domain.• The unique information about value function on the boundary comes from the

supersolution inequality.

We already claimed in Remark 1.2 that the discontinuous case can still be han-dled by mean of the bilateral approach. In this case, convexity assumptions overthe dynamics and running cost are essential to state the lower semicontinuity of thevalue function. Otherwise, we can only work with lower semicontinuous envelopes.Hence, the first point entails technical difficulties that can be treated anyway.

However, without additional compatibility assumptions involving the dynamicsand the state constraints set, there is no known technique that allows to identifythe value function as the unique solution, in a weak sense, of a HJB equation ina determined functional set. This is due mainly to the lack of information on theboundary of the state constraints. In particular, the HJB equation may have manysolutions in a same class of functions, and so no characterization is possible; see forinstance the discussion in [85, 29].

In the nowadays literature there are principally two approaches to deal with thesedifficulties that have drawn the attention of the Control Theory Community. Thefirst one consists in looking for condition in order to ensure that the value functionis uniformly continuous on its whole domain. This approach was started by Soner in[116] and then consecutively studied by many authors [102, 40, 103, 107, 85, 108,118, 50, 109, 69].

The second approach assumes that the value function may not be continuous,but seeks for conditions in order to ensure that the information coming from theinterior of the state contraints reach the boundary. This methodology was introducedby Frankowska and Vinter in [72] and then extended to more general situations in[71, 69].

The technique used in both cases heavily relies upon an approximation argumentknown as the Neighboring Feasible Trajectories (NFT) theorem, which basicallysays that any feasible trajectories can be approximated by a sequence of arcs whichremain in the interior of the state constraints. The approach presented in Section 2.2is also based on an approximation argument, however, of different nature.

2.1.1 Constrained Viscosity Solutions.

Early in this section we said that the value function is only a viscosity solution of theHJB equation on the interior of its domain and a supersolution in its whole domain.In other words, the value function, if continuous, is a constrained viscosity solutionof the HJB equation in the following sense:

Definition 2.1 A function v∈C(Ω)

is a constrained viscosity solution of HJ equa-tion if it is a viscosity subsolution in Ω and it is a viscosity supersolution in Ω , thatis, for any φ ∈ C 1(Rn) such that x ∈Ω is a local minimizer of v−φ relative to Ω ,we have

F(x,v(x),∇φ(x))≥ 0.

It was shown indeed by Soner in [116], that if the value function is uniformlycontinuous on its domains, then it is the unique constrained viscosity solution of theHJB equation, for a fairly wide class of constraints sets; see also [18, Chapter 4.5].

However, this result turned out the quest into finding sufficient conditions to en-sure the uniform continuity of the Value function. Here is when the compatibilityassumptions start playing a role. The first one that appeared in the literature is theso-called Inward Pointing Condition (IPC). It was introduced also by Soner for thecase when K is a N-dimensional embedded submanifold of RN with boundary andfor autonomous dynamics. In this case this condition can be stated as follows:

infu∈U〈 f (x,u),next(x)〉< 0, ∀x ∈ ∂K .

It has been object of subsequence extension to less restrictive situations; see forinstance [118, 50, 69]. The IPC has as main goal to provide a NFT theorem that toensure the continuity of the value function. From a geometrical point of view, it saysthat at each point of ∂K , there exists a controlled vector field pointing into K ; seeFigure 6 for a graphic example.

The IPC is only a sufficient condition for ensuring the continuity of the valuefunction, however, it is not difficult to construct an instance in which the IPC failsonly at one point and the value function is only lower semicontinuous; see [18,Example 5.3 page 276]. Furthermore, the IPC is not a generic property and may faileven for very simple cases as the following situation shows.

Example 2.1 Consider a mechanical system governed by a second order equationfor which the velocity and the position are bounded:

y = ϕ(y, y,u) y ∈ [a,b], y ∈ [c,d]

using the transformation y1 = y and y2 = y the systems can be rewrite as:(y1y2

)= f (y1,y2,u) :=

(y2

ϕ(y1,y2,u)

)(y1,y2) ∈K0 = [a,b]× [c,d].

In particular,

K

f(x1, u1)

f(x2, u2)

f(x3, u3)

Fig. 6: Example of Inward Pointing Condition.

〈 f (x1,x2,u),next(a,x2)〉=−x2, ∀x2 ∈ (c,d),∀u ∈U.

Notice that this quantity does not depend on the control nor in the initial dynamic ϕ

but only on the sign of x2, and so, for some values of x2 the dynamics will point intoK0 and for others it will point into R2 \K0. A similar analysis can be done for theboundary points contained in b× (c,d).

2.1.2 Bilateral Viscosity Solutions

We have seen that in many cases we can not expect the value function to be con-tinuous nor to be real-valued. This last situation may occur for instance if the set ofadmissible control is empty at some points or if the final cost incorporates implicitlya final constraint. So, we are encouraged to consider discontinuous value functionson HJB methodology.

The notion of constrained viscosity solution can be extended to a lower semicon-tinuous context in the following

Definition 2.2 A function v : Ω →R∪+∞ is a constrained bilateral viscosity so-lution of HJ equation if it is a viscosity supersolution in Ω in the sense of Definition2.1 and it is a viscosity supersolution in Ω of

−F(x,v(x),∇v(x)) = 0 x ∈Ω . (214)

In the not-continuous context is often to find the HJB equation expressed in termsof subdifferentials.

Definition 2.3 Let v : RN → R∪+∞ be a lower semicontinuous function and letx ∈ domv. A vector ζ ∈RN is called a viscosity subgradient of v at x provided thereexists a continuous function φ : RN→R differentiable at x such that ∇φ(x) = ζ and

v−φ attains a local minimum at x. The set of all viscosity subgradients of v at x isdenoted by ∂V ω(x).

So, with these definitions in hand, we have that v is a constrained bilateral vis-cosity solution if and only if:

F(x,v(x),ζ ) = 0, ∀x ∈Ω , ∀ζ ∈ ∂V v(x).

F(x,v(x),ζ )≥ 0, ∀x ∈ ∂Ω , ∀ζ ∈ ∂V v(x).

Furthermore, Vinter and Frankowska in [72] showed, for convex dynamics, thatthe value function is the unique lower semicontinuous function that is a constrainedbilateral viscosity solution of the HJB equation that satisfies

liminft→T−, yintK−→ x

v(t,y) = v(T,x).

provided a compatibility assumption, called the Outward Pointing Condition (OPC),holds. This condition in the case that K is a N-dimensional embedded submanifoldof RN with boundary can be stated as follows:

supu∈U〈 f (t,x,u),next(x)〉> 0, for all (t,x) ∈ (−∞,T ]×∂K .

This result was extended to a much larger class of state constraints in [69] where theauthors also studied the continuous case using an IPC.

On the other hand, since OPC is of similar nature that the IPC, it is not difficultto see that in Example 2.1, where we have exhibited that the IPC fails, the OPC alsofails. Furthermore, the OPC can be seen as an IPC for the backward dynamics andso it is not a generic property either.

2.2 Stratified approach for the HJB equations’

In this section we present the first approach announced at the beginning of the chap-ter. The theory we develop here aims to characterize the value function in termsof a bilateral HJB equation. We only focus on the infinite horizon problem withautonomous dynamics and running cost.

The class of control problems we are considering do not necessarily satisfy anyqualification hypothesis such as the pointing conditions. Nevertheless, we do as-sume a compatibility assumption between dynamics and state-constraints, however,of a different nature.

For sake of simplicity, the dynamics and the running cost are assume to be inde-pendent of time. We recall that in this case the value function is given by

v(x) := inf∫

∞

0e−λ t`(yu

x(t),u(t))dt∣∣∣∣ u ∈Ux

, ∀x ∈K ,

and the HJB equation takes the form

λv(x)+H(x,∇v(x)) = 0 x ∈K .

Throughout this section we assume that the control space U is compact subsetof Rm in addition to (H f ) and (H`). Note that the fact Lipschitz continuity of thedynamics implies that it also has linear growth, that is

∃c f > 0 such that ∀x ∈K , max| f (x,u)| : u ∈U ≤ c f (1+ |x|).

Let x ∈K and u ∈Ux. By (H f ), the control system (162) has a solution definedon [0,+∞) which is uniquely determined by x and u, and as we claimed earlier, itis denoted by yu

x . Furthermore, by the Gronwall Lemma and (H f ), each solution to(162) satisfies:

1+ |yux(t)| ≤ (1+ |x|)ec f t ∀t ≥ 0; (215)

|yux(t)− x| ≤ (1+ |x|)(ec f t −1) ∀t ≥ 0; (216)

|yux(t)| ≤ c f (1+ |x|)ec f t for a.e. t > 0; (217)

Moreover, by (H`) and since λ`≥ 1, the cost along trajectories satisfies the followingbound

`(yux(t),u(t))≤ c`(1+ |x|)λ`eλ`c f t , for a.e. t > 0. (218)

Now, when dealing with a distributed cost, it is usual to introduce an augmenteddynamics. For this end, we define

β (x,u) := c`(1+ |x|λ`)− `(x,u) ∀(x,u) ∈ RN×U.

We consider the augmented dynamic G : R×RN ⇒ RN×R defined by

G(τ,x) =(

f (x,u)e−λτ(`(x,u)+ r)

) ∣∣∣∣ u ∈U,0≤ r ≤ β (x,u)

, ∀(τ,x) ∈ R×RN .

It is not difficult to see that by (H`) this set-valued map has compact and nonemptyimages on a neighborhood of [0,+∞)×K . Moreover, in order to state the lowersemicontinuity of the value function we also suppose that

G(·) has convex images on a neighborhood of [0,+∞)×K . (H0)

Remark 2.1 Suppose that U is a convex set of Rm, the dynamical system is control-affine and the running cost is a convex with respect to the control (u 7→ `(x,u) isa convex function). Hence, under this extra structural assumptions, we can readilycheck that (H0) is satisfied.

2.2.1 Stratifiable state constraints.

The main feature of the theory we want to present is that the state constraints set isnot an arbitrary closed set, but it admits a sufficiently regular partition into smoothmanifolds or strata. More precisely,

K is a closed and stratifiable subset of RN . (H1)

We recall that a set is called stratifiable if there exists a locally finite collectionMii∈I of embedded manifolds of RN such that:

• K =⋃

i∈I Mi and Mi∩M j = /0 when i 6= j.• If Mi∩M j 6= /0, necessarily Mi ⊆M j and dim(Mi)< dim(M j).

The class of stratifiable sets is quite broad, it includes sub-analytic and semi-algebraic sets. Also definable sets of an o-minimal structure are stratifiable. In thesecases, the stratifications are even more regular and satisfy the so-called Whitneyproperties; we refer to [119, 87] for more details.

Remark 2.2 As the reader may guess, if a set is stratifiable, there is no a uniquestratification for it and, in fact, there are many others for which (H1) may also hold.However, by the Zorn’s Lemma, there exists is maximal stratification in the senseof inclusion relation, which turns out to be minimal with respect to the number ofstrata. This stratification is unique, after possible permutations among the indices.

Examples of stratifiable sets.

One of first and simpler example is when K is a closed manifold (compact manifoldwithout boundary); For example if K is the torus embedded in R3 as in Figure 7.In this case, the minimal stratification consist of only one stratum, K itself.

Fig. 7: Smooth manifold without boundary.

Consider that intK 6= /0 and ∂K is smooth, as in [116], then (H1) holds withonly two strata, namely, M0 = intK and M1 = ∂K ; for instance if K = B as inFigure 8.

Other example of interest in the nowadays literature is a network configuration.Indeed, in this case, the minimal stratification consists only of edges and junctions.

K

Fig. 8: Smooth manifold with boundary.

Figure 9a shows an example of a network with four edges, M1, . . . ,M4 and a singlejunction M0 := O.

More general networks can also be considered as in Figure 9b where the setK is a network embedded in the space R3. In the example we show, the minimalstratification consists of three branches that are smooth surfaces M1, M2 and M3,and a junction M0 that corresponds to the curve Γ

M1 M2

M3M4

K

O

(a) A network in R2.

M1

M3

M2

Γ

(b) A generalized network in R3.

Fig. 9: Examples of networks.

An important class of sets that admits a stratification as described above is theclass of polytopes in RN ; see Figure 10. In fact, these sets can be decomposed intoa finite number of open convex polytopes of the form:

P =

x ∈ RN

∣∣∣ 〈υk,x〉= αk, υk ∈ RN k = 1, . . . ,n,〈ηk,x〉< αk, ηk ∈ RN k = n+1, . . . ,m

.

K

Fig. 10: A polytope in R3.

2.2.2 Compatibility assumptions

The idea of considering stratifiable sets is to take as much advantage as possible ofthe structure of the set including the thin parts. In the NFT approach this can not bedone because the set of trajectories dwelling on the interior of the state constraintsis required to be dense in the set of all admissible trajectories; in particular it isrequired that int(K ) = K .

We define for each index i∈I , the multifunction Ui : Mi⇒U which correspondsto the intersection between the original control set U and the tangent controls to Mi,that is,

Ui(x) := u ∈U | f (x,u) ∈ TMi(x), ∀x ∈Mi.

This mapping is called the tangent controls to Mi and, as the following propositionshows, it is in general is only upper semicontinuous with possibly empty images.

Proposition 2.1 Assume that (H1) and (H f ) hold. Then, for each i ∈ I , the set-valued map of the tangent control to Mi has compact images and is upper semi-continuous on Mi. Moreover, it can be extended to an upper semicontinuous mapdefined on M i.

Proof. Thanks to the continuity of the dynamics, the images of Ui are closed.Since U is compact, the images of Ui are compact as well. Furthermore, by

Proposition 2.1 we only need to show that Ui has closed graph. Take x∈Mi arbitrary.Let xn ⊆Mi with xn → x and un ⊆U with un → u ∈U such that un ∈Ui(xn).Since Mi is an embedded manifold of RN , TMi(·) has closed graph on Mi. Hence,by continuity of the dynamics, f (xn,un)→ f (x,u) and since f (xn,un)∈ TMi(xn), weget that the multifunction Ui is upper semicontinuous on Mi.

The final conclusion ensues by considering the following limiting map which isby definition upper semicontinuous and coincides with Ui on Mi:

U i(x) :=

u ∈U∣∣∣∣ ∃xn ∈Mi with xn→ x and∀n ∈ N,∃un ∈Ui(xn) so that un→ u

, ∀x ∈M i.

On the other hand, the fact that Ui may have empty images is something we cansimply not avoid without imposing a further hypothesis. For this reason, we assumethat we can find a stratification of the state constraints set in such a way that theset-valued map of tangent control on a stratum has nonempty or empty images allalong Mi.

In view of the convention adopted for the Hausdorff distance, the hypothesis canbe simply written as follows:

dH(Ui(x),Ui(x)) ∈ R, ∀x, x ∈Mi. (H2)

Furthermore, to prove the sufficiency of the HJB equation for the characteriza-tion of the value function we require more regularity. So, we may also assume astrengthen version of (H2), that is:

Each Ui is locally Lipschitz on Mi w.r.t. the Hausdorff distance. (H]2)

Remark 2.3 Similarly as done in Proposition 2.1, Ui can be extended up to M i bydensity. Moreover, if Ui is locally Lipschitz, this extension turns out to be locallyLipschitz as well. So without loss of generality we assume that Ui is defined up toM i in a locally Lipchitz way.

Example 2.2 Consider the following dynamic:(y1y2

)=

(y2u

), u ∈U := [−1,1], y1(t),y2(t) ∈ [−r,r].

Many stratifications are possible for the set of state constraints. Note that (H]2) does

not hold for the minima stratification, which consists of the interior of the set, 4segments and 4 single points.

We represent one particular stratification in Figure 11 for which (H]2) does hold.

In this case, M0 is the interior of the square, M3, M4, M9, M10, M11 and M12 aresegments, and M1, M2, M5, M6, M7 and M8 are single points. We can check easily(H]

2), indeed, U0 =U, Ui = 0 for i = 1, . . . ,4 and Ui = /0 for i = 5, . . . ,12.It is clear in this example that neither the IPC nor the OPC condition is satisfied.

In figure 11, the green zone corresponds to the viable set, that is, the set of pointsfor which Ux 6= /0. Note that in this case, the viable set can be decomposed into aregular stratification which satisfies (H]

2) as well.

Finally, for technical reasons, an extra hypothesis of controllability on certainstrata will be required in order to complete the proof of the main theorem. For thispurpose, we denote by R(x; t) the reachable set at time t, that is, the set of allpossible positions that can be reached by an admissible trajectory. In mathematicalterms

R(x; t) =⋃

u∈Ux

yux(t), ∀x ∈K ,∀t ≥ 0.

M1 M2

M3

M4

M5

M7M8

M6

M9 M10

M11M12

M0

Fig. 11: Stratification of Example 2.2.

On the other hand, we also consider the reachable set through the stratum Mi whichcorresponds to the set of all possible positions that can be reached, at time t, by anadmissible trajectory lying in Mi on the whole interval [0, t):

Ri(x; t) =⋃

u∈Ux

yux(t) | yu

x(s) ∈Mi ∀s ∈ [0, t).

Hence the controllability hypothesis we require can be stated in the followingmanner: for every i ∈I If domUi 6= /0, then ∃εi,∆i > 0 such that

R(x; t)∩M i ⊆⋃

s∈[0,∆it]

Ri(x; t) ∀x ∈Mi, ∀t ∈ [0,εi]. (H3)

This assumption is made in order to approximate trajectories that may switchbetween two or more strata infinitely many times on a short interval (this couldhappen if the set Ai is nonempty).

Note that (H3) is trivial if Mi is an open set or more generally if Mi is of maximaldimension among the strata that forms K . Furthermore, The same remark holdswhenever Ui ≡U .

On the other hand, (H3) is straightforward if Mi is a single point. In this case, ifdomUi 6= /0 then R(x; t)∩M i = M i = Ri(x; t) for any x ∈Mi.

Let us also point out the fact that (H3) can be satisfied under an easy criterion offull controllability condition on manifolds. The most classical assumption of con-trollability is the following: ∀i ∈I with domUi 6= /0

∃ri > 0 such that TMi(x)∩B(0,ri)⊆ f (x,Ui(x)), ∀x ∈Mi. (219)

Indeed, this corresponds to the Petrov condition on manifolds. Hence, by adapt-ing the classical arguments to this setting, we can see that (219) implies the Lipschitzregularity of the minimum time function of the controlled dynamics restricted to themanifold Mi, and so (H3) follows; see for instance [18, Chapter 4.1]. However, letus emphasis on that (219) is only a sufficient condition to satisfy assumption (H3).Indeed, (H3) is still satisfied in some cases where Petrov condition does not hold.For instance, the double-integrator system in Example 2.2 fulfills the requirement(H3) and clearly does not satisfy the Petrov condition (219).

2.2.3 Characterization of the value function

The main aim of this subsection is to characterize the value function of the infinitehorizon problem in terms of a bilateral HJB equation. The definition of solution thatwill be introduced here is based on the classical notion of supersolution and on anew subsolution concept in a stratified sense.

This last demands us to introduce a new Hamiltonian associated with the tan-gential controls. Hence, for each index i ∈ I we define Hi : Mi ×RN ⇒ R, thetangential Hamiltonian on Mi, by

Hi(x,ζ ) = maxu∈Ui(x)

−〈ζ , f (x,u)〉− `(x,u) , ∀x ∈Mi, ∀ζ ∈ RN .

This hamiltonian is continuous under the assumptions we have made.

Proposition 2.2 Suppose that (H1) and (H]2) hold in addition to (H f ) and (H`). Then

for each i ∈I such that domUi 6= /0, Hi(·, ·) is locally Lipschitz on Mi×RN .

Proof. Let R > 0 fixed and let L f and L` be the Lipschitz constants of f and ` onMi ∩B(0,R)×U , respectively. We also set Li as the Lipschitz constant of Ui onMi∩B(0,R).

Fix ζ ∈ RN and take x,y ∈ Mi ∩B(0,R). Since U is compact, there exists ux ∈Ui(x) so that Hi(x,ζ ) =−〈ζ , f (x,ux)〉− `(x,ux). On the other hand, thanks to (H]

2),there exists uy ∈Ui(y) for which |ux−uy| ≤ Li|x− y|. Gathering all the informationwe get that

Hi(x,ζ )−Hi(y,ζ )≤ |ζ || f (y,uy)− f (x,ux)|+ |`(y,uy)− `(x,ux)|≤ |ζ |(L f +L`)(|x− y|+ |ux−uy|)≤ |ζ |(L f +L`)(1+Li)|x− y|

Since, x and y are arbitrary, we can interchange their roles and get that x 7→Hi(x,ζ )is Lipschitz on Mi∩B(0,R).

On the other hand, using (H f ) and a similar argument as above, we get

|Hi(x,ζ )−Hi(x,ξ )| ≤ c f (1+ |x|)|ζ −ξ |, ∀x ∈Mi, ∀ζ ,ξ ∈ B(0,R).

Finally, combining both partial Lipschitz estimations we get the result.

On the other hand, since we are considering possibly not bounded running cost,the value function may not be bounded either. Nonetheless, it still has a controlledgrowth rate

0≤ v(x)≤∫

∞

0e−λ tc`(1+ |x|)λ`eλ`c f tdt ∀x ∈ domv.

Hence, if λ > λ`c f , then v has superlinear growth in the following sense.

Definition 2.4 Let ψ : RN → R∪ +∞ be a given function. We say that ψ hasσ -superlinear growth on its domain if there exists cψ > 0 so that

|ψ(x)| ≤ cψ(1+ |x|)σ ∀x ∈ domψ.

Now we are in position to state the main result of this section.

Theorem 2.1 Suppose that (H0), (H1), (H]2) and (H3) hold in addition to (H f ) and

(H`). Assume also that λ > λ`c f (where λ` > 0 and c f > 0 are the constants givenby (H`) and (H f ), respectively). Then the value function v(·) of the problem (164)is the only lower semicontinuous function with λ`-superlinear growth which is +∞

on RN \K and that satisfies:

λv(x)+H(x,ζ )≥ 0 ∀x ∈K , ∀ζ ∈ ∂V v(x), (220)λv(x)+Hi(x,ζ )≤ 0 ∀x ∈Mi, ∀ζ ∈ ∂V vi(x), ∀i ∈I , (221)

where vi(x) = v(x) if x ∈M i and vi(x) = +∞ otherwise.

Recall that when int(K ) is a nonempty set, it is a smooth manifold of RN andtherefore, there is no loss of generality in assuming that it is one of the stratum, sayM0, of the stratification of K . In that case, H0 = H, and so, the constrained HJBequation proposed by Soner in [116] is included in the set of equations proposed inTheorem 2.1. Furthermore, if domUi = /0 for every i ∈ I \ 0, then (220)-(221)is exactly the constrained HJB equation. Hence, in this sense, this theorem com-pletes the already known theory of the Bellman approach for the problem with stateconstraints solved with the NFT techniques.

Remark 2.4 If for some i ∈I , Mi = x and Ui(x) 6= /0 (this is the case when forinstance K is a network with x being one of the junctions), then f (x,u) = 0 forany u ∈Ui(x) and so Hi(x,ζ ) =−min`(x,u) | u ∈Ui(x) for any ζ ∈ RN . Hence,(221) for this index corresponds the following inequality:

λv(x)≤ minu∈Ui(x)

`(x,u),

which basically says that the cost of leaving the point x should be lower than thecost of remaining at the point.

2.2.4 Proof of the main result

From this point on we start to prove Theorem 2.1. In order to make the proof eas-ier to understand we decompose it into several parts. In particular, we present anintermediate characterization of the value function in terms of the Dynamic Pro-gramming Principle.

In particular, by gathering Propositions 2.4, 2.5, 2.6 and 2.7, and Remarks 2.5and 2.6 the proof of Theorem 2.1 follows immediately.

Lower Semicontinuity and existence of optimal controls.

The next proposition is a classical type of result in optimal control and states the ex-istence of minimizer for the infinite horizon problem under a convexity assumptionover the dynamics. The same argument can used to prove that the value function islower semicontinuous.

Proposition 2.3 Suppose that (H f ), (H`) and (H0) hold. Assume that λ > λ`c f . Ifv(x) ∈ R for some x ∈K then there exists u ∈ Ux a minimizer of (164). Further-more, the value function is lower semicontinuous.

Proof. Let x ∈K such that v(x) ∈ R. This means that for every n≥ 0, there existsa control law un ∈Ux such that:

limn→+∞

∫∞

0e−λ t`(yn(t),un(t))dt = v(x), (222)

where yn is the solution to (162) with the initial condition yn(0) = x. Considerzn(t) = `(yn(t),un(t)) for a.e. t ∈ [0,+∞).

Consider the measure dµ = e−λ tdt and let L1 := L1([0,+∞);dµ) be the Banachspace of integrable functions on [0,+∞) for the measure dµ . Consequently, we de-note by W 1,1 the Sobolev space of L1 functions which have their weak derivativealso in L1.

Let ω : [0,+∞)→ R be given by ω(t) := c f (1+ |x|)ec f t for any t ≥ 0. By (H`),λ > c f because λ` ≥ 1. So, (217) implies that ω(·) is a positive function in L1

which dominates |yn|. Moreover, by (215) or (216) the sequence yn(t) is relativelycompact for any t ≥ 0, hence the hypothesis of theorem [14, Theorem 0.3.4] aresatisfied and so, there exist a function y ∈W 1,1 and a subsequence, still denoted byyn, such that

yn converges uniformly to y on compact subsets of [0,+∞),

yn converges weakly to y in L1.

On the other hand, given that λ > λ`c f and (218) holds, it is not difficult to seethat zn is equi-integrable with respect to dµ , then by the Dunford-Pettis Theoremthere exist a function z ∈ L1 and a subsequence, still denoted by zn, such that znconverges weakly to z in L1.

Let Γ (x) = G(0,x) ⊆ RN ×R for every x ∈ K . Hence, by (H f ) and (H`), Γ

is locally Lipschitz with closed images and by (H0) it has convex images. Thenthe Convergence Theorem [14, Theorem 1.4.1] implies that (y,z) ∈ Γ (y) for almostevery t ≥ 0. Thus, by the Measurable Selection Theorem (see [14, Theorem 1.14.1]),there exist two measurable functions u : [0,+∞)→U and r : [0,+∞)→ [0,+∞) suchthat satisfies

y(t) = f (y(t),u(t)) a.e. t > 0, y(0) = x.

z(t) = `(y(t),u(t))+ r(t) a.e. t > 0.

Since K is closed, y(t) ∈K for every t ≥ 0 and u ∈ Ux. Finally, since φ ≡ 1 ∈L∞([0,+∞);dµ), we have∫

∞

0e−λ t`(y(t),u(t))dt ≤

∫∞

0e−λ tz(t)dt = lim

n→+∞

∫∞

0e−λ tzn(t)dt = v(x).

Therefore, u is a minimizer of the problem.Now let us focus on the lower semicontinuity of v. Let xn ⊆K be a sequence

such that xn→ x. Without lost of generality we assume that |xn| ≤ |x|+1. We needto prove that

liminfn→+∞

v(xn)≥ v(x).

Suppose that there exists a subsequence, we eschew relabeling, so that xn⊆ domv.Otherwise the inequality holds immediately. Then, by the previous part, for any n ∈N there exists an optimal control un ∈Uxn . Let yn the optimal trajectory associatedwith un and xn. Notice that (215), (217) and (218) hold with xn instead of x. Hence,since |xn| is uniformly bounded (|xn| ≤ |x|+1) we can use the same technique as inthe previous part to find that there exists u ∈Ux such that∫

∞

0e−λ t`(yu

x(t),u(t))(t)dt ≤ liminfn→+∞

∫∞

0e−λ t`(yn(t),un(t))dt = liminf

n→+∞v(xn).

Finally, using the definition of the value function we conclude the proof.

Increasing principles along trajectories.

The Dynamic Programming Principle yields to two different monotonoticity proper-ties along admissible arcs. Indeed, the two elementary inequalities that define it canbe interpreted as a weakly decreasing and a strongly increasing principle, respec-tively. These two properties are also known in the literature ([18, Definition 3.2.31]for example) as the super and sub-optimality principles, respectively.

Definition 2.5 Let ϕ : K →R∪+∞ be a lower semicontinuous function, we saythat ϕ is:

i) weakly decreasing for the control system if for all x ∈ domϕ , there exists acontrol u ∈Ux such that

e−λ tϕ(yu

x(t))+∫ t

0e−λ s`(yu

x(s),u(s))ds≤ ϕ(x) ∀t ≥ 0. (223)

ii) strongly increasing for the control system if domU·⊆ domϕ and for any x∈Kand u ∈Ux we have

e−λ tϕ(yu

x(t))+∫ t

0e−λ s`(yu

x(s),u(s))ds≥ ϕ(x) ∀t ≥ 0. (224)

The importance of these definitions relies in the following comparison principlewhich is the fundamental type of result required to single out the value functionamong other lower semicontinuous functions.

Lemma 2.1 Suppose that (H f ), (H`) and (H0) hold, and that λ > λ`c f . Let ϕ :K → R∪+∞ be a lower semicontinuous function with λ`-superlinear growth.

i) If ϕ is weakly decreasing for the control system, then v≤ ϕ .ii) If ϕ is strongly increasing for the control system then v≥ ϕ .

Proof. First of all, note that if λ > λ`c f , then for any function ϕ with λ`-superlineargrowth and for any trajectory y(·) of (162) such that y(t) ∈ domϕ ,

limt→+∞

e−λ tϕ(y(t)) = 0. (225)

Case 1. Suppose ϕ is weakly decreasing for the control system. Let x ∈K , ifx /∈ domϕ then the inequality is trivial. Let x be in domϕ , there exists a controlu ∈Ux such that for all n ∈ N

e−λnϕ(y(n))+

∫∞

0e−λ s`(yu

x(s),u(s))1[0,n]ds≤ ϕ(x).

Therefore, by the Monotone Convergence Theorem, (225) and the definition of thevalue function we obtain the desired inequality v(x)≤ ϕ(x).

Case 2. Suppose ϕ is strongly increasing for the control system and let x ∈K .Assume that v(x) ∈ R, otherwise the result is direct. Let u ∈ Ux be the optimalcontrol associated with (164) and let y be the optimal trajectory associated with uand x. Then

e−λ tϕ(y(t))+

∫ t

0e−λ s`(y(s), u(s))ds≥ ϕ(x) ∀t ≥ 0.

Then by (225), letting t→+∞ we conclude the proof.

In view of the previous comparison lemma we can state an intermediate charac-terization of the value function in terms of the Definition 2.5, which implies par-ticularly that the value function is the unique solution to the functional equation(167).

Proposition 2.4 The value function v(·) is the only lower semicontinuous functionwith λ`-superlinear growth that is weakly decreasing and strongly increasing forthe control system at the same time.

Proof. Recall that the value function v(·) satisfies (167). So, it is weakly decreas-ing and strongly decreasing for the control system. The uniqueness and the growthcondition are consequences of Lemma 2.1.

Characterization of the weakly decreasing principle.

We now prove that the weakly decreasing principle is equivalent to a HJB inequality.This means that a function satisfies (223) if and only if it is a supersolution of theHJB equation. The idea of the proof uses very classical arguments and requires onlystanding assumptions of control theory. A proof for the unconstrained case withbounded value function can be found in [18, Chapter 3.2]. It is worth noting that thecited proof uses purely viscosity arguments.

We restrict our attention to a small class of viscosity subgradients called the prox-imal subgradients, and we then extended the result to all the viscosity subgradientsof the value function by means of a density result.

Definition 2.6 Let ϕ : RN →R∪+∞ be lower semicontinuous. A vector ζ ∈RN

is called a proximal subgradient of ϕ at x if it is a viscosity subgradient and forsome σ > 0 the test g can be taken as:

g(y) := 〈ζ ,y− x〉−σ |y− x|2.

The set of all proximal subgradients at x is denoted by ∂Pϕ(x).In other words, ζ ∈ ∂Pϕ(x) if there exist σ > 0 and δ > 0 such that

ϕ(y)≥ ϕ(x)+ 〈ζ ,y− x〉−σ |y− x|2, ∀y ∈ B(x,δ ).

Proposition 2.5 Suppose that (H f ), (H`) and (H0) hold. Consider a given lowersemicontinuous function with real-extended values ϕ : K → R∪+∞. Then ϕ isis weakly decreasing for the control system if and only if

λϕ(x)+H(x,ζ )≥ 0 ∀x ∈K , ∀ζ ∈ ∂Pϕ(x) (226)

Proof. Let us first prove the implication (⇒).Suppose ϕ is weakly decreasing for the control system. Let x ∈K , if ∂Pϕ(x) =

/0 then (226) holds by vacuity. If on the contrary, there exists ζ ∈ ∂Pϕ(x), thenx ∈ domϕ and there exists u ∈ Ux such that (223) holds. Let us denote y(·) thetrajectory of (162) associated with the control u and x. By the proximal subgradientinequality we have that ∃σ ,δ > 0 such that

ϕ(y(t))≥ ϕ(x)+ 〈ζ ,y(t)− x〉−σ |y(t)− x|2 ∀t ∈ [0,δ ).

Using that y(·) is a trajectory and (223) we get for any t small enough

(1− eλ t)ϕ(x)+∫ t

0[〈ζ , f (y(s),u(s))〉+ `(y(s),u(s))]ds≤ σ |y(t)− x|2

Since f and ` are locally Lipschitz we get

(1− eλ t)

tϕ(x)+

1t

∫ t

0[〈ζ , f (x,u(s))〉+ `(x,u(s))]ds≤ h(t) (227)

where h(t) is such that limt→0+ h(t)= 0. Therefore taking infimum over u∈U insidethe integral and letting t→ 0+ we get (226) after some algebraic steps.

Now, we turn to the second part of the proof (⇐). Let O ⊆ RN+1 be theneighborhood of [0,+∞)×K given by (H0) which we assume is open. Considerψ : [0,+∞)×K ×R→ R∪+∞ defined as

ψ(τ,x,z) =

e−λτ ϕ(x)+ z if x ∈K ,

+∞ otherwise,∀(τ,x,z) ∈ [0,+∞)×K ×R,

and Γ : R×RN×R×R⇒ R×RN×R×R given by

Γ (τ,x,z,w) = 1×G(τ,x)×0, ∀(τ,x,z,w) ∈ R×RN×R×R.

To prove that ϕ is weakly decreasing for the control system let us first show thatfor any γ0 ∈ epiψ , there exists an absolutely continuous arc γ : [0,T )→O×R2 thatsatisfies

γ ∈ Γ (γ) a.e. on [0,T ) and γ(0) = γ0 (228)

such that γ(t) ∈ epiψ for every t ∈ [0,T ), or in term of [120, Definition 3.1],(Γ ,epiψ) is weakly invariant on O×R2. We seek to apply [120, Theorem 3.1(a)].

Note that epiψ is closed because ϕ is lower semicontinuous and Γ has nonemptyconvex compact images on O×R2 because of (H0). Moreover, by (H f ) and (H`), Γ

has closed graph and satisfies the following growth condition:

∃cΓ > 0 so that sup|v| | v ∈ Γ (τ,x,z,w) ≤ cΓ (1+ |x|+ e−λτ |x|λ`).

Therefore, to prove the weak invariance of (Γ ,epiψ) we only need to show that, forS = epiψ , (226) implies

minv∈Γ (χ)

〈η ,v〉 ≤ 0 ∀χ ∈S ∩U, ∀η ∈N PS (χ). (229)

Let (τ,x,z,w) ∈S ∩U , then x ∈ domϕ . Consider η ∈N PS (τ,x,z,w), since this

is the normal cone to an epigraph, we can write η = (ξ ,−p) with p nonnegative.Suppose p > 0 then w = ψ(τ,x,z) and

1p

ξ ∈ ∂Pψ(τ,x,z)⊆ −λe−λτϕ(x)× e−λτ

∂Pϕ(x)×1.

Therefore, for some ζ ∈ ∂Pϕ(x) we have

minv∈Γ (τ,x,z,w)

〈η ,v〉 ≤ minu ∈U,

0≤ r ≤ β (x,u)

pe−λτ(−λϕ(x)+ 〈ζ , f (x,u)〉+ `(x,u)+ r)

≤ pe−λτ minu∈U

(−λϕ(x)+ 〈ζ , f (x,u)〉+ `(x,u)) .

Hence, by (226) we get min〈η ,v〉 | v ∈ Γ (τ,x,z,w) ≤ 0.Suppose now that p = 0, then (ξ ,0) ∈N P

S (τ,x,z,ψ(τ,x,z)) and by Rockafel-lar’s horizontality theorem (see for instance [113]), there exist some sequences(τn,xn,zn) ⊆ domψ , (ξn) ⊆ RN+2 and pn ⊆ (0,∞) such that

(τn,xn,zn)→ (τ,x,z), ψ(τn,xn,zn)→ ψ(τ,x,z),

(ξn, pn)→ (ξ ,0),1pn

ξn ∈ ∂Pψ(τn,xn,zn).

Thus, using the same argument as above we can show

min〈(ξn,−pn),v〉 | v ∈ Γ (τn,xn,zn,ψ(τn,xn,zn)) ≤ 0.

Hence, since Γ is locally Lipschitz, we can take the liminf in the last inequality andsince Γ (τ,x,z,ψ(τ,y,z)) = Γ (τ,x,z,w), we obtain (229).

So, by [120, Theorem 3.1(a)], for every γ0 = (τ0,x0,z0,w0) ∈S ∩O×R2 thereexists an absolutely continuous arc γ(t)= (τ(t),y(t),z(t),w(t)) which lies in O×R2

for a maximal period of time [0,T ) so that (228) holds and

e−λτ(t)ϕ(y(t))+ z(t)≤ w(t) ∀t ∈ [0,T ).

By the Measurable Selection Theorem (see [14, Theorem 1.14.1]), y(·) is a solutionof (162) for some u : [0,T )→U . Also, y(t) ∈ domϕ ⊆K , ∀t ∈ [0,T ).

Moreover, since w(t) = w0 and τ(t) = τ0 + t

z(t) =∫ t

0[e−λ (τ0+s)`(y(s),u(s))+ r(s)]ds, with r(s)≥ 0 a.e.

Notice that γ0 = (0,x,0,ϕ(x)) ∈ epiψ for any x ∈ domϕ , so to conclude theproof we just need to show that T = +∞. By contradiction, suppose T < +∞,then (τ(t),y(t))→ bdryO as t → T−. Nevertheless, since O is a neighborhood of[0,+∞)×K and τ(t) = t and y(t) ∈ K for any t ∈ [0,T ) this is not possible.Therefore, the conclusion follows.

Remark 2.5 Let ϕ be as in Proposition 2.5, then ϕ satisfies (226) if and only if italso satisfies

λϕ(x)+H(x,ζ )≥ 0 ∀x ∈K , ∀ζ ∈ ∂V ϕ(x). (230)

Indeed, since the proximal subgradient is always contained in the viscosity sub-gradient, the sufficient condition follows easily.

On the other hand, (226) holds, then by [33, Proposition 3.4.5] for any x∈ domϕ

and ζ ∈ ∂V ϕ(x) there exist two sequences xn ⊆ domϕ and ζn ⊆ RN such thatxn→ x, ϕ(xn)→ ϕ(x), ζn ∈ ∂Pϕ(xn) and ζn→ ζ . Furthermore

λϕ(xn)+H(xn,ζn)≥ 0 ∀n ∈ N.

Hence, by the compactness of U, passing into the limit in the previous inequality weget (230).

Characterization of Strongly increasing principle (necessity).

Now we show that satisfying inequality (224) implies to be a subsolution of the HJBequation on each stratum.

For sake of the exposition, we recall the definition of the Proximal normal coneand its relation with the proximal subgradient of Definition 2.6. For a further dis-cussion about this topic we refer the reader to [33].

Let S ⊆Rk be a locally closed set and x∈S . A vector η ∈Rk is called proximalnormal to S at x if there exists σ = σ(x,η)> 0 so that

|η |2σ|x− y|2 ≥ 〈η ,y− x〉 ∀y ∈S .

The set of all such vectors η is known as the Proximal normal cone to S at xand is denoted by N P

S (x). If S = epiϕ where ϕ : Rk → R∪ +∞ is a lowersemicontinuous function, then for every x ∈ domϕ , the following relation is valid:

∂Pϕ(x)×−1 ⊆N Pepiϕ(x,ϕ(x)), ∀x ∈ domϕ.

Before entering into the details of the proof, we need to state a fundamentalwhose proof can be found at the end of this section. This proposition implies theexistence of smooth trajectories for a given initial data, namely, initial point andinitial velocity.

Lemma 2.2 Suppose that (H f ), (H`), (H0), (H1) and (H]2) hold. Then, for any i∈I

such that Ui has nonempty images, for every x ∈ Mi and any ux ∈ Ui(x) thereexist ε > 0, a measurable control map u : (−ε,ε) → U, a measurable functionr : (−ε,ε)→ [0,+∞) and a continuously differentiable arc y : (−ε,ε)→ Mi withy(0) = x and y(0) = f (x,ux), such that

y(t) = f (y(t),u(t)) and limt→0−

1t

∫ 0

t

(e−λ s`(y(s),u(s))+ r(s)

)ds =−`(x,ux).

In view of the previous lemma, the necessity part of the strongly increasing prin-ciple can be state as follows.

Proposition 2.6 Suppose that (H f ), (H`), (H0), (H1) and (H]2) hold. Let ϕ : K →

R∪+∞ be a lower semicontinuous function. Suppose that ϕ is strongly increasing

for the control system, then

λϕ(x)+Hi(x,ζ )≤ 0 ∀x ∈Mi, ∀ζ ∈ ∂Pϕi(x), (231)

where ϕi(x) = ϕ(x) if x ∈M i and ϕi(x) = +∞ otherwise.

Proof. First of all note that ζ ∈ ∂Pϕi(x) if and only if ∃σ ,δ > 0 such that

ϕ(y)≥ ϕ(x)+ 〈ζ ,y− x〉−σ |y− x|2 ∀y ∈ B(x,δ )∩M i.

We only show (231) for any (i,x) ∈I ×K such that x ∈ dom∂Pϕi∩Mi∩domUi.Otherwise, the conclusion is direct.

Let (i,x) ∈ I×K as before and take ux ∈Ui(x), it suffices to prove

−λϕ(x)+ 〈ζ , f (x,ux)〉+ `(x,ux)≥ 0, ∀ζ ∈ ∂Pϕi(x). (232)

Let u : (−ε,ε)→U , r : (−ε,ε)→ [0,+∞) and y : (−ε,ε)→Mi be the measurablecontrol and smooth arc given by Lemma 2.2, respectively, where ε > 0 is also givenby this lemma. Let u ∈ Ux, then for all τ ∈ (0,ε) we define the control map uτ :[0,+∞)→U as follows:

uτ(t) := u(t− τ)1[0,τ](t)+ u(t− τ)1(τ,+∞)(t) for a.e. t ∈ [0,+∞).

Let yτ(·) be the trajectory associated with uτ starting from yτ(0) = y(−τ).Clearly, yτ(t) = y(t− τ) for any t ∈ [0,τ].

Moreover, uτ ∈Uy(−τ), so since ϕ is strongly increasing

e−λτϕ(x)+

∫τ

0

(e−λ s`(y(s− τ),u(s− τ))+ r(s− τ)

)ds≥ ϕ(y(−τ)).

Take ζ ∈ ∂Pϕi(x) and τ small enough, so that the proximal subgradient inequalityis valid. Then

ϕ(y(−τ))≥ ϕ(x)+ 〈ζ ,y(−τ)− x〉−σ |y(−τ)− x|2.

Hence,

e−λτ −1τ

ϕ(x)+e−λτ

τ

∫ 0

−τ

(e−λ s`(y(s),u(s))+ r(s)

)ds+

⟨ζ ,

x− y(−τ)

τ

⟩≥h(τ),

with limτ→0+ h(τ) = 0. Therefore, by Proposition 2.2, passing to the limit in the lastinequality we obtain (232) and so (231) follows.

Remark 2.6 Let ϕ be as in Proposition 2.6, then similarly as done in Remark 2.5,we can prove that ϕ satisfies (231) if and only if it satisfies

λϕ(x)+Hi(x,ζ )≤ 0 ∀x ∈K , ∀ζ ∈ ∂V ϕ(x). (233)

We only focus on showing that (231) implies (233). Let x ∈ domϕ and ζ ∈ ∂V ϕ(x),by [33, Proposition 3.4.5] we can find two sequences xn ⊆ domϕ and ζn ⊆RN

such that xn→ x, ϕ(xn)→ ϕ(x), ζn ∈ ∂Pϕ(xn) and ζn→ ζ for which

λϕ(xn)≤ 〈 f (xn,u),ζn〉+ `(xn,u) ∀n ∈ N, ∀u ∈Ui(xn).

Since Ui is in particular lower semicontinuous, if u ∈Ui(x) realizes the maximum inthe definition of the tangential hamiltonian Hi at (x,ζ ), we can find a sequence un ∈Ui(xn) such that un→ u. Therefore, evaluating at u = un in the previous inequalityand letting n→+∞, we get (233).

Characterization of Strongly increasing principle (sufficiency).

In this section we prove the converse of Proposition 2.6 under the controllabilityassumption (H3). The proof consists in analyze three different types of trajectoriesdefined on a finite interval of time [0,T ]. The first case corresponds to trajectoriesthat dwell on a single manifold but whose extremal points may not do so, as forinstance in Figure 12a. This case is treated independently in Lemma 2.4. The secondtype is studied in Step 1 of the proof of Proposition 2.7, these trajectories have thecharacteristic that can be decomposed into a finite number of first type trajectories;see an example in Figure 12b.

The third and more delicate type of trajectories to treat are those one that switchfrom one stratum to another infinitely many times in a finite interval as in Figure12c. The hypothesis (H3) is made to handle these trajectories. It allows to constructan approximate trajectory of type 2, as in Figure 12c, whose the corresponding costis almost the same.

M1

M0

y1(T ) x1y2(T ) x2

(a) Extremal switching times.

M1

M0

y(t1) xy(t3) y(t2)y(t4)y(t5)

y(T )

(b) Finite switching times.

M1

M0

y

y(T )y(T )

(c) Chattering trajectory and its approximation.

Fig. 12: Situation to be considered.

The proof we present is based on the following criterion for strong invarianceadapted to smooth manifolds. This proposition is similar in spirit to Theorem 4.1 in

[21]. The proof of this lemma is omitted for the moment, but can be found at theend of this section.

Lemma 2.3 Suppose M⊆Rk is locally closed, S ⊆Rk is closed with S ∩M 6= /0and Γ : M⇒ Rk is locally Lipschitz and locally bounded.

Let R > 0 and set MR = M∩B(0,R). Assume that there exists κ = κ(R) > 0such that

supv∈Γ (x)

〈x− s,v〉 ≤ κ distxS ∩M2 ∀x ∈MR, ∀s ∈ projS∩M(x). (234)

Then for any absolutely continuous arc γ : [0,T ]→M that satisfies

γ ∈ Γ (γ) a.e. on [0,T ] and γ(t) ∈MR ∀t ∈ (0,T ),

we havedistγ(t)S ∩M≤ eκt distγ(0)S ∩M ∀t ∈ [0,T ].

As we said, the proof of the sufficiency part is divided itself into many steps. Thestep zero is the following Lemma.

Lemma 2.4 Suppose that (H0), (H1) and (H]2) hold in addition of (H f ) and (H`).

Let ϕ : K → R∪ +∞ be a lower semicontinuous function. Assume that (231)holds. Then for any x ∈K , u ∈Ux and any 0 ≤ a < b <+∞, if y(t) := yu

x(t) ∈Mifor every t ∈ (a,b) with i ∈I , we have

ϕ(y(a))≤ e−λ (b−a)ϕ(y(b))+ eλa

∫ b

ae−λ s`(y,u)ds. (235)

Proof. First of all we consider a backward augmented dynamic defined for any(τ,x) ∈ R×Mi as follows:

Gi(τ,x) =−(

f (x,u)e−λτ(`(x,u)+ r)

) ∣∣∣∣ u ∈Ui(x),0≤ r ≤ β (x,u)

.

Thanks to (H0) and the definition of Ui(·), the mapping Gi has convex compactimages and by the statement of the proposition, Gi has nonempty images as well.Additionally, Gi is locally Lipschitz by (H]

2).Since y = yu

x ∈Mi on (a,b), then Ui has nonempty images we set Mi =R×Mi×R2 and define Γi : Mi⇒ RN+3 as

Γi(τ,x,z,w) = −1×Gi(τ,x)×0 , ∀(τ,x,z,w) ∈Mi.

Note that Mi is an embedded manifold of RN+3 and Γi satisfies the same assump-tions than Gi with nonempty images. Consider the closed set Si = epi(ψi) where

ψi(τ,x,z) =

e−λτ ϕi(x)+ z if x ∈M i,

+∞ otherwise ,∀(τ,x,z) ∈ [0,+∞)×M i×R.

Then if (231) holds, the following also holds

supv∈Γi(τ,x,z,w)

〈η ,v〉 ≤ 0 ∀(τ,x,z,w) ∈Si, ∀η ∈N PSi(τ,x,z,w). (236)

Indeed, if Si = /0 it holds by vacuity. Otherwise, take (τ,x,z,w)∈Si and (ξ ,−p)∈N P

Si(τ,x,z,w). Therefore, we have p≥ 0 because Si is the epigraph of a function.

Recall that Γi(τ,x,z,w) 6= /0 because Ui(x) 6= /0. Consider p > 0, then, by the samearguments used in Proposition 2.5, for any v ∈ Γi(τ,x,z,w) we have, for some u ∈Ui(x), r ≥ 0 and ζ ∈ ∂Pϕi(x)

〈(ξ ,−p),v〉= pe−λτ(λϕi(x)−〈ζ , f (x,u)〉− `(x,u)− r)

≤ pe−λτ(λϕi(x)−〈ζ , f (x,u)〉− `(x,u))

≤ pe−λτ(λϕi(x)+Hi(x,ζ )).

Since ϕi(x) = ϕ(x), (231) holds and v ∈ Γi(τ,x,z,w) is arbitrary, we can take supre-mum over v to obtain the desired inequality (236). Similarly as done for Proposition2.5, if p = 0 we use the Rockafellar Horizontal Theorem and the continuity of Hi toobtain (236) for any η .

Let R > R > 0 large enough so that yux([a,b])⊆ B(0, R) and

supX∈M∩B(0,R)

|projMi∩Si(X)|< R.

Let Li be the Lipschitz constant for Γi on Mi∩B(0,R), so (236) implies (234) withκ = Li. In particular, by Proposition 2.3 we have that for any absolutely continuousarc γ : [a,b]→Mi which satisfies (228) (with Γi instead of Γ ) and γ(t)∈Mi for anyt ∈ (a,b),

distγ(t)S ∩Mi ≤ eLit distγ(a)S ∩Mi ∀t ∈ [a,b]. (237)

Finally, consider the absolutely continuous arc defined on [a,b] by

γ(t) =(

a− t,y(a+b− t),−∫ t

aeλ (s−a)`(y(a+b− s),ul(a+b− s))ds,ϕ(b)

).

Since γ ∈ Γi(γ) a.e. on [a,b], γ(t) ∈Mi for any t ∈ (a,b) and γ(a) ∈Si we get thatγ(b) ∈Si which implies (235) after some algebraic steps.

Now we are in position to state a result on the converse of Proposition 2.6 andprovide its proof.

Proposition 2.7 Suppose that (H0), (H1), (H]2) and (H3) hold in addition of (H f )

and (H`). Let ϕ : K →R∪+∞ be a lower semicontinuous function with domU·⊆domϕ . If (231) holds, then ϕ is strongly increasing for the controlled system.

Proof. Let x ∈ domϕ and u ∈Ux. We want to show that inequality (224) holds fory= yu

x . For this purpose we fix T > 0 and we set IT (y) = i∈I : ∃t ∈ [0,T ], y(t)∈

Mi. Note that IT (y) is finite because the stratification is locally finite and so

[0,T ] =⋃

i∈IT (y)

Ji(y), with Ji(y) := t ∈ [0,T ] | y(t) ∈Mi.

We split the proof into two parts:

Step 1. Suppose first that each Ji(y) can be written as the union of a finite numberof intervals, this means that there exists a partition of [0,T ]

π = 0 = t0 ≤ t1 ≤ . . .≤ tn ≤ tn+1 = T

so that if tl < tl+1 for some l ∈ 0, . . . ,n, then there exists il ∈ IT (y) satisfying(tl , tl+1)⊆ Jil (y). Therefore, for any l ∈ 0, . . . ,n such that tl < tl+1 by Lemma 2.4we have

ϕ(y(tl))≤ e−λ (tl+1−tl)ϕ(y(tl+1))+ eλ tl∫ tl+1

tle−λ s`(y,u)ds.

Hence, using inductively the previous estimation and noticing that t0 = 0 and tn+1 =T we get exactly (224), so the result follows.

Step 2. In general, the admissible trajectories may cross a stratum infinitely manytimes in arbitrary small periods of times. In order to deal with this general situation,we will use an inductive argument in the number of strata where the trajectory canpass, let us denote this number by κ . The induction hypothesis (Pκ) is:

Suppose M is the union of κ strata and y(t) ∈M for every t ∈ (a,b), where 0≤ a < b≤ Tthen (235) holds.

By Lemma 2.4, the induction property holds true for the case when κ = 1 becausethe arc remains in only one stratum. So, let us assume that the induction hypothesisholds for some κ ≥ 1. Let us prove it also holds for κ +1.

Suppose that for some 0 ≤ a < b ≤ t, the arc y is contained in the union ofκ + 1 strata on the interval (a,b). By the stratified structure of K , we can alwaysassume that there exists a unique stratum of minimal dimension (which may bedisconnected) where the trajectory passes. We denote it by Mi and by M the unionof the remaining κ strata. Note that, Mi ⊆M and M is relatively open with respectto M . Two cases have to be considered:

Case 1: Suppose that y([a,b]) ⊆M ∪Mi. Without loss of generality we can as-sume that y(a),y(b) ∈Mi. Therefore, J := [a,b]\Ji(y) is open and so, for any ε > 0there exists a partition of [a,b]

b0 := a≤ a1 < b1 ≤ a2 < b2 ≤ . . .≤ an < bn ≤ b =: an+1

such that

meas

(J \

n⋃l=1

(al ,bl)

)≤ ε.

with y(al),y(bl) ∈ Ji and (al ,bl)⊆ J for any l = 1, . . . ,n. In particular, by the induc-tion hypothesis we have

ϕ(y(al))≤ e−λ (bl−al)ϕ(y(bl))+ eλal

∫ bl

al

e−λ s`(y,u)ds. (238)

Notice also thatn⋃

l=0

[bl ,al+1]\ Ji(y) = J \n⋃

l=1

(al ,bl).

Hence, if we set Jl := [bl ,al+1]\ Ji(y) and εl = meas(Jl), we have ∑nl=0 εl ≤ ε .

We now prove that there exists L > 0 so that

ϕ(y(bl))≤ eλεl

(e−λ (al+1−bl)ϕ(y(al+1))+ eλbl

∫ al+1

bl

e−λ s`(y,u)ds)+Lεl . (239)

On the other hand, there exists a countable family of intervals (αp,βp) ⊆[bl ,al+1] (not necessarily pairwise different) such that εl = ∑p∈N(βp−αp), y(t) ∈M for any t ∈ (αp,βp) and y(αp),y(βp) ∈Mi. If the number of intervals turns outto be finite, then (239) follows by the same arguments as in Step 1. So we assumethat (αp,βp)p∈N is an infinite family of pairwise disjoint intervals.

Since ε is arbitrary, we can assume that it is small enough such that εl < εi whereεi is given by (H3). So, for any p ∈N, there exists up : [0,+∞)→U measurable andδp > αp−βp such that

yp(t) ∈Mi, ∀t ∈ [αp,βp +δp], yp(αp) = y(αp), and yp(βp +δp) = y(βp)

where yp is the solution to (162) associated with up. Furthermore, there exists ∆i > 0such that δp < (1−∆i)(βp−αp).

Let Jli := [bl ,al+1]∩ Ji(y) and the measurable function ω : [bl ,al+1]→ R

ω(t) = 1Jli(t)+ ∑

p∈N

βp−αp +δp

βp−αp1(αp,βp)(t)> 0, ∀t ∈ [bl ,al+1].

Define ν(t) = bl +∫ t

blω(s)ds for every t ∈ [bl ,al+1]. Note that it is absolutely con-

tinuous, strictly increasing and bounded from above by cl+1 := ν(al+1) on [bl ,al+1],so it is an homeomorphism from [bl ,al+1] into [bl ,cl+1].

Let u : [bl ,cl+1]→U measurable defined as

u = u(ν−1)1Jli(ν−1)+ ∑

p∈Nup1(αp,βp)(ν

−1), a.e. on [bl ,cl+1],

and let y be trajectory of (162) associated with up such that y(bl) = y(bl). Note thatby construction y(ν(t)) = y(t) for any t ∈ Jl

i and y(t) ∈ Mi for any t ∈ [bl ,cl+1].Hence by Lemma 2.4

ϕ(y(bl))≤ e−λ (cl+1−bl)ϕ(y(al+1))+ eλbl

∫ cl+1

bl

e−λ s`(y(s), u(s))ds. (240)

By the Change of Variable Theorem for absolutely continuous function (see forinstance [95, Theorem 3.54]) we get∫ cl+1

bl

e−λ s`(y(s), u(s))ds =∫ al+1

bl

e−λν(s)`(y(ν(s)), u(ν(s)))ν ′(s)ds.

Furthermore, `(y(ν), u(ν))ν ′ = `(y,u) a.e. on Jli and by (218)

`(y(τ), u(τ))ν ′ ≤ L := max1,∆c`(1+ |x|)λ`eλ`c f (T+∆εl) a.e. on [bl ,al+1].

On the other hand, since `≥ 0 we get∫ cl+1

bl

e−λ s`(y(s), u(s))ds≤∫ al+1

bl

e−λν(s)`(y,u)ds+Lεl , (241)

and we finally get (239) from (240) and (241) since

ν(t)≥ bl +meas(Jli ∩ [bl , t]) = t−meas([bl , t]∩ Jl)≥ t− εl , ∀t ∈ [bl ,al+1].

By (238) and (239) the following also holds

ϕ(y(bl))≤ eλεl

(e−λ (bl+1−bl)ϕ(y(bl+1))+ eλbl

∫ bl+1

bl

e−λ s`(y,u)ds)+Lεl .

Therefore, by using an inductive argument we can prove that

ϕ(y(b0))≤ eλ ∑n−1l=0 εl

(e−λ (bn−b0)ϕ(y(bn))+ eλb0

∫ bn

b0

e−λ s`(y,u)ds)

+L

(n−1

∑l=0

εleλ ∑l−1k=0 εk

),

and using (240) on the interval [bn,an+1] we get

ϕ(y(b0))≤ eλ ∑nl=0 εl

(e−λ (an+1−b0)ϕ(y(an+1))+ eλb0

∫ an+1

b0

e−λ s`(y,u)ds)

+L

(n

∑l=0

εleλ ∑l−1k=0 εk

).

Finally, by the definition of b0 and an+1 we finally obtain:

ϕ(y(a))≤ eλε

(e−λ (b−a)

ϕ(y(b))+ eλa∫ b

ae−λ s`(y,u)ds

)+Leλε

ε.

Thus, letting ε → 0 we obtain the induction hypothesis for κ +1.

Case 2: We consider the case y(a) /∈M ∪Mi or y(b) /∈M ∪Mi.Suppose first that y(a) /∈M i\Mi and y(b) /∈M i\Mi, then there exists δ > 0 such

that y(t) ∈M ∪Mi for every t ∈ [a+ δ ,b− δ ] and disty(t)M i \Mi > 0 for everyt ∈ [a,a+ δ ]∪ [b− δ ,b]. So, we can partitionate [0,T ] into three parts [a,a+ δ ],[a+ δ ,b− δ ] and [b− δ ,b]. In view of Case 2 and the inductive hypothesis, (235)holds in each of the previous intervals. So, gathering the three inequalities we getthe induction hypothesis for κ +1.

Secondly, suppose that only y(a) /∈M ∪Mi, then there exists a sequence an ⊆(a,b) such that an→ a and y([an,b])⊆M \Mi. So, by Case 1,

ϕ(y(an))≤ e−λ (b−an)ϕ(y(b))+ eλan

∫ b

an

e−λ s`(yux ,u)ds.

Furthermore, since ϕ is lower semicontinuous and y(·) is continuous we can pass tothe limit to get (235), so the result also holds in this situation.

Finally, it only remains the situation y(b) ∈M i \Mi. Similarly as above, thereexists a sequence bn ⊆ (a,b) such that bn→ b and y([a,bn])⊆M \Mi such that

ϕ(y(a))≤ e−λ (bn−a)ϕ(y(bn))+ eλa

∫ bn

ae−λ s`(yu

x ,u)ds.

By (H3), for n ∈ N large enough, there exists a control un : (bn,b+ δn)→U and atrajectory yn : [bn,b+δn]→M i with yn(bn) = y(bn), yn(b+δn) = y(b) and yn(t) ∈Mi for any t ∈ [bn,b+δn). By Lemma 2.4

ϕ(y(bn))≤ e−λ (b−bn)ϕ(y(b))+ εn,

with εn → 0 as n→ +∞, then gathering both inequalities and letting n→ +∞ weget the induction hypothesis and so the proof is complete.

2.2.5 Proof of technical lemmas

In this final section we provide the proof of Lemma 2.2 and 2.3 that were statedwithout being proved.

Proof (Proof of Lemma 2.2). Let R > 0 and set MRi = Mi∩B(x,R). Consider the set

valued map Γi : MRi × (−1,1)→ RN ⇒ R given by

Γi(y, t) =(

f (y,u)e−λ t`(y,u)+ r

) ∣∣∣∣ u ∈Ui(y),0≤ r ≤ β (y,u)

, ∀(y, t) ∈MR

i × (−1,1).

Note that by the definition of Ui and thanks to (H f ) and (H`), Γi has closed imagesand since Ui has nonempty images, Γi has nonempty images as well. The definitionof Ui and (H0) imply that it also has convex images.

Besides, by (H]2), Γi is Lipschitz on MR

i × (−1,1), so it admits a Lipschitz selec-tion, gi : MR

i × (−1,1)→ RN ×R such that gi(x,0) = ( f (x,ux), `(x,ux)); see [15,

Theorem 9.4.3] and the subsequent remark. Notice also that

g(y, t) ∈ f (y,Ui(y))×R⊆ TMi(y)×R, ∀(y, t) ∈MRi × (−1,1).

Hence, by the Nagumo theorem (see for instance [14, Theorem 4.2.2]) and the Lip-schitz continuity of gi, there exists ε > 0 such that the differential equation

(y, z) = gi(t,y), y(0) = x, z(0) = 0

admits a unique solution which is continuously differentiable on (−ε,ε) such thaty(t) ∈Mi for every t ∈ (−ε,ε), y(0) = f (x,ux) and z(0) = `(x,ux).

On the other hand, since Γi(y, t) ⊆ G(t,y) for any (t,y) ∈ (−1,1)×MRi , by the

Measurable Selection Theorem (see [14, Theorem 1.14.1]), there exist a measurablecontrol u : (−ε,ε)→U and a measurable function r : (−ε,ε)→ [0,+∞) such that

(y, z) = ( f (y,u),e−λ t`(y,u)+ r), a.e. on (−ε,ε).

Finally, since y(t)∈Mi, we have that u∈Ui(y) a.e. on (−ε,ε), and so the conclusionfollows, because

z(t) =∫ t

0

(e−λ s`(y(s),u(s))+ r(s)

)ds, ∀t ∈ (−ε,ε).

Proof (Proof of Lemma 2.3). Let R > 0 so that γ([0,T ])⊆B(0, R). We denote by cΓ

and LΓ the corresponding bound for the velocities of Γ and the Lipschitz constantof Γ on M∩B(0, R). We take C1 > 0 such that

maxt∈[0,T ]

distγ(t)S ∩M≤C1

Let ε > 0 and set t0 = 0, we construct inductively a partition of [0,T ] in thefollowing way: Given ti ∈ [0,T ) take ti+1 ∈ (ti,T ] satisfying

ti+1 ≤ ti + ε and |γ((1− s)ti + sti+1)− γ(ti)| ≤1

LΓ

ε, ∀s ∈ [0,1].

Note that |γ((1− s)ti + st)− γ(ti)| ≤ cΓ (t − ti) for any s ∈ [0,1], so the choice ofsuch ti+1 is possible. Moreover, we can do this in such a way it produces a finitepartition of [0,T ] which we denote πε = 0 = t0 < t1 < .. . < tn < tn+1 = T. Notethat ‖πε‖ = maxi=0,...,n(ti+1− ti) ≤ ε . For any i ∈ 0, . . . ,n+ 1, we set γi = γ(ti)and choose si ∈ projS∩M(γi) arbitrary. Suppose first that γ(0) ∈M. We will showthe inequality only for t = T . For t ∈ (0,T ) the proof is similar.

Let s 7→ ω(s) := γ((1− s)ti + sti+1) defined on [0,1]. Hence, ω is an absolutelycontinuous function with ω(s) = γ((1− s)ti + sti+1)(ti1 − ti) a.e. s ∈ [0,1]. Thus

ω(1)−ω(0) = γi+1− γi = (ti+1− ti)∫ 1

0γ((1− s)ti + sti+1)ds

On the other hand, since Γ is locally Lipschitz

Γ (γ((1− s)ti + sti+1))⊆ Γ (γi)+LΓ |γ((1− s)ti + sti+1)− γ(ti)|B, ∀s ∈ [0,1].

By construction LΓ |γ((1− s)ti + sti+1)− γ(ti)| ≤ ε . Therefore, there exist two mea-surable functions vi : [0,1]→ Γ (γi) and bi : [0,1]→ B such that

γ((1− s)ti + sti+1) = vi(s)+ εbi(s), a.e. s ∈ [0,1].

Hence

distγi+1S ∩M2 ≤ |γi+1− si|2

= |γi− si|2 +2(ti+1− ti)∫ 1

0〈γi− si,vi(s)+ εbi(s)〉ds+ |γi+1− γi|2

≤ (1+2(ti+1− ti)κ)distγiS ∩M2+ ε(ti+1− ti)[2C1 + c2

Γ ],

where this last comes from (234), the definition of bi and the choice of ti.Let us denote σi = distγiS ∩M and δi = ti+1− ti. Then, using an inductive ar-

gument it is not difficult to show that

σ2n+1 ≤

n

∏i=0

(1+2δiκ)σ20 + ε[2C1 + c2

Γ ]n

∑j=0

n

∏i= j+1

(1+2δiκ)δ j.

≤(

n

∏i=0

(1+2δiκ)

)(σ

20 + ε[2C1 + c2

Γ ]n

∑j=0

δ j

).

Note thatn

∑j=0

δ j = T andn

∏i=0

(1+2δiκ)≤ e2κT ,

so we obtain

σ2n+1 ≤ e2κT (σ2

0 + ε[2C1 + c2Γ ]T ).

Since σn+1 = distγ(T )S ∩M and σ0 = distγ(0)S ∩M, letting ε → 0 we obtainthe desired result.

Suppose now that γ(0) /∈M. Then it is clear that for any δ > 0 small enough thetrajectory γ = γ|[δ ,T ] satisfies the previous assumptions, so the inequality is valid onthe interval [δ ,T ] for any δ > 0. Finally, since the distance function is continuous,we can extend the inequality up to t = 0 by taking limits.

2.3 Characterization of the epigraph of the value function

The second approach we present, leads to the computation of the value functionassociated with state constrained optimal control problems under very general as-sumptions on the dynamics and the set of state constraints K . For almost the wholesection we will consider a finite time horizon. The infinite horizon case will be dis-cussed at the end of the section.

We recall that the definition of the value vunction for a Bolza problem is givenby:

v(t,x) = infu∈Ut (x)

∫ T

t`(s,yu

t,x(s),u(s))ds+ψ(yut,x(T ))

(242)

where

Ut(x) :=

u : [t,T )→U measurable | yut,x(s) ∈K , ∀s ∈ [t,T )

.

As extensively discussed in Section 2.1, when state constraints are taken into ac-count some further compatibility assumptions between the dynamics and the setof state constraints are necessary in order to characterize v as the unique viscositysolution of the state-constrained HJB equation

−∂tv(t,x)+H(t,x,∇xv(t,x)) = 0, (t,x) ∈ (0,T )×K . (243)

Unfortunately, this kind of conditions may be not satisfied even for very simple andclassical problems, as Example 2.1 well shows.

Aside from the evident theoretical interest, the characterization of the value func-tion as the unique solution of equation (243) has the main advantage of allowing itsnumerical computation using numerical methods available for partial differentialequations of this form. Aim of this section is to present a new way for arriving tocompute v, by using PDE methods, without imposing any further assumption nei-ther on the dynamics nor on the set of state constraints. It is particularly importantto stress the intrinsic difference between what we are going to present and the ap-proaches previously discussed in Sections 2.1 and 2.2: here, we are not looking fora characterization of v as a solution of a suitable HJB equation under more or lessrestrictive assumptions, we are are instead developing a strategy for its numericalcomputation.

The method we present here relies one two main passages: first, interpret theoptimal control problem (242) as a reachability problem; second, solve the reacha-bility problem by a generalized level set approach. More precisely, at a first stage thestate-constrained optimal control problem (242) is translated into a state-constrainedtarget problem. The main advantage of this first passage is that a level set methodcan be used for solving the target problem and this turns out to be particularly usefulin presence of state constraints since they can be managed by an exact penalizationtechnique without requiring any further assumption on the system. The link betweenoptimal control problems and reachability has been originally investigated in [16]

and then exploited in several papers for characterizing the value function, more pre-cisely its epigraph, by viability tools (see [13, 42, 43] and the reference therein).However the work we present in this section aim to a computation of v based on aPDE approach and this leads us to the second step.

The second step consists in a generalization to the constrained case of the level-set method. This method was originally introduced by Osher and Sethian [111] in1988 for studying fronts propagation problems (see also Section 4.3). The main ideacontained in this work is that it is possible to describe a curve, the curve representingthe front in that case, as a level set of a suitable continuous function. Thanks to thisobservation, in [111] the propagation of the front is described by the evolution ofthis function, i.e. by an evolutionary PDE. In the later years the same idea has beenapplied with success by many authors in different fields. Among them we mention[63] for rendez-vous problems and [104, 105] for minimum time problems. Weare in particular interested in state-constrained reachability problems and in thisframework the level set method has been applied in [29], [89] and [10].

The presentation we give here strongly relies on [10]. For the extension of thisapproach to the stochastic framework, the reader can refer to [30]. Unless otherwisespecified, we will still work under the general assumptions (HU ), (H f ) and under theslightly less general assumption on the cost `:

(i) `(·, ·, ·) is continuous on [0,T ]×Rn×U ;(ii) ∃L` ≥ 0 such that ∀x,y ∈ Rn, t,s ∈ [0,T ],u ∈U :

|`(t,x,u)− `(s,y,u)| ≤ L`(|x− y|+ |t− s|).(H ′`)

We will also assume the Lipschitz continuity of Ψ .

2.3.1 The associated reachability problem

Given a real-valued function ϕ we denote by epiϕ its epigraph, that is

epiϕ :=(x,z) ∈ Rn+1 : ϕ(x)≤ z

.

Let us start with the following easy result:

Proposition 2.8 For any (t,x) ∈ [0,T ]×Rn, one has

v(t,x) = inf

z ∈ R : ∃u ∈ Ut(x) such that(yu

t,x(T ),zut,x,z(T )

)∈ epiψ

where zu

t,x,z(·) is the one-dimensional variable defined by

zut,x,z(·) := z−

∫ ·t`(s,yu

t,x(s),u(s))ds.

Proof. The proof follows immediately by the definition of v and zut,x,z(·).

In fact once defined the set

Z(t,x) :=

z ∈ R : ∃u ∈ Ut(x) such that(yu

t,x(T ),zut,x,z(T )

)∈ epiψ

,

for any z ∈Z(t,x) there exists u ∈ Ut(x) such that

z≥ ψ(yut,x(T ))+

∫ T

t`(s,yu

t,x(s),u(s))ds≥ v(t,x).

Hence, by the arbitrariness of z one obtains

infZ(t,x) ≥ v(t,x).

On the other hand, for any u ∈ Ut(x)

ψ(yut,x(T ))+

∫ T

t`(s,yu

t,x(s),u(s))ds ∈Z(t,x)

then

ψ(yut,x(T ))+

∫ T

t`(s,yu

t,x(s),u(s))ds≥ infZ(t,x).

Therefore the other inequality follows just taking the infimum over u ∈ Ut(x).

Proposition 2.8 expresses the link between the optimal control problem and a state-constrained reachability problem. In order to clarify this link, we can consider thedynamics yu

t,x, in the augmented state space Rn+1, given by ˙y(s) =(

f−`

)(s, y(s),u(s)) =: f (s, y(s),u(s)) s ∈ [t,T ]

y(t) = x≡ (x,z).. (244)

Let us define the target set Θ ⊂ Rn+1

Θ := epiψ.

The problem of the state-constrained reachability of the set Θ consists in charac-terizing for any t ∈ [0,T ] the set of all the initial position from which it is possibleto find an admissible trajectory that reaches the target at time T satisfying the stateconstraints on [t,T ], i.e.

RT ,K

(t) :=

x ∈ Rn+1 : ∃u ∈ Ut(x) such that yut,x(T ) ∈Θ

. (245)

It is immediate to observe that Proposition 2.8 gives

v(t,x) = inf

z ∈ R : (x,z) ∈RT ,K

(t). (246)

Equality (246) shows that the values of v can be computed starting by the character-ization of the set R

T ,K(t) for any t ∈ [0,T ]. This result establishes the desired link

between the optimal control problem (242) and a state-constrained target problem(245) and it completes the first step of our approach.

2.3.2 The level-set approach

Let us introduce the following convexity assumption:

∀(t,x) ∈ [0,T ]×Rn the set( f ,−`)(t,x,u),u ∈U

is convex. (H1)

Under assumption (H1) the set of trajectories yut,x(·) solutions of (244) is a compact

set (see [14] for instance) and v can be proved to be a LSC function. As a conse-quence epiv(t, ·) is a closed set and

epiv(t, ·) = RT ,K

(t)

for any t ∈ [0,T ]. Hence, in this case, equality (246) reads as the well-known relation

v(t,x) = inf

z ∈ R : (x,z) ∈ epiv(t, ·).

As announced, we are going to solve the reachability problem, i.e. to compute the setRΘ ,K , by a level set approach. The main idea is to define an auxiliary unconstrainedoptimal control problem (different from (242) and setted in Rn+1) such that thebackward reachable set R

T ,K(t) corresponds to a level set of the associated value

function.Let us start introducing a level set function g : Rn→ R such that:

g is Lipschitz continuous;g(x)≤ 0⇔ x ∈K .

(H2)

The following optimal control problem is then considered:

w(t,x,z) = infu∈U (t,T )

(ψ(yu

t,x(T ))− zut,x,z(T )

)∨ max

s∈[t,T ]g(yu

t,x(s)). (247)

where the notation a∨b = max(a,b) is used.This is what, throughout the section, we will call the auxiliary optimal controlproblem. It is important to stress that (247) is an unconstrained problem. In-deed, the state constraint involved in (242) only appears in the penalization term“maxs∈[t,T ] g(yu

t,x(s))” and the minimization takes into account all the measurablefunctions in U .Here we shall use problem (247) to characterize the epigraph of v, or equivalentlythe set R

T ,K(t). We point out that the use of an exact penalization in a maximum

form for the study of the reachability analysis for state-constrained nonlinear sys-tems goes back to [29, 90] and [106].

Theorem 2.2 Let assumptions (H1)-(H2) be satisfied. Then for any t ∈ [0,T ] onehas

RT ,K

(t) = epiv(t, ·) =

x ∈ Rn : w(t,x,z)≤ 0. (248)

Moreover

v(t,x) = inf

z ∈ R : w(t,x,z)≤ 0. (249)

Proof. The first equality in (248) has already been discussed and it follows by as-sumption (H1). The latter statement is a consequence of the first one and equality(246).Let us assume that (x,z) ∈ epiv(t, ·). Then there exists a minimizing sequence oftrajectories yun

t,x such that

limn→+∞

ψ(yunt,x(T ))+

∫ T

t`(s,yun

t,x(s),un(s))ds− z = v(t,x)− z≤ 0

and yunt,x(s) ∈K for all s ∈ [t,T ]. In particular, thanks to assumption (H2), the fact

that the sequence of trajectories is admissible (that is satisfies the state constraint)implies maxs∈[t,T ] g(y

unt,x(s)) ≤ 0 for any n ∈ N. Hence, recalling the definition of

zt,x,z(·), one has

w(t,x,z)≤ liminfn→+∞

(ψ(yun

t,x(T ))− zunt,x,z(T )

)∨ max

s∈[t,T ]g(yun

t,x(s))≤ 0.

Conversely, let us assume w(t,x,z) ≤ 0. Under assumption (H1), for any (t,x,z) ∈[0,T ]×Rn×R the infimum in (247) is attained for some U-valued measurable func-tion. Let u∗ be such a minimizer and (yu∗

t,x(·),zu∗t,x,z(·)) the associate optimal trajectory.

One has

0≥ w(t,x,z) =(ψ(yu∗

t,x(T ))− zu∗t,x,z(T )

)∨ max

s∈[t,T ]g(yu∗

t,x(s)).

Therefore, on one hand, maxs∈[t,T ] g(yu∗t,x(s)) ≤ 0 and yu∗

t,x(·) satisfies the state con-straints, one the other hand, recalling the definition of zu∗

t,x,z(·), one has

v(t,x)− z≤ ψ(yu∗t,x(T ))+

∫ T

t`(s,yu∗

t,x(s),u∗(s))ds− z≤ 0,

which gives the desired result.

Theorem 2.2 contains the main result of the section. It states that, in order to solvethe state-constrained optimal control problem (242), from now on we can deal onlywith the auxiliary value function w and then reconstruct the values of v thanks to(249).

Remark 2.7 We point out that the value function w depends on the choice of g.However the set

(x,z) ∈ Rn+1 : w(t,x,z)≤ 0

is independent of this choice and for any g, g satisfying (H2) one has

(x,z) ∈ Rn+1 : w(t,x,z)≤ 0=

(x,z) ∈ Rn+1 : w(t,x,z)≤ 0

,

for any t ∈ [0,T ], where w and w denote the value functions defined by (247) asso-ciated respectively with the choice g and g.

2.3.3 The HJB equation for the auxiliary optimal control problem

The important feature of the auxiliary problem (247) is that it is not under state con-straints. As a consequence, the function w enjoys more regularity properties and canbe characterized by a HJB equation without assuming any compatibility assump-tion between dynamics and state constraints. We mainly refer to [22] and [23] forthe study of optimal control problems with the cost in a maximum form.

Remark 2.8 We point out that if g ≥ 0, equality (249) still holds if w is defined bythe following Bolza problem

infu∈U (t,T )

(ψ(yu

t,x(T ))− zut,x,z(T )

)++∫ T

tg(yu

t,x(s))ds

where ( · )+ denotes the positive part, i.e. (a)+ = max(a,0). Of course the theo-retical analysis of this kind of problem is more standard. However it turns out that,from the numerical point of view, the formulation in (247) involving the maximumcost brings to better results (see the examples proposed in [25]).

The following lemma establishes the Dynamic Programming Principle satisfiedby w. It is well known that this is the main result we need in order to derive the HJBequation that characterizes w.

Lemma 2.5 (DPP) Let assumption (H2) be satisfied. For any t ∈ [0,T ) and τ ∈(t,T ),

w(t,x,z) = infu∈U (t,τ)

w(τ,yu

t,x(τ),zut,x,z(τ))∨ max

s∈(t,τ)g(yu

t,x(s)). (250)

Proof. Let us denote by J : [0,T ]×Rn×R×U →R the cost functional associatedwith (247), i.e.

J(t,x,z,u) :=(ψ(yu

t,x(T ))− zut,x,y(T )

)∨ max

s∈[t,T ]g(yu

t,x(s)).

and

w(t,x,z) = infu∈U (t,T )

J(t,x,z,u).

For any control u ∈ U (t,T ) we denote by u1 and u2 its restriction to the interval[t,τ) and [τ,T ), respectively. One has

J(t,x,z,u)

= (ψ(yut,x(T ))− zu

t,x,z(T ))∨ maxs∈(τ,T )

g(yut,x(s))∨ max

s∈(t,τ)g(yu1

t,x(s))

= J(τ,yu1t,x(τ),z

u1t,x,z(τ),u2)∨ max

s∈(t,τ)g(yu1

t,x(s))

≥ w(τ,yu1t,x(τ),z

u1t,x,z(τ))∨ max

s∈(t,τ)g(yu1

t,x(s))

≥ infu∈U (t,τ)

w(τ,yu

t,x(τ),zut,x,z(τ))∨ max

s∈(t,τ)g(yu

t,x(s)),

where we also used the fact that, for any u ∈U and s ∈ [τ,T ], one has

yut,x(s) = yu

τ,yu1t,x(τ)

(s) and zut,x,z(s) =

z uτ,y

u1t,x(τ),z

u1t,x,z(τ)

(s).

Then, thanks to the arbitrariness of u we can conclude

w(t,x,z)≥ infu∈U (t,τ)

w(τ,yu

t,x(τ),zut,x,z(τ))∨ max

s∈(t,τ)g(yu

t,x(s)).

For the other inequality, let u1 ∈U (t,τ),u2 ∈U (τ,T ) and

u(s) = u1(s)1s∈[t,τ)+u2(s)1s∈[τ,T ] ∈U .

One has

w(t,x,z)

≤ (ψ(yut,x(T ))− zu

t,x,z(T ))∨ maxs∈(τ,T )

g(yut,x(s))∨ max

s∈(t,τ)g(yu1

t,x(s))

= J(τ,yu1t,x(τ),z

u1t,x,z(τ),u2)∨ max

s∈(t,τ)g(yu1

t,x(s)).

By the arbitrariness of u2 we get

w(t,x,z)≤ w(τ,yu1t,x(τ),z

u1t,x,z(τ))∨ max

s∈(t,τ)g(yu1

t,x(s))

and taking the infimum over u1 ∈U (t,τ) we obtain the desired result.

The following result concerns the regularity properties of w.

Proposition 2.9 Let assumption (H2) be satisfied. The value function w is locallyLipschitz continuous on [0,T ]×Rn×R.

Proof. By using the definition of w and the simple inequalities

|a∨b|− |c∨d| ≤ |a− c|∨ |b−d| and infn

an− infn

bn ≤ supn

(an−bn),

one has for any x≡ (x,z), x′ ≡ (x′,z′) ∈ Rn+1, t ∈ [0,T ]

|w(t, x)−w(t, x′)| ≤ supu∈U (t,T )

∣∣ψ(yut,x(T ))−ψ(yu

t,x′(T ))+ zut,x′,z′(T )− zu

t,x,z(T )∣∣

∨ maxs∈(t,T )

∣∣g(yut,x(s))−g(yu

t,x′(s))∣∣

≤ supu∈U (t,T )

L max

s∈(t,T )

∣∣yut,x(s)− yu

t,x′(s)∣∣

where L≥ 0 is a constant depending on Lψ and Lg. Thanks to assumptions (H f ) and(H ′`) on the dynamics f , for s ∈ (t,T ),

|yut,x(s)− yu

t,x′(s)| ≤ eL(s−t)|x− x′| ≤ eLT |x− x′|

where L, depending only on L f and L`, is the Lipschitz constant of f . Hence, forany t ∈ [0,T ]

|w(t, x)−w(t, x′)| ≤ LeLT |x− x′|. (251)

In order to prove the Lipschitz continuity with respect to time, we will use the DPP.Let t, t ′ ∈ (0,T ) and t ≤ t ′. Observing that w(t, x) = w(t, x)∨ g(x) and using (250)and (251), one has

|w(t ′, x)−w(t, x)|

=

∣∣∣∣ infu∈U (t,t ′)

w(t, yu

t,x(t′))∨ max

s∈(t,s)g(yu

t,x(s))−w(t, x)∨g(x)

∣∣∣∣≤ sup

u∈U (t,t ′)

|w(t, yu

t,x(t′))−w(t, x)|∨ max

s∈(t,t ′)|g(yu

t,x(s))−g(x)|

≤ supu∈U (t,t ′)

LeLT |yu

t,x(t′)− x|∨Lg max

s∈(t,t ′)|yu

t,x(s)− x|.

By Gronwall’s estimates, for every s∈ (t, t ′), denoted Cϕ := max(s,u)∈[0,T ]×U |ϕ(s,0,u)|for ϕ = f , f , one has

maxs∈(t,t ′)

|yut,x(s)− x| ≤ (C f +L f |x|)eL f T |t− t ′|

and|yu

t,x(t′)− x| ≤ (C f +L f |x|)e

L f T |t− t ′|.Therefore one can conclude that there exists C > 0 such that

|w(t ′, x)−w(t, x)| ≤C(1+ |x|)|t ′− t|.

Combining the inequalities above, one obtains

|w(t, x)−w(t ′, x′)| ≤C(1+ |x|)(|t− t ′|+ |x− x′|).

Remark 2.9 Notice that when K is a bounded set (let us say K ⊂ B(0,R),∃R ≥0), then it is possible to modify f , ` and g outside K in order to get a boundedauxiliary value function. Indeed it is sufficient to define the function ρ(x) :=dist(x,B(0,R)) and consider fR(t,x,u)= f (t,x,u)(1−ρ(x))+, `R(t,x,u)= `(t,x,u)(1−ρ(x))+ and gR(x) =max(g,R). In this way the function wR defined using ( fR, `R,gR)is still Lipschitz continuous and satisfies wR(t,x,z)≤ 0⇔ w(t,x,z)≤ 0.

Let us denote by H the Hamiltonian defined on [0,T ]×Rn×R×Rn×R by:

H(t,x,z, p,q) := sup−〈 f (t,x,u), p〉+ `(t,x,u)q | u ∈U .

Thanks to Lemma 2.5, we are ready to characterize w as the unique viscosity solu-tion of a HJB equation:

Theorem 2.3 Let assumption (H2) be satisfied. The value function w is the uniquecontinuous viscosity solution of the following HJB equationmin

(−∂tw+H(t,x,z,∇xw,∂zw) , w−g(x)

)= 0 in [0,T )×Rn×R

w(T,x,z) = (ψ(x)− z)∨g(x) in Rn×R.(252)

Proof. The terminal condition is satisfied thanks to the definition of w. Let us provethe super-solution property. By DPP, for any h > 0, one has

w(t,x,z)≥ infu∈U(t,t+h)

w(t +h,yu

t,x(t +h),zut,x,z(t +h))

,

then, by classical arguments in viscosity theory (see [18] for instance), one obtains

−∂tw+H(t,x,z,∇xw,∂zw)≥ 0

in the viscosity sense. Moreover, by the very definition of w, it follows that

w(t,x,z)≥ infu∈U (t,T )

max

s∈[t,T ]g(yu

t,x(s))≥ g(x).

Therefore we can conclude that

min(−∂tw+H(t,x,z,∇xw,∂zw) , w−g(x)

)≥ 0

in the viscosity sense, i.e. w is a viscosity super-solution.

It remains to prove that w is a sub-solution. Let (x,z) ∈ Rn×R and t ∈ [0,T ). Ifw(t,x,z)≤ g(x) we have done. Let us suppose that w(t,x,z)> g(x). For any h > 0,and s ∈ [t, t + h], the trajectory yu

t,x(s) is in a neighborhood of x. Therefore, thanksto the continuity of g and w, for any u ∈ U (t,T ) there exists h := h(u) > 0 smallenough such that w(s,yu

t,x(s),zut,x,z(s))> g(yu

t,x(s)) for any s ∈ [t, t+h]. Applying theDPP with this choice of h, one has

w(t,x,z) = infu∈U (t,T )

w(t +h,yu

t,x(t +h),zut,x,z(t +h))

and, again by classical arguments such those in [18], one obtains

−∂tw+H(t,x,z,∇xw,∂zw)≤ 0

in the viscosity sense, so that the sub-solution property

min(−∂tw+H(t,x,z,∇xw,∂zw) , w−g(x)

)≤ 0

is finally satisfied. Uniqueness follows by a comparison principle between sub- andsuper-solutions to equation (252). We refer to [19] or the appendix in [10] for arigorous proof.

Remark 2.10 Equation (252) is referred in literature as an obstacle problem, wherethe obstacle is represented by the function g. This kind of equations also arise fromoptimal stopping time problems (see [18] for instance).

2.3.4 Extensions: Infinite horizon problems and two players games

In this section, two extensions of the presented approach are considered. On oneside, we deal with infinite horizon optimal control problems, on the other side, weshow how the technique presented applies to the two players framework.

Infinite horizon problems

For the first extension we will consider the functions f and ` independent of time,satisfying (H f ), (H ′`) and (H1). The infinite horizon optimal control problem we aimto solve is

v(x) := infu∈U(x)

∫∞

0e−λ t`(yu

x(t),u(t))dt

(253)

where

U(x) := u : [0,+∞)→U measurable | yux(s) ∈K , ∀s≥ 0 .

Let g : Rn → R be a function satisfying (H2). Following the ideas presented untilnow we define the following unconstrained auxiliary optimal control problem

w(x,z) := infu∈U (0,+∞)

(∫ +∞

0e−λ t`(yu

x(t))dt− z)∨ max

t∈(0,+∞)e−λ tg(yu

x(t)). (254)

In this case the Hamiltonian H : Rn×R×Rn×R→ R is defined by

H(x, p,q) := supu∈U−〈 f (x,u), p〉+ `(x,u)q .

The following theorem can be proved, extending the main results of the previoussection to the infinite horizon case:

Theorem 2.4 Let assumptions (H1)-(H2) be satisfied.Then

• v(x) = infz ∈ R : w(x,z)≤ 0;• w is Lipschitz continuous on Rn×R;• w is the unique continuous viscosity solution to

min(

λw+H(x,∇xw,∂zw) , w−g(x))= 0 in Rn×R.

Two players games

The second extension we deal with is the case of two players games (see [18, Chap-ter VIII] and the references therein). In this framework the final horizon T is finite,but in addition to the player taking control values in U , we consider another nonempty and compact subset B of Rm. The set of controls for the second player is

B(t,T ) := β : (t,T )→ B measurable .

We also introduce the set of non-anticipative strategies:

Γt :=

γ : B(t,T )→U (t,T ) such that ∀s ∈ [t,T ],β ,β ′ ∈B(t,T )(β (θ) = β

′(θ) a.e. θ ∈ [t,s])⇒(

γ[β ](θ) = γ[β ′](θ) a.e. θ ∈ [t,s])

.

We consider a dynamics f : [0,T ]×Rn ×U ×B→ Rn and a distributed cost ` :[0,T ]×Rn×U×B→ R that depend also on the second control and satisfy

(i) f (·, ·, ·, ·) is continuous on [0,T ]×Rn×U×B;(ii) ∃L f ≥ 0 such that ∀x,y ∈ Rn, t,s ∈ [0,T ],u ∈U,β ∈ B :

| f (t,x,u,β )− f (s,y,u,β )| ≤ L f (|x− y|+ |t− s|).(H ′f )

(i) `(·, ·, ·, ·) is continuous on [0,T ]×Rn×U×B;(ii) ∃L` ≥ 0 such that ∀x,y ∈ Rn, t,s ∈ [0,T ],u ∈U,β ∈ B :

|`(t,x,u,β )− `(s,y,u,β )| ≤ L`(|x− y|+ |t− s|).(H ′′` )

As usual we denote by f : [0,T ]×Rn+1×U ×B→ Rn+1 the augmented dynamicsand by yu,β

t,x (·)≡ (yu,βt,x (·),zu,β

t,x,z(·)) the associated trajectory in Rn+1 starting at time tfrom x≡ (x,z). The following convexity assumption on f is considered:

for any (t, x) ∈ [0,T ]×Rn+1,β ∈ B (H ′1)f (t,x,u,β ),u ∈U

is a convex set .

The value function for the first player is given by:

v(t,x) := infγ∈Γt

supβ∈B(t,T )

∫ T

t`(s,yγ[β ],β

t,x (s),γ[β ](s),β (s))ds

+ψ(yγ[β ],βt,x (T ))

∣∣∣∣ yγ[β ],βt,x (s) ∈K ,∀s ∈ [t,T ]

.

We introduce the following auxiliary unconstrained optimal control problem:

w(t,x,z) = infγ∈Γt

supβ∈B

(ψ(yγ[β ],β

t,x (T ))− zγ[β ],βt,x,z (T )

)∨ max

s∈[t,T ]g(yγ[β ],β

t,x (s)). (255)

The following result gives a characterization of the value function v for two playersgames with state constraints, by using a continuous viscosity approach and withoutany controllability assumption:

Theorem 2.5 Let (H ′f ),(H ′′` ),(H ′1) and (H2) hold. Then:

• v(t,x) = infz ∈ R : w(x,z)≤ 0;• w is locally Lipschitz continuous on [0,T ]×Rn×R;• w is the unique continuous viscosity solution of the following Hamilton-Jacobi-

Isaacs (HJI) equationmin(−∂tw+H(t,x,z,∇xw,∂zw) , w−g(x)

)= 0 in [0,T )×Rn×R

w(T,x,z) = (ψ(x)− z)∨g(x) in Rn×R

where H(t,x,z, p,q) := maxu∈U

minβ∈B−〈 f (t,x,u,β ), p〉+ `(t,x,u,β )q.

2.3.5 A numerical example

The numerical example we propose, aims to show how the method we describedcan be used for solving the classical Zermelo’s navigation problem in presence of

state constraints. Let y(t) = (y1(t),y2(t)) represent the position of a boat at time t.The boat moves in a canal R× [−2,2] according with the following dynamics:

y1 = υ cos(u)+a1−a2y22,

y2 = υ sin(u).(256)

Here, u and v are two controls: u, taking values in [0,2π], represents the angle theboat formes with the x-axis and υ , with values in [0,υmax] is the speed of the boat.The quantity a1− a2y2

2 is the current drift. We aim to steer the boat toward a tagetΘ := B((1,0),r0) (a buoy for instance), with the minimum fuel consumption. Onecan assume the consumption being proportional to the quantity∫ T

tυ(s)ds.

We can also consider the presence of some rocks on the way, represented by tworectangular obstables

Ri := x ∈ R2 : |x1− ci,1| ≤ ri,1, |x2− ci,2| ≤ ri,2 i = 1,2,

so thatK = (R× [−2,2])\ (R1∪R2).

The optimal control problem is

v(t,x) = inf(u,υ)∈U (t,T )

∫ T

tυ(s)ds | yu,υ

t,x (T ) ∈Θ and yu,υt,x (s) ∈K , ∀s ∈ [t,T ]

where U (t,T ) is the set of measurable functions with values in [0,2π]× [0,υmax].The level set function gK : R2→ R can be defined by

gK (x) = maxi, j=1,2

(ri, j−|x j− ci, j|

).

We also define the level set function gΘ : R2 → R associated with the terminalconstraints:

gΘ (x) = |x− (1,0)|− r0.

Then, defined zu,υt,x,z(·) := z− ∫ ·t υ(s)ds, the following unconstrained auxiliary opti-

mal control problem is considered

w(t,x,z) = inf(u,υ)∈U (t,T )

(−zu,υ

t,x,z(T ))∨gΘ (yu,υ

t,x (T ))∨ maxs∈[t,T ]

gK (yu,υt,x (s))

,

so that

v(t,x) = inf

z ∈ R : w(t,x,z)≤ 0

and the value function w can be numerically approximated by solving the followingHJB equationmin

(−∂tw+max(0, |∇xw|+∂zw)−a1 +a2y2

2∂y1w , w−gK (x))= 0 in [0,T )×R3

w(T,x,z) = (−z)∨gΘ (x)∨gK (x) in R3.

T a1 a2 υmax r0 c1,1 c1,2 r1,1 r1,2 c2,1 c2,2 r2,1 r2,2

8 2.5 0.5 1 0.3 -3 0 0.3 0.6 -1 1 0.5 0.5

Table 3: Choice of parameters.

Figure 13 represents the backward reachable set of the problem, i.e. the set ofstarting points x ∈ R2 from which it is possible to reach the target Θ at the terminaltime T satisfying the state constraints at every time in [t,T ]. This set correspondsto the finite values of the function v. The solution of the HJB equation is computedusing the C++ code ROC-HJ (available at http://uma.ensta-paristech.fr/soft/ROC-HJ/) by mean of a ENO2 scheme (see Section 3.7) with RK2 timestepping.

Fig. 13: Plot of the reachable set for the proposed Zermelo’s navigation problem.Discretization parameters used: ∆x1 = 0.07,∆x2 = 0.04,∆ t = 0.01493.

3 High-order schemes for Hamilton–Jacobi–Bellman equations

by Smita Sahu1

In this section, we focus on the construction of high-order numerical schemesfor time-dependent first order Hamilton–Jacobi–Bellman (HJB) equations of thefollowing form

vt +H(x,∇v) = 0, (t,x) ∈ [0,T ]×Rn

v(0,x) = v0(x), x ∈ Rn (257)

• Assumptions on Hamiltonian and initial data: Let us denote by | · | the Eu-clidean norm on Rn (d ≥ 1).(A1) v0 is Lipschitz continuous function i.e. there exist L0 > 0 such that for everyx ,y ∈ Rn,

|v0(x)− v0(y)| ≤ L0|x− y|. (258)

(A2) H : Rn×Rn→Rn satisfies: ∀ R≥ 0, ∃CR ≥ 0, ∀ p,q,x,y ∈Rn s.t. |p|, |q| ≤ R:

1 Department of Mathematical Sciences, Durham University, United Kingdom.e-mail: smita.sahu@durham.ac.uk

|H(y, p)−H(x, p)| ≤CR(1+ |p|)|y− x|, (259)

and,|H(x,q)−H(x, p)| ≤CR(1+ |x|)|q− p|. (260)

Under assumptions (A1) and (A2) there exists a unique viscosity solution for(257) (same arguments as in Ishii [84] also discussed in Section 1). Furthermore vis locally Lipschitz continuous on [0,T ]×Rn.

For the sake of clarity, first we focus on the one-dimensional case and considerthe following simplified problem:

vt +H(x,∇v) = 0, (t,x) ∈ [0,T ]×Rv(0,x) = v0(x), x ∈ R, (261)

later on in this section we shall also discuss multidimensional dimensions. We areinterested in computing the numerical approximation of solutions of (261). In theone-dimensional case, viscosity solutions of HJB equations can be written as thespace integral of an entropy solution of hyperbolic conservation laws for which sev-eral numerical schemes are available (see references [79] [80], [53], [81], and [75]).Thanks to this link, many theoretical and numerical results has been extended totime-dependent HJB equations. For example, well-known high-order essentiallynon-oscillatory (ENO) schemes have been introduced by A. Harten et al. in [81]for hyperbolic conservation laws and then extended to HJB equations by Osher andShu [112] showing a good accuracy of approximation. However, until now and tothe best of our knowledge, an exhaustive and general study of the convergence ratesof such class of schemes is still missing (for partial results cf. [101], [68], [67] and[45]). Let us also mention as other approximation approaches the semi-Lagrangian,the discontinuous Galerkin (DG) and the semi-discrete central schemes [88]. In thesemi-Lagrangian methods [60, 44, 62] the solution is obtained through an approxi-mation of the characteristics of the problem: this permits to obtain some high-orderconvergence results in the case of smooth solutions [45, 64]. Differently from finitedifferences, the semi-Lagrangian schemes can extended to unstructured grids. DGfinite element methods were originally devised for convection-dominated problemsand later on extended to HJB equations. The DG spatial discretization was latercombined with a Runge-Kutta (RK) temporal discretization, giving birth to Runge-Kutta DG (RKDG) methods, introduced by Cockburn and Shu in [51] for hyperbolicconservation laws and later on extended for HJB equations. DG methods are flexi-ble and permits to compute complicated geometries, different boundary conditionsand various local approximations. They use compact stencils to achieve high orderaccuracy and therefore are suitable for parallel implementation. Some variations ofDG methods for HJB equations are presented in [97, 96]. Finally, other high-orderschemes have been presented by Abgrall in [1] and [2] where the method is basedon a particular decomposition of the initial data and a consequent decompositionof the Hamiltonian. In general, high-order schemes lead to a high-order of accu-racy with fewer degrees of freedom when the solution to approximate is smoothenough. This is particularly relevant for long time advection phenomena, where the

excessive artificial viscosity of the low order schemes generates solutions of littlephysical meaning. When solutions are non-smooth, instead, high-order schemes cangenerate spurious oscillations around discontinuities and their accuracy is limited tofirst-order with a higher computational cost compared to low-order methods. Thedesign of a method able to combine accurate, high-order resolution around smoothparts of the solution with cheap, monotone, low-order schemes around discontinu-ities is the main motivation behind the numerical method presented in this section.

We firstly discuss monotone first-order schemes and high-order schemes for(261). Then we consider an recent hybridization between high and low orderschemes, the filtered schemes [110, 27, 114]. The main idea behind this ap-proach was derived from the work of Froese and Oberman [73] in the contestof Monge-Ampere equations. In particular, we describe some recent results con-tained in [27, 114] where some convergence results are obtained thanks to someε-monotone properties (cf. [26]).

For monotone schemes we refer to the seminal work of Crandall and Lions [53]and for the high-order filtered scheme we follow the recent work of Bokanowski etal. [27, 114].

3.1 Filtered schemes

Let ∆ t > 0 be a constant time step, and tn = n∆ t, n ∈ [0 . . .N], and ∆x > 0 be a stepsize of a spatial grid where (xi, tn) = (i∆x,n∆ t) denote a uniform mesh with i ∈ Z.

The construction of a filtered scheme needs three ingredients:

• a monotone scheme, denoted SM

• a high–order scheme, denoted SA

• a bounded “filter” function, F : R→ R.

The high-order scheme need not be convergent nor stable; the letter A stands for“arbitrary order”, following [73]. For a start, SM will be based on a finite differencescheme. Later on we will also discuss a definition of SM based on a semi-Lagrangianscheme.

The filtered scheme SF for (261) reads :

vn+1i = SM(vn)i + ε∆ tF

(SA(vn)i−SM(vn)i

ε∆ t

), (262)

where ε = ε∆ t,∆x > 0 is a parameter satisfying

lim(∆ t,∆x)→0

ε = 0. (263)

The scheme is initialized in the standard way, i.e.

v0i := v0(xi), ∀i ∈ Z. (264)

Now we make precise some requirements on SM , SA and the function F .• SM(v)i is a monotone finite difference scheme (see Crandall and Lions [53]), writ-ten as vn+1 = SM(vn) with

SM(vn)(x) := vn(x)−∆ t hM(x,D−vn(x),D+vn(x)), (265)

and

D±v(x) :=±v(x±∆x)− v(x)∆x

,

where hM corresponds to a monotone numerical Hamiltonian that will be made pre-cise below. We will denote also SM(vn)i := SM(vn)(xi). The fully discrete schemereads, for all i ∈ Z, ∀n≥ 0:

vn+1i := vn

i −∆ t hM(xi,D−vni ,D

+vni ), (266)

D±vni :=±vn

i±1− vni

∆x. (267)

(A3) Assumptions on SM

(i) hM is a Lipschitz continuous function.(ii) (consistency) ∀x, ∀p, hM(x, p, p) = H(x, p).(iii) (monotonicity) for any functions u,v,

u≤ v =⇒ SM(u)≤ SM(v).In pratice condition (A3)-(iii) is only required at mesh points and the condition

reads

ui ≤ vi, ∀i, ⇒ SM(u)i ≤ SM(v)i, ∀i. (268)

Definition 3.1 For each pair (tn,xi), we define the local truncation error in (tn,xi)for the scheme S by the quantity

τni (∆ t,∆x) = S(vn

i ) (269)

where v(t,x) is the smooth solution of HJ equation (257). We will define the consis-tency error of the scheme by

τ(∆ t,∆x) = supi,n|τn

i |. (270)

Definition 3.2 A scheme S is consistent if

lim(∆ t,∆x)→(0,0)

τ(∆ t,∆x) = 0. (271)

Definition 3.3 A scheme S is of order p in time and order q in space if and only ifwe have that

τ(∆ t,∆x) = O(∆ t p +∆xq). (272)

The number τ(∆ t,∆x) is then an overall assessment of the error that allows us tocheck the consistency and accuracy of the method.

Remark 3.1 Under assumption (i), the consistency property (ii) is equivalent tothat, for any v ∈C2([0,T ]×R), there exists a constant CM ≥ 0 independent of ∆xsuch that ∣∣∣∣hM(x,D−v(x),D+v(x))−H(x,vx)

∣∣∣∣≤CM∆x‖∂xxv‖∞. (273)

The same statement holds true if (273) is replaced by the following consistencyerror estimate:

τSM :=∣∣∣∣v(t +∆ t,x)−SM(v(t, .))(x)

∆ t−(vt(t,x)+H(x,vx(t,x)))

∣∣∣∣≤ CM

(∆ t‖∂ttv‖∞ +∆x‖∂xxv‖∞

). (274)

Remark 3.2 Assuming (i), it is easily shown that the monotonicity property (iii) isequivalent to that hM = hM(x, p−, p+) satisfies, a.e. (x, p−, p+) ∈ R3:

∂hM

∂ p−≥ 0,

∂hM

∂ p+≤ 0, (275)

(also denoted hM = hM(·,↑,↓)), and the CFL condition

∆ t∆x

(∂hM

∂ p−(x, p−, p+)− ∂hM

∂ p+(x, p−, p+)

)≤ 1. (276)

When using finite difference schemes, it is assumed that the CFL condition (276) issatisfied, and that can be written equivalently in the form

c0∆ t∆x≤ 1, (277)

where c0 is a constant independent of ∆ t and ∆x.

Proposition 3.1 Let the Hamiltonian H and the initial data v0 be Lipschitz contin-uous (satisfies (A1)) and v0

i = v0(xi). For fixed ∆ t > 0 and ∆x > 0, let the mono-tone finite difference scheme (265) (with numerical Hamiltonian satisfies (A3)) withstandard CFL condition (277). Then there exists a constant C such that for anyn≤ T/∆ t, we have

‖v− vn‖∞ ≤C√

∆x. (278)

for ∆ t→ 0, ∆ t = c∆x.

• SA(vn) is the high-order (or of “arbitrary order”), scheme: we consider an iterativescheme of “high–order” in the form vn+1 = SA(vn), written as

SA(vn)(x) = vn(x)− (279)

∆ thA(x,Dk,−vn(x), . . . ,D−vn(x),D+vn(x), . . . ,Dk,+vn(x)),

where hA corresponds to a “high-order” numerical Hamiltonian, and

D`,±v(x) :=±v(x± `∆x)− v(x)∆x

f or `= 1, . . . ,k.

To simplify the notation we may write (279) in the more compact form

SA(vn)(x) = vn(x)−∆ thA(x,D±vn(x)) (280)

even if there is a dependency on ` in (D`,±vn(x))`=1,...,k.(A4) Assumptions on SA:

(i) hA is a Lipschitz continuous function.

(ii) (high–order consistency) There exists k ≥ 2, for all ` ∈ [1, . . . ,k], for anyfunction v = v(t,x) of class C`+1, there exists CA,` ≥ 0,

τSA :=∣∣∣∣v(t +∆ t,x)−SA(v(t, .))(x)

∆ t− (vt(t,x)+H(x,vx(t,x)))

∣∣∣∣ (281)

≤ CA,`

(∆ t`∥∥∥∂

`+1t v

∥∥∥∞

+∆x`∥∥∥∂

`+1x v

∥∥∥∞

). (282)

Here v`x denotes the `-th derivative of v w.r.t. x.

Remark 3.3 The high-order consistency implies, for all ` ∈ [1, . . . ,k], and for v ∈C`+1(R), ∣∣hA(x, . . . ,D−v,D+v, . . .)−H(x,vx)

∣∣≤ CA,`

∥∥∥∂`+1x v

∥∥∥∞

∆x`.

For example, the central finite difference scheme for (257)

SA(vni ) = vn

i −∆ thA(

vni+1− vn

i+1

∆x

). (283)

For high order in time, a typical example with k = 2 is obtained with the centeredapproximation in space and the TVD-RK2 scheme in time (or Heun scheme):

S1(vni ) := vn

i −hA(

vni+1− vn

i+1

∆x

), SA(vn

i ) :=12(v+SA1(SA1(vn

i ))). (284)

• Filter function: In the case of second order steady elliptic PDES [73] Froese andOberman used following filter function.

F(x) = sign(x)max(1−||x|−1|,0) =

x |x| ≤ 1.0 |x| ≥ 2.−x+2 1≤ x≤ 2.−x−2 −2≤ x≤−1.

Here we will use another filter function from [27, 110, 114]

F(x) := x1|x|≤1 =

x if |x| ≤ 1,0 otherwise. (285)

The idea of the filter function (285) is to keep the high–order scheme when |hA−hM| ≤ ε ( | SA−SM

∆ tε | ≤ 1⇒ SF = SM + τεF( SA−SM

τε) ≡ SA, whereas F = 0 and SF =

SM) if that bound is not satisfied, i.e., the scheme is simply given by the monotonescheme itself.

Fig. 14: (a) Froese and Oberman’s [73] filter function F , (b) New filter function Ffrom [27, 110, 114].

3.2 Consistency error for the filtered scheme:

For any regular function v = v(t,x), for all x ∈ R, t ∈ [0,T ], then for some constantC0 independent of ∆ t,∆x ,

|τSF |=∣∣∣∣v(t +∆ t,x,y)−SF(v(t, .))(x)

∆ t−(vt +H(vx))

∣∣∣∣≤CM

(∆ t‖∂ttv‖∞ +∆x‖∂xxv‖∞

)+ εC0∆ t∆x.

(286)

The filtered scheme (262) is “ε-monotone” in the sense that

ui ≤ vi, ∀ i, ⇒ SF(u)i ≤ SF(v)i + ε∆ t ‖F‖L∞ , ∀ i. (287)

with ε→ 0 as (∆ t,∆x)→ 0. This implies the convergence of the scheme by Barles-Souganidis convergence theorem (see [20, 2]).

3.3 Filtered scheme in 2D

Let us consider the two dimensional case (d = 2 in (257))vt +H(x,y,vx,vy) = 0, (t,x,y) ∈ [0,T ]×R2

v(0,x,y) = v0(x,y), x ∈ R2 (288)

Here we will discuss filtered scheme in dimension two from [114]. Let ∆ t > 0 be aconstant time steps, and tn = n∆ t, n ∈ [0 . . .N], and ∆x > 0 be a step size of a spatialgrid (xi,y j) = (i∆x, j∆x) denote a uniform mesh with i, j ∈ Z. Hence the filteredscheme is written as :

vn+1i j = SM(vn)i j + ε∆ tF

(SA(vn)i j−SM(vn)i j

ε∆ t

), (289)

where ε = ε∆ t,∆x > 0 is the same parameter as in one-dimensional case and satisfies(263). F is the same filter function (285) and SM(vn)i j is the monotone scheme [53]of following form

vn+1i j (x) = SM(vn

i j)(x) = vni j−∆ t hM(xi,y j,D−x vn

i j,D+x vn

i j,D−y vn

i j(x),D−y vn

i j), (290)

where hM is numerical Hamiltonian. For example one can consider simple Lax-Friedrich flux i.e.

hM,LF((x,y), p−, p+,q−,q+) := H((x,y),

p−+ p+

2,

q−+q+

2

)−cx

2(p+− p−)− cy

2(q+−q−).

(291)

with maxx,p,q|∂pH((x,y), p,q)| ≤ cx, max

x,p,q|∂qH((x,y), p,q)| ≤ cy and

cx∆ t∆x

+ cy∆ t∆x≤ 1

(local bounds for cx,cy can also be used, see for instance [112]). SA is the high orderscheme:

SA(vni j)(x) = vn

i j)−∆ thA(D−x vni j,D

+x vn

i j,D−y vn

i j,D−y vn

i j), (292)

For example, the centered finite difference scheme is based on SA as in (284), and

S0(vn)i, j := vni, j−∆ tH

((xi,y j),

vni+1, j− vn

i−1, j

2∆x,

vni, j+1− vn

i, j−1

2∆x

). (293)

Similarly one can extend the scheme to any dimensions. At the end of this sectionwe will also show a three-dimensional numerical example.

3.4 A convergence result for the filtered scheme

The following theorem gives several basic convergence results for the filteredscheme. Note that the high-order assumption (A4) will not be necessary to get theerror estimates (i)-(ii). It will be only used to get a high-order consistency error es-timate in the regular case (part (iii)). Globally the scheme will have just an O(

√∆x)

rate of convergence for Lipschitz continuous solutions because the jumps in thegradient prevent high-order accuracy on the kinks.

Theorem 3.1 Assume (A1), and let v0 be bounded. We assume also that SM satisfies(A2), and |F | ≤ 1. Let vn denote the filtered scheme (262) and v is the exact solutionof (257). Assume

0 < ε ≤ c0√

∆x (294)

for some constant c0 > 0.(i) The scheme vn satisfies the Crandall-Lions estimate

‖v− vn‖∞ ≤C√

∆x, ∀ n = 0, ...,N. (295)

for some constant C independent of ∆x.(ii) (First order convergence for classical solutions.) If furthermore, the exact solu-tion v belongs to C2([0,T ]×R), and ε ≤ c0∆x (instead of (294)), then, we have

‖v− vn‖∞ ≤C∆x, n = 0, ...,N, (296)

for some constant C independent of ∆x.(iii) (Local high-order consistency.) Assume that SA is a high–order scheme satisfy-ing (A4) for some k ≥ 2. Let 1≤ `≤ k and v be a C`+1 function in a neighborhoodof a point (t,x) ∈ (0,T )×R. Assume that

(CA,1 +CM)

(‖vtt‖∞ ∆ t +‖vxx‖∞ ∆x

)≤ ε . (297)

Then, for sufficiently small tn− t, xi− x, ∆ t, ∆x, it holds

SF(vn)i = SA(vn)i

and, in particular, a local high-order consistency error for the filtered scheme SF

holds:τSF (vn)i ≡ τSA(vn)i = O(∆x`)

(the consistency error τSA is defined in (281)).

For the proof we refer reader to see [27].

Remark 3.4 In our simulations, in the time-dependent setting, we will use ε = c1∆xwhere c1 is a constant. It is natural to take ε ≤ c0

√∆x for the convergence of the

scheme. It is also natural to take ε smaller than c1∆x so that when the solution isonly C2 differentiable we still have the estimate (ii) of Theorem 3.1. Finally in orderto switch to high order in a convenient way we will see in section 3.6 that a lowerbound for ε should be also of the form c1∆x. with a specific estimate for the constantc1. This choice is different from the one of [110], for stationary equations, where ε

of the order of√

∆x is used.

Remark 3.5 Filtered semi-Lagrangian scheme. Let us consider the case of

H(x, p) := minb∈B

maxa∈A− f (x,a,b).p− `(x,a,b), (298)

where A ⊂ Rm and B ⊂ Rn are non-empty compact sets (with m,n ≥ 1), f : Rn×A×B→ Rn and ` : Rn×A×B→ R are Lipschitz continuous w.r.t. x: ∃ L ≥ 0,∀ (a,b) ∈ A×B, ∀ x,y:

max(| f (x,a,b)− f (y,a,b)|, |`(x,a,b)− `(y,a,b)|) ≤ L|x− y|. (299)

(We notice that (A2) is satisfied for Hamiltonian functions such as (298).) Let [v]denote the P1-interpolation of v in dimension one on the mesh (xi), i.e.

x ∈ [xi,xi+1] ⇒ [v](x) :=xi+1− x

∆xvi +

x− xi

∆xvi+1. (300)

Then a monotone semi-Lagrangian scheme can be defined as follows:

SM(vn)i := mina∈A

maxb∈B

([vn](xi +∆ t f (xi,a,b)

)+∆ t`(xi,a,b)

). (301)

A filtered scheme based on semi-Lagrangian can then be defined by (262) and (301).Convergence result as well as error estimates could also be obtained in this frame-work. (For error estimates for the monotone semi-Lagrangian scheme, we referto [117, 62].)

3.5 Adding a limiter

The basic filtered scheme (262) is designed to be of high–order where the solutionis regular. However, for instance in the case of front propagation, it can be observedthat the filter scheme may let small errors occur near extrema, when two possibledirections of propagation occur in the same cell.

This is the case for instance near a minima for an eikonal equation. In order toimprove the scheme near extrema, we propose to introduce a limiter before doing

the filtering process. Limiting correction will be needed only when there is someviscosity aspect (it is not needed for advection).

Let us consider the case of front propagation, i.e., equation of type (257), nowwith

H(x,vx) = maxa∈A

(f (x,a)vx

)(302)

(i.e., no distributive cost in the Hamiltonian function).In the one-dimensional case, a viscosity aspect may occur at a minima detected

at mesh point xi if

mina f (xi,a)≤ 0 and maxa f (xi,a)≥ 0. (303)

In that case, the solution should not go below the local minima around this point,i.e., we want

vn+1i ≥ vmin,i := min(vn

i−1,vni ,v

ni+1), (304)

and, in the same way, we want to impose that

vn+1i ≤ vmax,i := max(vn

i−1,vni ,v

ni+1). (305)

If we consider the high-order scheme to be of the form vn+1i = vn

i −∆ thA(vn),then the limiting process amovnts to saying that

hA(vn)i ≤ hmaxi :=

vni − vmin,i

∆ t.

andhA(vn)i ≥ hmin

i :=vn

i − vmax,i

∆ t.

This amounts to define a limited hA such that, if (303) holds at mesh point xi, then

hA(vn)i := min(

max(hA(vn)i, hmini ), hmax

i

).

and, otherwise,hA

i :≡ hAi .

Then the filtering process is the same, using hA instead of hA for the definition ofthe high-order scheme SF .

For two and three dimensional equations a similar limiter could be developedin order to make the scheme more efficient at singular regions. However, for thenumerical tests of the next section (in two and three dimensions) we will simplylimit the scheme by using an equivalent of (304)-(305). Hence, instead of the schemevalue vn+1

i j = SA(vn)i j for the high–order scheme, we will update the value by

vn+1i j = min(max(SA(vn)i j,vmin

i j ),vmaxi j ), (306)

where vmini j = min(vn

i j,vni±1, j,v

ni, j±1) and vmax

i j = max(vni j,v

ni±1, j,v

ni, j±1).

3.6 Choice of ε

The scheme should switch to a high–order scheme when some regularity of the datais detected, and in that case we should have∣∣∣∣SA(v)−SM(v)

ε∆ t

∣∣∣∣ = ∣∣∣∣hA(·)−hM(·)ε

∣∣∣∣ ≤ 1.

In a region where a function v = v(x) is regular enough, by using Taylor expan-sions, zero order terms in hA(x,D±v) and hM(x,D±v) vanish (they are both equalto H(x,vx(x))) and it remains an estimate of order ∆x. More precisely, by using thehigh–order property (A4) we have

hA(xi,D±vi) = H(xi,vx(xi))+O(∆x2).

On the other hand, by using Taylor expansions,

Dv±i = vx(xi)±12

vxx(xi)∆x+O(∆x2),

hence, denoting hM = hM(x,v−,v+), it holds at points where hM is regular,

hM(xi,Dv−i ,Dv+i ) = H(xi,vx(xi))+12

vxx(xi)

(∂hM

i∂v+

− ∂hMi

∂v−

)+O(∆x2).

Therefore,

|hA(v)−hM(v)|= 12|vxx(xi)|

∣∣∣∣∂hMi

∂v+− ∂hM

i∂u−

∣∣∣∣∆x+O(∆x2).

hence we will make the choice to take ε roughly such that

12|vxx(xi)|

∣∣∣∣∂hMi

∂v+− ∂hM

i∂v−

∣∣∣∣∆x≤ ε (307)

(where hMi = hM(xi,vx(xi),vx(xi))). Therefore, if at some point xi (307) holds, then

the scheme will switch to the high-order scheme. Otherwise, when the expecta-tions from hM and hA are different enough, the scheme will switch to the monotonescheme. In conclusion we have upper and lower bounds for the switching parameterε:

• Choose ε ≤ c0√

∆x for some constant c0 > 0 in order that the convergence anderror estimate result holds (see Theorem 3.1).

• Choose ε ≥ c1∆x, where c1 is sufficiently large. This constant should bechoosen roughly such that

12‖vxx‖∞

∥∥∥∥∂hM

∂v+(.,vx,vx)−

∂hM

∂v−(.,vx,vx)

∥∥∥∥∞

≤ c1.

where the range of values of vx and vxx can be estimated, in general, from thevalues of (v0)x, (v0)xx and the Hamiltonian function H. Then the scheme isexpected to switch to the high-order scheme where the solution is regular.

3.7 An essentially non-oscillatory (ENO) scheme of second order

We recall here a simple ENO method of order two based on the work of Osher andShu [112] for Hamilton Jacobi equation (the ENO method was designed by Hartenet al. [81] for the approximation solution of non-linear conservation law). Let m bethe minmod function defined by

m(a,b) =

a if |a| ≤ |b|, ab > 0b if |b|< |b|, ab > 00 ifab≤ 0

(308)

(other functions can be considered such as m(a,b) = a if |a| ≤ |b| and m(a,b) = botherwise). Let D±v j =±(v j±1− v j)/∆x and

D2v j :=v j+1−2v j + v j−1

∆x2 .

Then the right and left ENO approximation of the derivative can be defined by

D±v j = D±v j∓12

∆x m(D2v j,D2v j±1)

and the ENO (Euler forward) scheme by

S0(v) j := v j−∆ thM(x j, D−v j, D+v j).

The corresponding RK2 scheme can then be defined by S(v) = 12 (v+S0(S0(v))).

3.8 Numerical examples.

In this section we will present numerical examples where we compare the filteredscheme [27, 114] with the high order ENO schemes presented in the subsection3.7 (for more details see [112]), and central finite difference schemes. For time dis-

cretization we will use the Heun scheme(see the reference [75]). Hereafter, in all thenumerical test we will use the centered scheme (284) as high-order scheme and thescheme in (265) as the monotone block with the filter function (285). For the choiceof the switching parameter ε we will be precise in each example. Numerically wehave observed that taking ε = c1∆x with c1 sufficiently large does not change muchthe numerical results, so we will precise in the value of c1 in each example. All thetested filtered schemes (apart from the obstacle equations) fit within the presentedconvergence framework, so in particular there is a theoretical convergence of order√

∆x under the usual CFL condition. In the case of front propagation, it can be ob-served that the filtered scheme may produce small errors near extrema, when twopossible directions of propagation occur in the same cell. For improving the orderin all numerical tests before doing the filtering process, limiter correction will beneeded (as defined in subsection 3.5). In the last three examples of this subsection,the filtered scheme is extended to higher dimensions with the addition of limiter. Inthe front merging case regularity is lost, in that case the local error in the L2 norm iscomputed in some subdomain D, which, at a given time tn, corresponds to

eL2loc

:=

(∆x ∑i, xi∈D

|v− vni |2)1/2

and similarly L1 and L∞ errors also computed.

Example 3.1 (Advection equation). In this test we consider the linear advectionequation in one dimension

vt + vx = 0, t > 0, x ∈ (−2,2),v(x,0) = v0(x), x ∈ (−2,2) (309)

with periodic boundary conditions on (−2,2), terminal time T = 0.5 and the fol-lowing initial data:

v0(x) =−max(0,1−|x|2)4. (310)

This ”smooth” initial data is chosen in order to have at least a 3rd order continuousderivative at x =±1. The monotone upwind Hamiltonian is used (hM(x,v−,v+) :=v−). Results are given in Table 4 for the errors in L2 norms, where is compared thecentral finite difference scheme, the ENO scheme (second order ENO scheme) withRK2 in time, as we discussed (for more details see reference [75] and [112]), andthe filtered scheme using ε = 4∆x. In this test the CFL number (= ∆ t

∆x ) is 0.37. Errorsare numerically comparable in that case, all schemes are second order, also centralfinite difference scheme is numerically stable without filtering. More precisely weobserve that central finite difference scheme and filtered scheme give identical re-sults, which means that the filtering has no effect here. So the filtering at least doesnot deteriorate the good behavior of the central finite difference scheme. (Resultsare similar for the L1 and L∞ errors.) Then, we have also tested a third order fil-tered scheme. More precisely, to settle the third order scheme, the derivative vx wasestimated using a third order backward difference:

vx(xi)≡1

∆ t

(116

v(xi)−3v(xi−1)+32

v(xi−2)−13

v(xi−3)

)≡ (vx)i.

and the corresponding high order Hamiltonian, simply hA(D∓v)i := H((vx)i). Forthe time descratization we recall TVD-RK3 method [75]:

vn,1i := vn

i −∆ thA(D∓vni ). (311)

vn,2i :=

34(vn

i +14

vn,1i −∆ thA(D∓vn,1

i ). (312)

vn+1i = SA(vn

i ) :=13

vni +

23

vn,2i −

23

∆ thA(D∓vn,2i ). (313)

Results are given in Table 5, using CFL= 0.3. It is indeed also observed near tothird order convergence. This is only true for small enough CFL numbers though(CFL≤ 0.35), otherwise it was numerically observed a switch to second order con-vergence.

Filtered ε = 4∆x Centered ENO2M N L2 error order L2 error order L2 error order

20 4 4.97E-02 - 4.97E-02 - 7.95E-02 -40 8 1.26E-02 1.98 1.26E-02 1.98 2.29E-02 1.7980 16 3.07E-03 2.03 3.07E-03 2.03 5.96E-03 1.95160 32 7.66E-04 2.00 7.66E-04 2.00 1.51E-03 1.98320 64 1.90E-04 2.01 1.90E-04 2.01 3.77E-04 2.00640 128 4.76E-05 2.00 4.76E-05 2.00 9.41E-05 2.00

Table 4: (Example 3.1 with initial data (310)) Global L2 errors for filtered, centralfinite difference scheme (centered) and second order ENO scheme with RK2 intime.

M N L1 error order L2 error order L∞ error order

20 4 1.12E-01 - 7.74E-01 - 8.82E-02 -40 8 1.67E-02 2.78 1.19E-02 2.71 1.41E-02 2.6480 16 2.21E-03 2.92 1.60E-03 2.89 1.86E-03 2.93160 32 2.77E-04 2.99 2.07E-04 2.95 2.87E-04 2.69320 64 3.43E-05 3.02 2.64E-05 2.97 4.78E-05 2.58640 128 4.51E-06 2.93 3.43E-06 2.94 7.26E-06 2.72

Table 5: (Example 3.1 with initial data (310)) Global Errors for the third order filterscheme (ε = 4∆x).

Example 3.2 Eikonal equation. We now consider the case of

vt + |vx|= 0, t ∈ (0,T ), x ∈ (−2,2), (314)

v(0,x) = v0(x) := max(0,1− x2)4, x ∈ (−2,2) (315)

In Table 6 we compare, filtered schemes (with ε = 4∆x) with central finite differencescheme (unconditionally unstable) and ENO (second order) scheme with RK2 intime. Using CFL= 0.37 and terminal time T = 0.3. (Results are similar for the L1

and the L∞ errors.) In this example central finite difference scheme alone is unstable.Also, filtered and ENO schemes are numerically comparable in that case. A limiteris added on top of the evaluation of scheme in order to improve the numerical resultsnear extrema before filtering.

Filtered ε = 4∆x Centered ENO2M N L2 error order L2 error order L2 error order

40 8 1.06E-02 1.91 1.18E-01 - 1.65E-01 1.7280 15 3.42E-03 1.56 1.14E-01 0.06 4.43E-03 1.90160 30 7.12E-04 2.26 1.13E-01 0.00 1.20E-03 1.88320 59 1.74E-04 2.03 1.13E-01 0.00 3.25E-04 1.89640 118 4.32E-05 2.01 1.13E-01 0.00 8.29E-05 1.97

Table 6: (Example 3.2) with initial data (315) L2 errors for filtered scheme, centralfinite difference scheme, second order ENO scheme with RK2 in time.

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1.5

−1

−0.5

0

0.5

1

1.5

t=0

Exact

Scheme

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1.5

−1

−0.5

0

0.5

1

1.5

t=0.3

Exact

Scheme

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−1.5

−1

−0.5

0

0.5

1

1.5

t=0.3

Exact

Scheme

Fig. 15: (Example 3.2) With initial data (315) (left), and plots at time T = 0.3 bycentral finite difference scheme - middle - and filtered scheme - right (M = 160mesh points).

Example 3.3 Anisotropic eikonal equation. We now consider the case of

vt +bvx + c|vx|= 0, t ∈ (0,T ), x ∈ (−2,2), (316)

with the same initial data (315) and b = 0.5 and c = 1 with periodic boundarycondition and terminal time is T = 0.3 and CFL=0.37.

In Table 7 we compare, L2 errors of filtered schemes (with ε = 10∆x) with centralfinite difference scheme and ENO (second order) scheme with RK2 in time. UsingCFL= 0.37 and terminal time T = 0.3. Results are similar for the L1 and the L∞

errors. In this example the limiting process before filtering was necessary in orderto obtain the second order behavior for the filtered scheme. Filtered scheme showsnice results.

Errors Filtered ε = 10∆x Centered ENOM N error order error order error order

40 8 1.20E-02 2.00 2.12E-02 1.53 3.58E-02 1.3780 15 2.80E-03 2.10 7.98E-03 1.41 1.13E-02 1.66160 30 6.94E-03 2.01 3.22E-03 1.31 3.49E-03 1.70320 59 1.70E-03 2.03 1.31E-03 1.30 1.12E-03 1.64640 118 4.26E-04 2.00 9.09E-04 0.53 3.59E-03 1.64

Table 7: (Example 3.3) with initial data (315). Global L2 errors for filtered, centralFinite difference scheme and second order ENO scheme with RK2 in time.

Example 3.4 (Advection equation with an obstacle.) Here we consider an obsta-cle problem, which is taken from [25]:

min(vt + vx, v−g(x)) = 0, t > 0,x ∈ [−1,1], (317)v0(x) = 0.5+ sin(πx) x ∈ [−1,1], (318)

together with periodic boundary condition. The obstacle function is g(x) := sin(πx).In this case exact solution is given by:

v(t,x) :=

max(v0(x−at),g(x)) if t < 1

3max(v0(x−at),g(x),−10x∈ [0.5,1]) if t ∈ [ 1

3 ,13 +

12 ],

max(v0(x−at),g(x),1x∈ [−1,t− 13− 1

2 ]∪[0.5,1]) if t ∈ [ 1

3 +12 ,1],

(319)

Remark 3.6 For an obstacle problem of the form min(vt +H(vx),v−g(x)) = 0, thefinite difference scheme used here is

vn+1i := max(vn+1,1, g(xi)),

where vn+1,1 is the usual one-time step filtered scheme for the HJ equation vt +H(vx) = 0 starting from the previous scheme values vn. Such schemes for obstacleproblems are well known (see for instance [28, 25]).

Results are given in Table 8, for terminal time t = 0.5. Errors are computed awayfrom singular points, i.e., in the region [−1,1]\

(∪i=1,3 [si−δ ,si +δ ]

)(where s1 =

−0.1349733,s2 = 0.5 and s3 = 2/3 are the three singular points. Filtered scheme isnumerically of second order (ENO gives comparable results here).

Errors filtered ε = 5∆x centered ENO2M N error order error order error order

40 20 7.93E-03 2.03 1.63E-02 1.54 2.14E-02 1.5980 40 1.84E-03 2.10 2.98E-02 -0.87 7.75E-03 1.46160 80 3.92E-04 2.24 1.46E-02 1.03 1.07E-03 2.86320 160 9.67E-05 2.02 8.02E-03 0.86 2.72E-04 1.97640 320 2.40E-05 2.01 4.10E-03 0.97 6.92E-05 1.98

Table 8: (Example 3.4) Local L∞ errors away from singular points, for filteredscheme, centeral finite difference scheme, and second order ENO scheme.

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0

Exact

Scheme

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0.33333

Exact

Scheme

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0.5

Exact

Scheme

Fig. 16: (Example 3.4) Plots at T=0 (initial data), T=0.3, T=0.5.

Example 3.5 Eikonal equation with an obstacle. We consider an Eikonal equa-tion with an obstacle term, also taken from [25]:

min(vt + |vx|,v−g(x)) = 0, t > 0,x ∈ [−1,1], (320)v0(x) = 0.5+ sin(πx) x ∈ [−1,1], (321)

with periodic boundary condition on (−1,1) and g(x) = sin(πx). The exact solutionis given by: v(t,x) = max(v(t,x),g(x)) where v is the solution of the Eikonal equa-tion vt + |vx| = 0. The formula v(t,x) = miny∈[x−t,x+t] v0(y) holds, which simplifiesto

v(t,x) :=

v0(x+ t) if x <−0.5− t−0.5 if x ∈ [−0.5− t,−0.5+ t],min(v0(x− t),v0(x+ t)) if x≥−0.5+ t,

(322)

Results are given in Table 9 for terminal time T = 0.2 and it contains only er-rors in L2 norms of filtered and second order ENO scheme. Central finite differencescheme is unconditionally unstable. Plots are also shown in Figure 17 for differenttimes (for t ≥ 1

3 solution remains unchanged).

Errors filtered ENO2M error order error order

40 3.74E-03 - 6.85E-03 -80 6.26E-04 2.58 2.12E-03 1.69

160 1.13E-04 2.47 6.80E-04 1.64320 2.26E-05 2.32 2.18E-04 1.64640 5.50E-06 2.04 6.96E-05 1.65

Table 9: (Example 3.5) Filtered scheme and second order ENO scheme at time t =0.2

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0

Exact

Scheme

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0.2

Exact

Scheme

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1.5

−1

−0.5

0

0.5

1

1.5

t=0.4

Exact

Scheme

Fig. 17: (Example 3.5) Plots at times t = 0, t = 0.2 and t = 0.4. The dark line is thenumerical solution, similar to the exact solution, and the light line is the obstaclefunction.

Example 3.6 (2D rotation) We now apply the filtered scheme to an advection equa-tion in dimension two. We consider the domain Ω = (−2.5,2.5)2 and

vt − yvx + xvy = 0, (x,y) ∈Ω , t > 0, (323)

v(0,x,y) = v0(x,y) = min(√

(x−1)2 + y2−0.5, 0.5). (324)

with Dirichlet boundary condition v(t,x) = 0.5, x /∈Ω and final time T := π

2 . In thisexample for the monotone scheme (290) we have taken nvmerical Hamiltonian as

hM(v−x ,v+x ,v−y ,v

+y ) := max(0, f1(a,x,y))v−x +min(0, f1(a,x,y))v+x

+max(0, f2(a,x,y))v−y +min(0, f2(a,x,y))v+y .(325)

and for the high order scheme we have taken a central finite difference discretizationin space and RK2 in time (293). The filtered scheme is set with ε = 4∆x, and theCFL condition is

C(∆ t∆x

+∆x∆y

)≤ 1. (326)

Hence, computations are done for CFL is 0.6126. The central finite differencescheme is unconditionally unstable. Local errors have been computed in the regionD := (x,y)|v(tn,x,y)| ≤ 0.1.

Filtered ε = 4∆x Centered ENO2Mx = Nx N L2 error order L2 error order L2 error order

20 50 7.85E-02 - 7.85E-02 - 1.58E-01 -40 100 1.60E-02 2.30 1.59E-02 2.30 2.72E-02 2.5480 200 5.99E-03 1.42 7.54E-03 1.08 5.14E-03 2.41160 400 1.50E-03 2.00 2.34E-03 1.69 1.21E-03 2.08320 800 3.25E-04 2.20 8.10E-04 1.53 3.01E-04 2.01

Table 10: (Example 3.6) Local L2 errors for filtered scheme, central finite differenceand second order ENO schemes with RK2 in time.

Example 3.7 Eikonal equation in 2D In this example we consider the eikonalequation

vt + |∇v|= 0, (x,y) ∈Ω , t > 0 (327)

in the domain Ω := (−3,3)2. The initial data is defined by

v0(x,y) = 0.5−0.5 max(

max(0,1− (x−1)2− y2

1− r20

)4, max(0,1− (x+1)2− y2

1− r20

)4).

The zero-level set of v0 corresponds to two separates circles or radius r0 and cen-tered in A = (1,0) and B = (−1,0) respectively. Dirchlet boundary conditions areused as the previous example. The monotone Hamiltonian hM used in the filteredscheme is Lax-Friedrichs (291) and CFL = 0.37. Also, the 2D limiter (306) is usedfor the filtered scheme as described in Section 3.5. It is needed in order to get a goodsecond order behavior at extrema of the numerical solution. The filter coefficient isset to ε = 20∆x as in the previous example. Numerical results are given in Table 11,showing the global L2 errors for the filtered scheme, central finite difference scheme,and second order ENO scheme, at time t = 0.6. We observe that the central finitedifference scheme has some unstabilities for fine mesh, while the filtered performsas expected.

Example 3.8 Merging of regular fronts in 3DThe motivation to present this example is that if we have more than two fronts, thenthe filtered scheme is still of second order. We consider the equation

vt + |∇v|= 0 (x,y,z) ∈Ω = (−2,2)3, t ∈ [0,0.6] (328)

with the initial data

filtered (ε = 20∆x) centered ENO2Mx Ny L2 error order L2 error order L2 error order

25 25 5.39E-01 - 6.00E-01 - 5.84E-01 -50 50 1.82E-01 1.57 2.25E-01 1.41 2.11E-01 1.47100 100 3.72E-02 2.29 8.46E-02 1.41 6.88E-02 1.62200 200 9.36E-03 1.99 3.53E-02 1.26 2.02E-02 1.76400 400 2.36E-03 1.99 1.36E-01 -1.95 5.98E-03 1.76

Table 11: (Example 3.7) Global L2 errors for filtered scheme, centered and secondorder ENO schemes.

①

②

①

②

①

①

②

①

②

①

Fig. 18: (Example 3.7) Plots at times t = 0 (top) and t = π/2 (bottom) for the filteredscheme with M = 50 mesh points. The figures to the right represent the 0-level sets.

vk(x,y,z, ,0) = r0− r0 max(

0,1− (x− xk)

2− (y− yk)2− (z− zk)

2

1− r20

)4

,k = 1, ..,5.

Where r0 = 0.25 and centers (xk,yk,zk) are (1,0,0), (−1,0,0), (−1,0,0), (0,−1,0),(0,1,0) for k = 1, ...,5 respectively with Dirichlet boundary condition. In this exam-ple we have extended the filtered scheme (290) to dimension three. For monotonescheme (290) we have used Lax-Friedrich (291) flux (in 3D) as numerical mono-tone Hamiltonian. For high order scheme we used central finite difference scheme(293) (for 3D). It is clear from the Table 12 that central finite difference is not stable(as expected). In the table 12 error calculations are local. We eliminated the ballB(Pk,ε1) of radius ε1 = 0.2 and Pk are points where fronts touches for k = 1, ...,5respectively. It is clear from the figure 18 that singularities propagates hence onecannot expect to have high order convergence.

Fig. 19: (Example 3.5) plots at time T = 0 (left) and T = 0.6 (right) for the filteredscheme with mesh point M = N = P = 100.

Errors Filtered ε = 20∆x Centered ENOM N P L2 error order L2 error order L2 error order

25 25 25 1.43E-01 - 1.69E-01 - 1.30E-01 -50 50 50 6.37E-02 1.17 1.54E-01 0.14 4.18E-02 1.64100 100 100 1.50E-02 2.09 1.46E-01 0.08 1.20E-02 1.79200 200 200 3.95E-03 1.92 2.57E+01 -7.46 3.75E-03 1.68

Table 12: (Example 3.8), Local errors for filtered scheme, centered scheme, andENO second order scheme.

3.9 Conclusion

We discussed a recent high-order filtered scheme [110, 27, 114] which is obtainedby mixing a monotone scheme with a high-order method. The filter is designedin order to get a high–order scheme when the solution is smooth and a low-ordermonotone scheme otherwise. With respect to other high-order schemes proposed inthe literature, the filtered scheme has simple structure which results in a rather easyimplementation, and error bounds of order

√∆x. It is interesting to note that these

bounds give the same order of the Crandall-Lions estimates for monotone schemesand they hold both for time-dependent or stationary equations. More importantly,in the case the solution is smooth we have obtained a high-order consistency errorestimate. We have discussed several numerical examples in several dimensions thatthe filtered scheme stabilizes unstable scheme and gives high-order convergence.

4 Some Recent Applications of Hamilton–Jacobi–Bellmanequations

by Adriano Festa1 and Achille Sassi2

For its same nature of describing some extremal geometry in generalizations ofproblems from the calculus of variations, an accurate resolution of an Hamilton Ja-cobi equation is a critical point in many applicative problems. In this section, with-out the presumption to be complete, we present some applications where Hamilton-Jacobi equations naturally appear.

In order to give an idea of the size of the various fields involved, it will be pre-sented some problems coming from different areas: the Shape-from-Shading, a clas-sic problem of computer vision, a navigation problem applied to sailing boats, afront propagation problem, coming, for example, from geological applications anda case of hybrid control applied to a vehicle fuel consumption.

In this section we deal with the following stationary, first order, Hamilton Jacobiequation on an open subset Ω of RN

λv(x)+H (x,∇v(x)) = 0 x ∈Ω

v(x) = g(x) x ∈ ∂Ω(329)

where the Hamiltonian can take different forms depending on the special applica-tion. In the following, most of the cases are contained in the class of hamiltoniansof the form

H (x, p) := sup〈− f (x,u), p〉− `(x,u) | u ∈U,typical for being connected to optimal control problems with infinite temporal hori-zon. The space of controls U is assumed compact and convex and f , ` are functionssufficiently regular (a detailed presentation in [18]).

In this section this equation is solved with a standard semi-lagrangian scheme(cf. the more general presentation in Section 1.5). We briefly present the numericalscheme: for further details we refer to [17] and to the monograph [62].

Let us consider a structured grid on Ω made by a family of simplices S j, suchthat Ω ∈ ∪ jS j. Name xi, i = 1, ...,N the nodes of the triangulation,

h := maxj

diam(S j), (330)

the size of the mesh (diam(S) is the diameter of the set S). Let be G the set of theinternal nodes of the grid and, consequently, ∂G is the set of its boundary points; inthe case of a bounded Ω we call also Φ the nodes corresponding to the set Rn \Ω ,those nodes typically act as ghost nodes. We remark that this discretization spaceincludes the classical case of regular meshes.

1 Johann Radon Institute for Computational and Applied Mathematics, Linz, Austria.e-mail: adriano.festa@oeaw.ac.at2 Applied Mathematics Department, ENSTA ParisTech, France.e-mail: achille.sassi@ensta-paristech.fr

We map all the values at the nodes in V = (V (1), ...,V (N)). By a standard semi-lagrangian discretization of (329), it is possible to obtain the following scheme infixed point form

V = G(V ), (331)

where G : RN → RN is defined component-wise, for a parameter h > 0 by

[G(V )]i =

minu∈U

1

1+λh I[V ](xi−h f (xi,a,b))−h`(xi,a,b)

xi ∈ G ,

g(xi) xi ∈ ∂G ,+∞ xi ∈Φ .

4.1 The Shape from Shading problem

The Shape-from-Shading (SfS) problem consists in reconstructing the 3D shape ofa scene from the brightness variation (shading) in a greylevel photograph of thatscene (see Fig. 20). The study of the Shape-from-Shading problem started in the

Fig. 20: Initial image (left) and reconstructed surface (right).

70s (see [83] and references therein) and since then a huge number of papers haveappeared on this subject. More recently, the mathematical community was inter-ested in Shape-from-Shading since its formulation is based on a first order partialdifferential equation of Hamilton-Jacobi type.

To render the problem mathematically manageable, we assume (see [58]):

H1 - The image reflects the light uniformly and then the albedo (ratio between energyreflected and energy captured) is constant.

H2 - The material is Lambertian, i.e. the intensity of the reflected light is proportionalto the scalar product between the direction of the light and the normal to thesurface.

H3 - The light source is unique and the rays of light which lighten the scene areparallel.

H4 - Multiple refections are negligible.H5 - The aberrations of the objective are negligible.H6 - The distance between the scene and the objective is much larger than that be-

tween the objective and the CCD sensor.H7 - The perspective deformations are negligible.H8 - The scene is completely visible by the camera, i.e. there are not hidden regions.

Remark 4. We point out that assumptions H1 and H2 are false in the cases of reflec-tive materials (as metal or glass).

Assumption H3 means that we can describe the light direction by a unique andconstant vector. Note that this is true in the case where the light source is very farfrom the scene (for example, if the scene is illuminated by the sun).

Assumption H7 means that the camera is very far from the scene. For the caseof perspective SfS where the camera is to a finite distance from the object, we referi.e. to [32].

Let us briefly derive the model for Shape-from-Shading under general assump-tions. Let Ω be a bounded set of R2 and let v(x,y) : Ω → R be a surface whichrepresents the three-dimensional surface we want to reconstruct. The partial dif-ferential equation related to the Shape-from-Shading model can be derived by the“image irradiance equation”

R(n(x,y)) = I(x,y) (332)

where I is the brightness function measured at all points (x,y) in the image, R isthe reflectance function giving the value of the light reflection on the surface as afunction of its orientation (i.e. of its normal) and n(x,y) is the unit normal to thesurface at point (x,y,v(x,y)). If the surface is smooth we have

n(x;y) =(−vx(x,y),−vy(x,y),1)√

1+ |∇v(x,y)|2. (333)

The brightness function I(x,y) is the datum in the model since it is measured oneach pixel of the image in terms of a gray level, for example from 0 = black to255 = white or, after a rescaling, from 0 to 1. To construct a continuous model wewill assume hereafter that I takes real values in the interval [0,1].

Clearly, equation (332) can be written in different ways depending on whichassumptions H1-H8 hold true.

Under H1-H8, we have

R(n(x,y)) = ω ·n(x,y) (334)

Fig. 21: Two different surfaces corresponding to the same image I.

where ω ∈ R3 is a unit vector which indicates the direction of the light source. Then,equation (332) can be written, using (333)

I(x)√

1+ |Dv(x,y)|2 + 〈(ω1,ω2),∇v(x,y)〉−ω3 = 0, (x,y) ∈Ω (335)

which is a first order non-linear partial differential equation of the Hamilton-Jacobitype.

In order to compute a solution we have to impose some boundary conditions on∂Ω and/or inside Ω . A natural choice is to consider Dirichlet type boundary condi-tions to take into account at least two different possibilities. The first corresponds tothe assumption that the surface is standing on a flat background, i.e. we set

v(x,y) = 0 (x,y) ∈ ∂Ω . (336)

The second possibility occurs when the height of the surface on the boundary isknown

v(x,y) = g(x,y) (x,y) ∈ ∂Ω . (337)

The above boundary conditions are widely used in the literature although they areoften unrealistic since they assume a previous knowledge of the surface.

We note that in equation (335) the light source is vertical, i.e. ω = (0,0,1), thenequation simplifies to the eikonal equation

|∇u(x,y)|=(√

1I(x,y)2 −1

), (x,y) ∈Ω . (338)

Points (x,y) where I is maximal (i.e. equal to 1) correspond to the particular situationwhen ω and n point in the same direction. These points are usually called “singularpoints” and, if they exist, equation (338) is said to be degenerate. The notion ofsingular points is strictly related to that of concave/convex ambiguity which webriefly recall in the following example.

Example 1. The SFS problem is one well-known examples of ill-posed problem.Consider for example the two surfaces z1 = +

√1− x2− y2 and z2 = −z1 (see

Fig. 21). It is easy to see that they have the same brightness function I and ver-ify the same boundary condition so that they are virtually indistinguishable by themodel. As a consequence, even if we compute a viscosity solution of the equation,it is possible that the solution we obtained is different from the surface we expect.Note that this is an intrinsic problem and it can not be completely solved without amodification of the model.

In order to overcome this difficulty, the problem is usually solved by adding someinformations such as the height at the singular points (see [100]). More recently,an attempt as been made to eliminate the need for a priori additional information bymeans of the characterization of the maximal solution (see [36, 35]). A result by Ishiiand Ramaswamy [86] guarantees that if I is continuous and the number of singularpoints is finite, then a unique maximal solution exists. Following this approach,some algorithms to approximate the unique maximal solutions were proposed (seefor example [58] and references therein).

Standard cases. We compute an approximation of the viscosity solution of(335) in the case of vertical light. We assume therefore assuming that the viscositysolution is the correct solution of the Shape from Shading problem. We use a semi-Lagrangian scheme of the form (331). We are in the case considered, with (verticallight, as in (338))

c(x) =

(√1

I(x)2 −1

)x ∈Ω . (339)

Let us focus on two important points:

• In the case of maximal gray tone (I(x) = 1), we are not in the standard Hypoth-esis of non-degeneracy of the c term. We overcome this difficulty, as suggest in[35]. We regularize the problem making a truncation of c. We solve the problemwith the following cε

cε =

c(x) =

(√1

I(x)2 −1)

if c > ε

ε if c≤ ε(340)

It is possible to show that this regularized problem goes to the maximal sub-solution of the problem with ε → 0+. And that this particular solution is thecorrect one from the applicative point of view. For more details see [35, 58].

We start with a simple example, a pyramid. The datum is synthetic, we mean thatwe have build a surface and then take a picture of it. After this, we try to reconstruct

Fig. 22: Pyramid: original shape and SfS-datum.

Fig. 23: Pyramid: reconstructed shape.

the original surface. We take Ω := [−1.2,+1.2]2 and the pyramidal surface is thefollowing

s(x) =

min(1−|x1|,1−|x2|) if max(|x1|, |x2|)< 10 otherwise (341)

the synthetic datum and the original surface is shown in Figure 22. We build a nu-merical approximation, with h = 0.01, ∆ t = 0.001 and ε = 10−5. We obtain the re-sult shown in Figure 23. In this case the result is excellent. We have to say, anyway,that this is a very simple case with no points where the eikonal equation degenerates.

The second case which we consider is a bit more complicated. We consider ahalf sphere, so we find a point where f runs to zero. Also in this case we make asynthetic image from an original shape and then we try to reconstruct it. We takeΩ := [−1.2,+1.2]2, the original shape is

s(x) =

√1− x2

1− x22 if dist(x1,x2)< 1

0 otherwise(342)

the synthetic datum and the original surface is shown in Figure 24. We build a nu-merical approximation, with h = 0.01, ∆ t = 0.001 and ε = 10−5. We obtain theresult shown in Figure 25.

Fig. 24: Half sphere: original shape and SfS-datum.

Fig. 25: Half sphere: reconstructed shape.

Fig. 26: Vase: Sfs-datum and reconstructed image.

The last example is relative to a real image of a vase. It is a 256×256 image. Wehave to remark that this case is more difficult that the previous ones. We have severalpoints of maximum value of I and some noise in the SfS-datum. Furthermore wehave the problem of the boundary, where we do not now, a priori, the correct value

of s. We have chosen to impose zero across the lateral silhouette of the shape, and ahalf circle on the superior and inferior silhouette of the vase. This choice it is madeto preserve the convexity of the shape also in these areas. We take ∆ t = 0.0005 andε = 10−6. The result is shown in Figure 26.

Fig. 27: Sfs-data and solution with various boundary values.

Discontinuities in the Shape-from-Shading model. In many cases there is aninterest to introduce some discontinuities in the SfS model, in this section we discussthis issue following the approach presented in [66].

Let us firstly consider a simple problem in 1D. Let the function I be

I =

√

1− x2 if −1≤ x≤ 0.2√2

2 if 0.2≤ x≤ 10 otherwise

(343)

we can see that we have a discontinuity on x = 0.2; despite this, the solution will becontinuous. For this reason we can see that changing the boundary condition of theproblem, the solution will be the maximal Lipschitz solution that verifies continu-ously the boundary condition. To see this we have solved this simple monodimen-sional problem with various Dirichlet condition, in particular we require v(−1) = 0,and v(1) = −1,0.5,0,0.5,1. With h = 0.01 and ∆ t = 0.002, we obtain the resultsshown in Figure 27.

We can realize, in this way, an intrinsic limit of the model. It can not representan object with discontinuities. We make another example that is more complicatedand more close to a real application.

We consider a simplified sfs-datum for the Basilica of Saint Paul Outside theWalls in Rome, as shown in Figure 28. We have not the correct boundary value onthe silhouette of the image and on the discontinuities, so we impose simply v ≡0 on the boundary. Computing the equation with ∆ t = 0.001 we get the solutiondescribed on Figure 29.

Fig. 28: Basilica of Saint Paul Outside the Walls: satellite image and simplified Sfs-data.

Fig. 29: Test 3: reconstructed shape without boundary data.

We can see as the main features of the shape, as the inclination of the roofs arewell reconstructed. An unsatisfactory aspect is related to the boundaries: we cantry to get better results adding the correct height of the surface along the silhouetteas discussed above and, in this case, we get the solution shown on Figure 30. We

can notice a more convincing shape, but also in this case it is quite not satisfactory.For example we notice as the correct boundary conditions that we imposed are notattained in some points, creating some discontinuities on some parts of them. Thisis due to the fact that they can be not compatible with the statement of the problem.Essentially the limit which we can see, as described above, is that we cannot havediscontinuities on the viscosity solution.

We propose a different model for this problem (cf. [66] where this approach waspresented) which allows discontinuous solutions. At this point we do not care aboutthe physical interpretation of it, instead we are trying to find a solution closer to thecorrect solution. We want to solve the equationmax

|u|≤1

〈−∇v(x),

2∑

k=1ukσk(x,y)〉

=√

1I2(x,y) −1 for x ∈Ω ,

v(x) = g(x) for x ∈ ∂Ω ,(344)

with the map σ : Ω → R2,2 is

σ(x,y) =((1+ |I(x−h,y)− I(x+h,y)|)−p 0

0 (1+ |I(x,y−h)− I(x,y+h)|)−p

),

(345)where p∈R is a tuning parameter. Obviously this choice of the anisotropic evaluatorσ is a bit trivial. This pick is done for the sake of simplicity. More complicatedproposal can be found for example in [12].

In this way we use the results about the degeneracy of the dynamics permittingto the viscosity solution to be discontinuous. Of course this is, in some sense, theopposite situation with respect to the classical formulation: in this case every nonsmooth point of the surface is interpreted as discontinuity and we try to reconstructit using the data coming from the silhouette.

The results are shown in Figure 31 and in Table 13 we can see an accuracycomparison of the various procedure.

Test || · ||∞ || · ||1with no information on the boundary 1.7831 1.5818with boundary data 0.8705 0.5617with boundary and disc. detect. 0.7901 0.3062

Table 13: Test 3: Comparison between various methods. The discrete norms aredefined for v = (v1, ...,vn) as ||v||∞ := maxi|vi| and ||v||1 := ∑

ni |vi|/n

Fig. 30: Test 3: Dirichlet condition on the silhouette.

Fig. 31: Test 3: Dirichlet condition and discontinuous dynamics.

4.2 Route planning for sailing boats

Here we want to show as the resolution of an HJ equation can be used to solve aproblem of optimal route planning through the Bellman’s approach. We consider aproblem of route planning for sailing boats.

A sailboat has some restrictions to his choice of trajectories, due to some physicaldynamic basics. Simplifying a bit it can not move upwind with an angle of incidencewith respect the wind smaller than a certain angle α . The angle α is a variable thatdepend from the kind of the boat, the speed of the wind, the features of the sailsused. Typically, the goal is to minimize the time of arrival on a waypoint (buoy)which is placed upwind. So we can not move directly in the direction of the target,but we have to alternate some piece-wise regular sections of trajectory moving tothe left side and to the right. This is the typical beating (to windward).

In our model we introduce the dynamic systemy(t) = f (y(t),u(t)),y(0) = x (346)

where, said the unitary vector w = [w1,w2] the direction of wind, the function f is

f (y,u) =

u if 〈u,w〉 ≤ cos(α)0 otherwise (347)

where the space of the controls u ∈ B(0,1) unitary ball around zero.We impose the target set T = (0,1), and a constant running cost equal to one.In our first simulation we consider the case of constant wind w = [0,−1]. Instead

solving the optimal control problem, we approach the problem solving numericallythe HJ equation associated. After this delicate step, the relative optimal control isobtained solving a synthesis problem, namely

u ∈ argmin〈 f (y,u),∇v〉+1 | u ∈ B(0,1).

We can observe in Figure 32 the results. In this figure we have also traced the optimaltrajectories starting from the start, that is the line x2 =−0.9. For every starting pointwe have traced in blue, the trajectories that choose preferably the left side of therace field and in red the ones that prefer the right side. We have to remark thatfor every starting point the blue trajectory and the red one are both optimal. Arealso optimal the other trajectories with optimal angle with respect to the wind andincluded between the red and the blue one.

In the second simulation we consider the most complicated case of a piecewiseconstant wind vectorfield. We assume w as follows:

w(x) =

(0,−1) if x2 ≤−0.3(cos( 5

4 π)sin( 54 π))

if x2 ≥ 0.3(cos( 11

8 π)sin( 118 π)

)otherwise .

(348)

This fact change prominently the situation. As it is shown in Figure 33 fromthe starting points where there is just one trajectory, the optimal choice is unique,where there are sections of the same that have a different red way and blue one,there are possible infinity choices of optimal paths. The situation presented is anextreme simplification of the problem: many points are not discussed here: the lackof convexity in the control set (against the standard assumption of the method), the

Fig. 32: Sail Optimization: solution and optimal trajectories.

Fig. 33: Sail Optimization: solution and optimal trajectories with changes of wind.

possibility of an hybrid dynamics for the system (like for a change of sails cf. thelast example of this Section) and many others. For a more complete presentation ofthe subject (which has in any case many points still under investigation) we refer to[9].

4.3 Front Propagation

Here we briefly present some basic concepts of the theory of curve evolution. A caseof front propagation was already presented as example in Section 3 Example 3.8.We integrate and extend that case to the case of a front propagation with a specialattention about its propagation through a non-homogeneous medium.

We consider curves to be deforming in time. Let C (p, t) : S1× [0,T )→ R2 de-note a family of closed (embedded) curves, where t parameterizes the family and p

parametrizes the curve. Assume that this family of curves obeys the following PDE:

∂tC (p, t) = u(p, t)τ(p, t)+ c(p, t)n(p, t) (349)

with C0(p) as the initial condition. Here τ and n are, respectively, the unit tangentand the inward unit normal. This equation has the most general form and means thatthe curve is deforming with u velocity in the tangential direction and c velocity inthe normal direction. If we are just interested in the geometry of the deformation,but not in its parametrization, this flow can be further simplified following the resultof Epstein and Gage [59], i.e., if c does not depend on the parametrization, then theimage of C (p, t) that satisfies equation (349) is identical to the image of the familyof curves C (p, t) that satisfies

∂tC (p, t) = c(p, t)n(p, t). (350)

In other words, the tangential velocity does not influence the geometry of the defor-mation, just its parametrization.

Curves and surfaces can be, often, represented in an implicit form, as level sets ofa higher-dimensional surface. A typical example is the representation of a curve asthe level set of a function. The pioneering and fundamental work on the deformationof level sets was introduced in [111]. This implicit representation has a good numberof advantages, like stability, accuracy, topological changes allowed etc.

Let us represent this curve as the zero level set of an embedding function v(x, t) :R2× [0,T )→ R:

C (x, t) =(x, t) ∈ R2× [0,T ) : v(x, t) = 0

(351)

differentiating this equation and combining with (350), we obtain

∂tv = c(x, t)|∇u| (352)

where all the level sets of the function v is moving according to (349). With anappropriate initial function v0, the zero level set represent the motion of the curveC . For details see [115].

If we suppose that c(x, t) ≡ c(x) > 0 we can use an alternative representation.Supposing that T (x) : R2→ [0,+∞) is the function representing the time at whichthe curve crosses the point x, we can show that the function time-of-arrival T satis-fies

c(x)|∇T |= 1. (353)

the function T gives a representation of the motion of C as level sets, i.e. C (t) =x : T (x) = t. We can use also in this case a semiLagrangian scheme to give anapproximation of the time-of-arrival T in the case of discontinuous velocities c.

Front propagation in a non-homogeneous mediumWe consider a simple situation: there is a source of a signal in the point (−1,1) andthe velocity of the signal is not constant through the whole domain, instead, it varies

following the function c(x). We can see at this situation as an evolution of a curve(at time zero a point) through an non-homogeneous field of velocities.

We take the following value for the function c(x)

c(x1,x2) =

10 if −0.2≤ x1 ≤ 0.21 otherwise .

(354)

We get the results shown in Figure 34. The front react to the change of medium witha deformation of its shape and speed of propagation accordingly to the well knownphenomena of diffraction.

Fig. 34: Front propagation: solution (mesh and level sets).

We consider also a more complicated situation. This could represent the time ofarrival of an electromagnetic radiation of planar waves that moves from a sourceplaced in the point (−1,1). The function c(x) modelizes the index of refraction of adiscontinuous media. We take a c(x) : [−1,1]2→ (0,M] as presented in Figure 35,of the form

c(x1,x2) =

1 if x2 ≥ log(e+x1)2 + 1

103 if log(e+x1)

2 − 210 ≤ x2 ≤ log(e+x1)

2 + 110

20 if ex13 − 1

5 ≤ x2 ≤ log(e+x1)2 − 2

101 if x2 ≤− 3

20 x1− 35

2 otherwise .

(355)

the approximated solution of the problem is presented in Figure 35. We canalso, minding the physical model, try to reconstruct some trajectories of the wavesthrough the media. The results are presented in Figure 36.

It is clear that the same model has been applied to many situation of interest:flame propagation in the management of a fire, as numerical tool for some imageprocessing methods (segmentation, optic flow reconstruction) and many others.

Fig. 35: Front propagation: velocity field and solution (level sets).

Fig. 36: Front propagation: level set of the solution and some optimal trajectories.

4.4 Hybrid control

A hybrid control system is a system that, in addition to the usual continuous control,has the ability to switch between different dynamics in a discontinuous way. Hybridcars are a clear example of this kind of systems: in order to control the vehicle, thedriver has the ability to stir and operate the accelerator, switch between two engines(internal combustion and electric) and also change gear.Among the various mathematical formulations of optimal control problems for hy-brid systems, we will adopt here the one given in [31, 55]. Let I be a finite set, andconsider the controlled system (X ,Q) satisfying:

y(t) = f (y(t),Q(t),u(t)),y(0) = x, Q(0+) = q,

(356)

where x ∈Rn, and q ∈I . Here, y and Q denote respectively the continuous and thediscrete component of the state. The function f : Rn×I ×U → Rn represents thecontinuous dynamics, for a set of continuous controls given by:

U = u : (0,∞)→U |u measurable, U compact.

The trajectory undergoes discrete transitions when it enters two predefined sets A(the autonomous jump set) and C (the controlled jump set), both of them subsets ofRn×I . More precisely:

• On hitting A, the trajectory jumps to a predefined destination set D ⊂ Rn×I .The jump driven by a transition map g :Rn×I ×V →D, where V is a discretefinite control set. Denoting by τi a time at which the trajectory hits A, the newstate will be (y(τ+i ),Q(τ+i )) = g(y(τ−i ),Q(τ−i ),wi), for a control wi ∈ V .• Entering the set C, the controller can choose either to jump or not. If the con-

troller chooses to jump, then the continuous trajectory is moved to a new pointin D. Denoting by ξk one such time of jump, we will have (y(ξ−k ),Q(ξ−k )) ∈Cand (x′,q′) = (y(ξ+

k ),Q(ξ+k )) ∈ D.

The trajectory starting from x ∈ Rn with discrete state q ∈ I is therefore com-posed of branches of continuous evolution given by (356) between two discretejumps at the transition times τi or ξk.

Now, considering an optimal control problem in the infinite horizon form, andincluding all control actions in a control strategy

θ :=(u(·),wii∈N,(ξk,x′k,q

′k)k∈N

)we associate to θ a cost defined by:

J(x,q;θ) :=∫ +∞

0`(X(t),Q(t),u(t))e−λ tdt

+∞

∑i=0

Ca(X(τ−i ),Q(τ−i ),wi)e−λτi

+∞

∑k=0

Cc(X(ξ−k ),Q(ξ−k ),X(ξ+k ),Q(ξ+

k ))e−λξk , (357)

where λ > 0 is the discount factor, ` : Rn×I ×U → R+ is the running cost, Ca :A×V →R+ is the autonomous transition cost and Cc :C×D→R+ is the controlledtransition cost. The value function v of the problem is then defined as:

v(x,q) := infθ

J(x,q;θ). (358)

We point out that, in this generality, the problem requires strong assumptions to bemathematically well-posed. In particular, it should be ensured that the value function(358) is continuous, and that the so-called “Zeno executions” (i.e., the occurrenceof an infinite number of transitions in a finite time interval) are avoided. We refer

to [55] for a precise set of assumptions, whereas in the examples we will apply thenumerical technique under consideration in more general situations, showing thatthe recipe is robust enough to handle them.

First, we recall some basic analytical results about the value function (358). Via asuitable generalization of the Dynamic Programming Principle, it can be proved thatthe Bellman equation of the problem is in the form of a Quasi-Variational Inequality,and more precisely, once defined the Hamiltonian by

H(x,q, p) := sup〈− f (x,q,v), p〉− `(x,q,v) | u ∈U

and the transition operators M and N by:

M φ(x,q) := infφ(g(x,q,w))+Ca(x,q,w) | w ∈ V (x,q) ∈ A,

N φ(x,q) := infφ(x′,q′)+Cc(x,q,x′,q′) | (x′,q′) ∈ D (x,q) ∈C,

we have the following

Theorem 9 ([55]). Under the basic assumptions, the function v is the unique boundedand Holder continuous viscosity solution of:

λv(x,q)+H(x,q,∇xv(x,q)) = 0 (x,q) ∈ (Rn×I )\ (A∪C),

max(v(x,q)−N v(x,q),v(x,q)+H(x,q,∇xv(x,q)) = 0 (x,q) ∈C,

v(x,q)−M v(x,q) = 0 (x,q) ∈ A.(359)

We can identify in such system a collection of Hamilton Jacobi related to everydynamics of the system (first line), a rule of selection (second line) giving a choicebetween the various value functions, and a condition of switch (third line) which hasthe objective to preserve the properties of a viscosity solution through the interfaceof switching.Note that uniqueness follows from a strong comparison principle, which also allowsto use the Barles–Souganidis theorem [20] for proving convergence of stable andmonotone schemes.

In order to set up a numerical approximation for (359), we construct a discretegrid of nodes (x j,q) in the state space and fix the discretization parameters ∆x and∆ t. In what follows, we will denote the discretization steps in compact form by∆ = (∆ t,∆x) and the approximate value function by V ∆ .

Following [65], we write the value iteration form of the scheme at (x j,q) as

V ∆ (x j,q) =

MV ∆ (x j,q) if (x j,q) ∈ Amin

NV ∆ (x j,q),Σ(x j,q,V ∆ )

if (x j,q) ∈C

Σ(x j,q,V ∆ ) else.(360)

To define more explicitly the scheme, as well as to extend the approximate valuefunction to all x ∈ Rn and q ∈ I , we use an interpolation I constructed on thenode values, and denote by I

[V ∆](x,q) the interpolated value of V ∆ computed at

(x,q). With this notation, a natural definition of the discrete jump operators M andN is given by

MV ∆ (x,q) := minI [V ∆ ](g(x,q,w))+Ca(x,q,w) | w ∈ V

(361)

NV ∆ (x,q) := minI [V ∆ ](x′,q′)+Cc(x,q,x′,q′) | (x′,q′) ∈ D

. (362)

On the other hand, a standard semi-Lagrangian discretization of the Hamiltonianrelated to continuous control is provided (see [62]) by

Σ

(x j,q,V ∆

)=min

∆ t `(x j,q,u)+ e−λ∆ t I

[V ∆

](x j +∆ t f (x j,q,u),q) | u ∈U

.

(363)It can be proved (see [65]) that, if the interpolation I is monotone (e.g., a P1 orQ1 finite element interpolation), then the scheme is also monotone and thereforeconvergent.

Hybrid vehicle control

In the following example we consider the optimal control problem of driving a Pi-aggio Vespa 50 Special equipped with a three-gears engine, focusing on the accel-eration strategy and the commutation between gears. The problem is solved withthe scheme (361)-(363) and the discrete solution is obtained with a policy iterationapproach (for details [65]).The state equation for the speed of the vehicle is

y(t) = f(y(t),q(t),α(t)

)∀t ∈ (0, tf]

with initial conditions y(0) = x and q(0) = d. For each gear i ∈ 1,2,3 we define

f (y,qi,α) :=1m

(T (βiy)

rρiα− cdy2

)where T (ω) := τ

(ω

ν−(

ω

ν

)3)

is the power band of the engine, τ and ν are respec-

tively its maximum torque and r.p.m., βi := 60rπρi

is a conversion coefficient with

ρi :=transmission shaft r.p.m.

crankshaft r.p.m.

r is the radius of the wheel and cd the drag coefficient.The control α(t) ∈ [0,1] represents the fraction of maximum torque used and the

cost functional is a linear combination of y and α:

`(y,α) = cyy+ cα α

cα e cy are scalar weights. Lastly we set λ = 1 and define ci, j as the switching costfrom from qi to q j.

The following results are obtained by assigning realistic values (Table 14) to theparameters defining in the dynamics.

m ρ1 ρ2 ρ3 r cd τ ν tf140 [kg] 0.06 0.09 0.12 0.2 [m] 0.3 10 [Nm] 6 ·103 [min−1] 10 [s]

Table 14: Choice of parameters, multigear vehicle

Figure 37 shows the power band corresponding to our choice of τ and ν . Costparameters are set to

cα = 1 cy =−0.5 ci, j =

0.1 i 6= j0 i = j

Figure 38 shows the optimal solution obtained with ∆ t = 0.027 [s], ∆y = ∆ t|| f ||∞and (x,d) = (0.28 [m/s],q1). The optimal strategy is to reach the highest gear as fastas possible and then stabilize at the value ≈ 0.5.

Figure 39 shows a different scenario when we set the initial state to (x,d) =(14.58 [m/s],q2): the control lets the vehicle decelerate and then switches t the thirdgear in order to replicate the strategy described previously.

Biographical notes

In this chapter the numerical tests and illustrations are novel, excepted for the Shape-from-Shading problem presented in Section 4.1 (which is similar to the one of [66]).

Fig. 37: Power band of the engine.

6.3. Modello per un motore a tre marce

Con questi parametri la dinamica del sistema per α ≡ 1 e q(t) ≡ d assume

l’andamento riportato in figura 6.4, da cui si ottiene il vincolo y(t) ∈ (0, 52.6].

0 2 4 6 8 10 12 14 16 18 200

5

10

15

20

25

30

35

40

45

50

52.6

t [s]y(t

) [k

m/h

]

d=q1

d=q2

d=q3

Figura 6.4: Andamento della velocita in funzione del tempo per una marcia fissata

La figura 6.5 riporta i risultati elaborati con la scelta di ∆t = 0.027 e ∆y =

∆t||f ||∞ = 0.591 e (x, d) = (1 km/h, q1).

0 5 10 15 20 25 30 35 40 45 50 52.6

!25

!20

!15

!10

!5

0

x [km/h]

v(x

,d)

d=q1

d=q2

d=q3

0 5 100

20

40

52.6

t [s]

yopt(t

) [k

m/h

]

0 5 10

1

2

3

t [s]

q(t

)

0 5 10

0

0.5

1

t [s]

!opt(t

)

Figura 6.5: Funzioni valore, traiettoria e controllo ottimi

Come si puo notare dai grafici il controllo cerca di arrivare il prima possibile

45

Fig. 38: Value functions, trajectory and optimal control.

Capitolo 6. Test numerici

alla marcia piu alta accelerando al massimo, per poi stabilizzarsi intorno al 50% per

mantenere costanti velocita e consumo di carburante.

Cambiando invece lo stato iniziale in (x, d) = (52.5 km/h, q2) la figura 6.6 raffigura

uno scenario differente: il controllo lascia decelerare il veicolo per qualche secondo

per poi ingranare la terza marcia riportandosi ad una situazione simile a quella del

caso precedente.

0 5 10 15 20 25 30 35 40 45 50 52.6

!25

!20

!15

!10

!5

0

x [km/h]

v(x

,d)

d=q1

d=q2

d=q3

0 5 100

20

40

52.6

t [s]

yopt(t

) [k

m/h

]

0 5 10

1

2

3

t [s]

q(t

)

0 5 10

0

0.5

1

t [s]

!opt(t

)

Figura 6.6: Funzioni valore, traiettoria e controllo ottimi

La tabella 6.4 mette a confronto i due schemi nello stesso scenario delle sezioni

precedenti con risultati analoghi, mostrando ancora una volta il vantaggio in termini

di prestazioni della strategia di Policy Iteration.

ε NValue Iteration NPolicy Iteration

10−3 337 6

10−6 586 6

10−12 1084 6

Tabella 6.4: Numero di iterazioni dei due algoritmi in funzione di ε

46

Fig. 39: Value functions, trajectory and optimal control.

5 A Brief Survey on Semi-Lagrangian Schemes for Mean FieldGames

by Roberto Guglielmi1 and Francisco J. Silva2

5.1 Introduction

In one of it simplest form, Mean Field Games (MFGs) can be synthesized in thefollowing system of two Partial Differential Equations (PDEs):

−∂tv(x, t)− 12 σ2∆v(x, t)+H(x,∇v(x, t)) = F(x,m(t)) , Rn× (0,T ),

∂tm(x, t)− 12 σ2∆m(x, t)−div

(∇pH(x,∇v(x, t))m(x, t)

)= 0 , Rn× (0,T ),

v(x,T ) = G(x,m(T )) , for x ∈ Rn ,

m(0) = m0 ∈P1(Rn).

(364)

In the above notation σ ∈ R, H : Rn×Rn→ R is a convex function with respect toits second variable ‘p’, the gradient operator ∇ and the Laplacian ∆ act on the spacevariable x, F , G : Rn×P1(Rn) 7→ R, where P1(Rn) denotes the set of probabilitymeasures m on Rn with first order moments, i.e.

∫Rn |x|dm(x)< ∞.

System (364) has been introduced by J.-M. Lasry and P.-L. Lions in the papers[92, 93, 94] in order to model the asymptotic behavior of symmetric and stochasticdifferential games with a very large number of small players. We refer the reader tothe lectures of Lions in [99], and the surveys [41, 74] for more information on themodeling and the theoretical analysis of (364).

In order to provide a heuristic interpretation of system (364), consider a proba-bility space (Ω ,F ,P), a filtration F := (Ft)t∈[0,T ] and a n-dimensional Brownianmotion W (·) adapted to F. Loosely speaking, if for a given (x, t) ∈ Rn× (0,T ) anda square-integrable control process α(·) adapted to (Ft)t∈[0,T ], we define Xx,t [α](·)as the unique solution of

dX(s) =−α(s)ds+σdW (s), s ∈ (t,T ), X(t) = x, (365)

then, formally, the solution v of Hamilton-Jacobi-Bellman (HJB) equation in (364)can be represented as

v(x, t) = infα

E(∫ T

t

[H∗(Xx,t [α](s),α(s))+F(Xx,t [α](s),m(s))

]ds), (366)

1 Johann Radon Institute for Computational and Applied Mathematics, Linz, Austria.e-mail: roberto.guglielmi@oeaw.ac.at2 Faculte des Sciences et Techniques Universite de Limoges, France.e-mail: francisco.silva@unilim.fr

where H∗ denotes the Fenchel transform of H w.r.t. the second variable. Thus, wecan interpret v(x, t) as the optimal cost of an agent whose initial position is x at timet and faces, at each time s ∈]t,T ], a criterion that depends on the distribution m(s)of all the players at time s. Formally again, the optimal solution of the optimizationproblem in (366) is given in feedback form by α(x, t) = ∇pH(x,∇v(x, t)) and, as aconsequence, the optimal trajectory, denoted as Xx,0, solves

dX(s) =−∇pH(X(s),∇v(X(s), t))ds+σdW (s), s ∈ (t,T ), X(t) = x. (367)

Therefore, if all the players of the initial distribution m0 behave optimally, classicalarguments in diffusion theory imply that the evolution µ(·) of the initial distributionm0 will be characterized by the Fokker-Planck (FP) equation

∂t µ− 12 σ

2∆ µ−div

(∇pH(x,∇v(x, t))µ

)= 0, Rn× (0,T ), µ(0) = m0

and the equilibrium condition reads m (the distribution that each player faces insolving its optimal control problem) must equal µ .

Numerical methods to solve system (364) have been a very active area of researchover the last years. Finite-difference schemes for (364) have been introduced anstudied deeply by Y. Achdou and I. Capuzzo-Dolcetta and several collaborators (see[6, 4, 5, 7, 8]). We refer the reader to [3, Chapter 1] for an interesting survey on thesemethods. Let us point out that in [8] the convergence of the finite-difference schemewhen the interaction is local, i.e. F(x,m(t)) = F(x,m(x, t)) is proved in some caseswhere classical solutions of (364) are not expected to exist. In the same contextof local interactions, system (364) can be interpreted as the optimality system ofan associated variational problem. Numerical methods based on this formulationare presented in [91] and recently in [24]. Finally, when H(x, p) = 1

2 |p|2 a specificscheme has been studied in [78, 77].

In this Section, we present a brief summary of the results provided in [37, 48, 47,49], which consider a different approach in order to derive a numerical scheme for(364). The viewpoint is trajectorial, in the sense that the optimal control problemassociated to the first equation in (364) is discretized first in time and then in space.Similarly, also the second equation in (364) is discretized first in time and then inspace, taking advantage of the well-known result which says that the solution m(t)of the Fokker-Planck equation at time t is given by the probability distribution ofXX0,0(t), when the initial condition X0 in (367) is random, independent of W (·), andhas m0 as a probability distribution. Based on both trajectorial interpretations, wepropose a semi-Lagrangian scheme for the HJB equation (see [34, 62]) coupled witha conservative and also trajectory-based scheme for the Fokker-Planck equation. Inorder to keep the analysis at a reasonable length, we have decided to present inmore detail the work [48] which deals with the case σ = 0, and thus the optimalcontrol problem is deterministic. Even if we do not provide the proofs of the mainresults when the discretization is performed in time and space, we explain in theAppendix the details in the more simpler case when the discretization is performedonly in time. Let us point out that this analysis presents some new improvementswith respect to the study exposed in [37].

5.2 Notations and main asssumptions

In what follows, we suppose that H(x, p) = 12 |p|2 and that the dynamics in (552) has

the more general form

dX(s) =−α(s)ds+σ(t)dW (s), s ∈ (t,T ), X(t) = x, (368)

where σ : [0,T ] 7→Rn×r and 1≤ r ≤ n. Arguing heuristically as in the introduction,the MFG system takes now the form

−∂tv− 12 tr(σ(t)σ>(t)∆v)+ 1

2 |∇v|2 = F(x,m(t)), in Rn× (0,T ),

∂tm− 12 tr(σ(t)σ>(t)∆m)−div

(∇vm

)= 0, in Rn× (0,T ),

v(x,T ) = G(x,m(T )) for x ∈ Rn, m(0) = m0 ∈P1(Rn),

(369)

where tr(A) = ∑ni=1 Aii for all A ∈ Rn×n. The set P1(Rn) is endowed with the

Kantorovich-Rubinstein distance

d1(µ,ν) = sup∫

Rnφ(x)d[µ−ν ](x) ; φ : Rn→ R is 1-Lipschitz

. (370)

In the entire Section c > 0 will denote a generic constant, whose value can changefrom line to line and we suppose that the following assumptions hold true:(H1) F and G are continuous over Rn×P1(Rn).(H2) There exists a constant c0 > 0 such that for any m ∈P1(Rn)

‖F(·,m)‖C2 +‖G(·,m)‖C2 ≤ c0,

where ‖ f (·)‖C2 := supx∈Rn| f (x)|+ |∇ f (x)|+ |∆ f (x)|.(H3) Denoting by σ` : [0,T ]→ Rn (` = 1, . . . ,r) the ` column vector of the matrixσ , we assume that σ` is continuous.(H4) The initial condition m0 ∈P1 is absolutely continuous with respect to theLebesgue measure, with density still denoted by m0, which belongs to L∞(Rn). Wealso suppose that the support of m0, denoted by supp(m0), is compact.

Since the second order operator in (369) is not uniformly elliptic, in general wecannot expect the existence of regular solutions for (369). By the same reason, wecannot expect neither the strict positivity of m(x, t). Since these properties are crucialin the convergence proof for the finite difference schemes proposed in [6, 4, 5],in order to solve (369) we consider another type of scheme, that we call Semi-Lagrangian (SL) scheme.

5.3 A Semi-Lagrangian scheme for the first order MFG system

In this section we suppose that σ ≡ 0 in (369), i.e. we consider the system

−∂tv(x, t)+ 12 |∇v(x, t)|2 = F(x,m(t)) in Rn× (0,T ),

v(x,T ) = G(x,m(T )) in Rn,

∂tm(x, t)−div(∇v(x, t)m(x, t)) = 0 in Rn× (0,T ),

m(0) = m0 in Rn.

(371)

The existence of at least one solution (v,m) ∈W 1,∞loc (Rn× [0,T ])×L1(Rn× [0,T ])

of (371), where the first equation is satisfied in the viscosity sense and the secondone is satisfied in the distributional sense, has been proved in [94] (see also [41]).

In order to introduce the SL discretization, we consider both equations in (371)separately. Given µ ∈C([0,T ],P1(Rn)) we know (see e.g. [18]) that there exists aunique viscosity solution v[µ] of the Hamilton-Jacobi-Bellman (HJB)

−∂tv(x, t)+ 12 |∇v(x, t)|2 = F(x,µ(t)), in Rn× (0,T ),

v(x,T ) = G(x,µ(T )) in Rn.(372)

Moreover, given (x, t) ∈ Rn× (0,T ), α ∈ L2([0,T ];Rn) and defining

Xx,t [α](·) := x−∫ ·

tα(s)ds,

we have that v[µ](t,x) can be represented as the optimal cost of the following opti-mal control problem (see e.g. [18])

v[µ](x, t) = infα

∫ T

t

[ 12 |α(s)|2 +F(Xx,t [α](s),µ(s))

]ds+G(Xx,t [α](T ),µ(T )). (373)

Therefore, by the Dynamic Programming Principle (DPP) (see e.g. [18]), for all0≤ h < T − t we have that

v[µ](x, t)= infα

v[µ](Xx,t [α](t +h), t +h)+

∫ t+h

t

[ 12 |α(s)|2 +F(Xx,t [α](s),µ(s))

]ds, (374)

and v[µ](·,T ) = G(·,µ(T )). Now, given a time step h > 0 and N ∈ N such thatNh = T , a natural approximation of v[µ] is obtained by discretizing in time the DPP.For k = 0, . . . ,N and x ∈ Rn let us define recursively

vk[µ](x) = infα∈Rn

vk+1[µ](x−hα)+ 1

2 h|α|2+hF(x,µ(tk)),

vN [µ](x) = G(x,µ(tN)).(375)

A rigorous study of the semi-discrete approximation vk[µ] of v[µ] has been carriedout in [56, 57] in a much broader framework. In particular, it is shown that extendingvk[µ](x, t) for the intermediate times t ∈ (tk, tk+1) as

vh[µ](x, t) = vk[µ](x) if t ∈ [tk, tk+1), (376)

we have that vh[µ]→ v[µ] uniformly on compact sets as h ↓ 0.The fully discrete approximation of (372) is induced by the semi-discrete scheme

(375) and linear interpolation in the space variable. More precisely, given ρ > 0,let us define Gρ := xi = ρi ; i ∈ Zn and consider a regular triangulation Tρ ofRn with vertices belonging to Gρ . In this triangulation, we define a P1-basis, i.e. acollection of continuous functions βii∈Zn , affine on each element of Tρ , satisfyingthat βi(x j) = δi j for all x j ∈ Gρ (the Kronecker symbol). Moreover, βi results to havecompact support, 0 ≤ βi ≤ 1, and ∑i∈Zn βi(x) = 1 for all x ∈ Rn. Given a functionf : Gρ 7→ R we define its interpolate

I[ f ](·) := ∑i∈Zn

fiβi(·) for f ∈ B(Gρ). (377)

By discretizing the unknown space-dependent functions using the interpolation op-erator I, we get the following fully-discrete schemevi,k[µ] = infα∈Rn

I[v·,k+1[µ]](xi−hα)+ 1

2 h|α|2+hF(xi,µ(tk)),

vi,N [µ] = G(xi,µ(tN)).(378)

The scheme is extended as a function defined in Rn× [0,T ] as

vρ,h[µ](x, t) := I[v·,k](x) if t ∈ [tk, tk+1). (379)

As long as (ρ,h)→ (0,0) with ρ2/h→ 0, the convergence of vρ,h[µ] to v[µ] isproved in [61] in a more general case involving more accurate discretizations (seealso [62] for a nice overview on the subject).

Now, let us discuss the discretization of the Fokker-Planck equation, which in thisdeterministic framework (σ = 0) is often called the continuity or Liouville equation.In order to motivate the scheme, let us consider the equation

∂tm(x, t)+div(b(x, t)m(x, t)

)= 0, in Rn× (0,T ),

m(x,0) = m0(x) in Rn.(380)

where b : Rn× (0,T )→ Rn is bounded and Lipschitz with respect to x, uniformlyin t ∈ (0,T ). It is well known (see e.g. [41]), that (380) admits a unique solution,m∈C([0,T ],P1(Rn)). Moreover, defining the flow Φ(t1, t2, ·), with 0≤ t1≤ t2≤ T ,as Φ(t1, t2,x) = X(t2) where X solves

X(t) = b(X(t), t) for t ∈ (t1,T ), X(t1) = x,

we have that m(t) = Φ(0, t, ·)]m0, which means that m(t)(A) = m0(Φ(0, t, ·)−1(A))for all A ∈B(Rn), or in other words,∫

Rnφ(x)dm(t)(x) =

∫Rn

φ(Φ(0, t,x))dm0(x) ∀ φ ∈Cb(Rn),

where Cb(Rn) denotes the set of continuous and bounded real-valued functions de-fined on Rn.

The above result motivates the following time-discretization (recall that h >0 is the time step and N ∈ N is such that Nh = T ): define the discrete flowΦh(tk, tk+1,x) := x+hb(x, tk) and the sequence of measures

mk+1 := Φh(tk, tk+1, ·)]mk k = 0, . . . ,N−1, (381)

where we recall that m0 is given. The sequence of measures mk are extended to anelement of C([0,T ];P1(Rn)) as follows

mh(t) :=tk+1− t

hmk +

t− tkh

mk+1 if t ∈ [tk, tk+1]. (382)

In order to discretize the measures in the space, we use the grid Gρ , the P1-basisβii∈Zn and for every i ∈ Zn and k = 0, . . . ,N−1 we set

Ei := [xi± 12 ρe1]× ...× [xi± 1

2 ρen], Φi,k := Φh(tk, tk+1,xi) = xi+hb(xi, tk), (383)

where e1 . . .en is the canonical basis in Rn. We consider the following fully dis-cretization of (380)

mi,k+1 = ∑j∈Zn

βi(Φ j,k

)m j,k, mi,0 =

∫Ei

m0(x)dx. (384)

Note that this scheme is conservative

∑i∈Zn

mi,k = ∑j∈Zn

m j,k ∑i∈Zn

βi(Φ j,k

)= ∑

i∈Znmi,k = ∑

i∈Znmi,0 = 1.

For some technical reasons, the scheme is modified by extending its domain to Rn×(0,T ) as follows: if t ∈ [tk, tk+1) we set

mρ,h(x, t) :=1

ρn

[tk+1− t

h ∑i∈Zn

mi,kIEi(x)+t− tk

h ∑i∈Zn

mi,k+1IEi(x)

]. (385)

In this manner, mρ,h can be identified to an element of C([0,T ];P1(Rn)), whichfor every time t has an essentially bounded density mρ,h(·, t). Moreover, mρ,h ∈L∞(Rn× (0,T )). The scheme (384) can be formally derived using the weak-formu-lation of (380) and its extended version (385) can be proved to be weakly convergent([46]).

We will apply the previous scheme to discretize the continuity equation in (371).Given µ ∈C([0,T ];P1(Rn)), recall that v[µ] denotes the unique viscosity solutionof (372). Using the representation formula (374) and our assumptions on F and G,we have that v[µ] is Lipschitz and thus, for all t ∈ (0,T ) we have that ∇v[µ](x, t)exists for a.a. x∈Rn. Moreover, it can be proved that v[µ] is uniformly semiconcavewith respect to the space variable x (see e.g. [39]), i.e. there exists c > 0 such that

v[µ](x+h, t)−2v[µ](x, t)+ v[µ](x−h, t)≤ c|h|2 ∀ h ∈ Rn. (386)

(we say v[µ](·) is semiconcave with constant c and it is uniformly semiconcavesince c is independent of (µ, t)). Now, let us consider the equation

∂tm(x, t)−div(∇v[µ](x, t)m(x, t)

)= 0 in Rn× (0,T ),

m(x,0) = m0(x) Rn.(387)

Note that in this case the velocity field −∇v[µ] is not regular and so we cannotapply directly the existence and uniqueness result mentioned above. However, it ispossible to prove that for any 0≤ t1 ≤ t2 ≤ T and a.a. x ∈ Rn, the equation

X(t) =−∇v[µ](X(t), t) for t ∈ (t1,T ), X(t1) = x, (388)

admits a unique solution. The key argument in the the proof is the fact that the ve-locity field is minus the derivative w.r.t. to the initial condition of the value functionwhich has a unique optimizer for a.e. initial condition x ∈ Rn (see [39] and [41]for more on this). Using this fact and defining the flow Φ [µ](t1, t2,x) as the uniquesolution of (388) for a.e. x ∈Rn it is easy to show that m[µ](t) := Φ [µ](0, t, ·)]m0 isa distributional solution of (387). The tricky fact is the uniqueness of the solution of(387), which can be proved using the semiconcavity property (386) and a superpo-sition principle proposed in e.g. [11] (see [41] for a self-contained proof using thisargument).

Therefore, it is natural to propose (381) as a discretization in time of (387) withb(x, tk) replaced by−∇vk[µ](x) (see (375)). On the other hand, there are some issuesin defining the corresponding fully-discrete scheme (384) since the corresponding“velocity field” −∇vρ,h[µ](x, t) does not exists if x ∈ Gρ . Therefore, we have toregularize vh,ρ [µ] with a mollifier. More precisely, given a regular function ρ ∈C∞

c (Rn) satisfying that ρ ≥ 0 and∫Rn ρ(x)dx = 1, for ε > 0 consider the mollifier

ρε(x) := 1εn ρ

( xε

)and set

vερ,h[µ](·, t) := ρε ∗ vρ,h[µ](·, t) for all t ∈ [0,T ]. (389)

Thus, we consider the scheme (384) with b(xi, tk) =−∇vερ,h[µ](xi, tk) and we denote

Φεi,k[µ] := xi−h∇vε

ρ,h[µ](xi, tk). This gives the schememε

i,k+1[µ] = ∑ j∈Zn βi

(Φε

j,k[µ])

mεj,k[µ],

mεi,0[µ] =

∫Ei

m0(x)dx,(390)

We extend the previous scheme to a function mερ,h[µ] ∈ L∞(Rn× (0,T )) defined as

mερ,h[µ](x, t) :=

(tk+1− t

h

)mε [µ](tk,x)+

(t− tk

h

)mε [µ](tk+1,x) if t ∈ [tk, tk+1], (391)

where

mε [µ](tk,x) :=1

ρn ∑i∈Zn

mεi,kIEi(x). (392)

Let us summarize the problem that we have considered so far.(i) The continuous case: To solve (371) is equivalent to solve the following fixedpoint problem

Find µ ∈C([0,T ];P1(Rn)) such that µ = m[µ]. (393)

(ii) The semi-discrete in time problem: Given the semi-discrete scheme (375) for theHJB equation (372) and the semi-discrete scheme (381) (with b =−∇vh[µ]) for thecontinuity equation (387), we consider the following semi-discrete in time schemefor (393):

Find µ ∈C([0,T ];P1(Rn)) such that µ = mh[µ]. (394)

(ii) The fully-discrete problem: Given the fully SL discretization (378) for the HJBequation (372) and the fully-discrete scheme (390) for (387), we consider the fol-lowing fully-discrete scheme for (393):

Find µ ∈C([0,T ];P1(Rn)) such that µ = mερ,h[µ]. (395)

The existence of at least one fixed point in problems (393), (394) and (395) followsfrom standard arguments, taking advantage of the fact that the mappings µ 7→m[µ],µ 7→mh[µ] and µ 7→mε

ρ,h[µ] are continuous and take values in suitable pre-compactsets. The delicate issue is the proof of the convergence of solutions of (394) and(395) to solutions of (393). In the appendix we provide a more or less completeanalysis of problem (394) and we provide a convergence result. The correspondingproof of the convergence for solutions of (395) is much more elaborate and it isavailable only in the scalar case n = 1. The crucial point is that only when n = 1 weare able to prove uniform L∞-bounds for mε

ρ,h[µ] and these bounds are fundamentalin order to justify the convergence. We only state the result referring the reader to[48] for a complete proof.

Theorem 10. Suppose that n = 1 and that ρn = o(hn), hn = o(εn) as εn→ 0.Then any limit point in C([0,T ];P1) of discrete solutions mεn

ρn,hnof (395) solves

(393). In particular, if the continuous problem has a unique solution (v,m), thenmεn

ρn,hn→ m in C([0,T ];P1) and also in L∞ (Rn× [0,T ])-weak-∗.

5.4 The SL scheme in the second order case

In this section we consider an extension of the previous scheme to the second orderand possibly degenerate case (368). Because of space constraints, we limit ourselvesto the presentation of the scheme and the statement of the convergence result. Amuch more detailed presentation, as well as the proof of the convergence of thefully discrete-scheme, can be found in [49].

As presented before, we consider the discretization of both equations in (369)separately. The main difference between the first and second order case in the trajec-torial approach devised to construct our scheme, is that the representation formulasfor the solutions of the second order HJB and FP equations depend on some diffu-sion processes instead than on deterministic trajectories. Indeed, since the drivingequation is given by (368), a natural way to discretize the stochastic optimal con-trol problem is to approximate the underlying Brownian motion by a convenientlyscaled random walk in Rn. Thus, if for a given µ ∈C([0,T ];P1(Rn)) we considerthe HJB equation

−∂tv− 12 tr(σ(t)σ>(t)∆v)+ 1

2 |∇v|2 = F(x,m(t)), in Rn× (0,T ),

v(x,T ) = G(x,µ(T )) for x ∈ Rn,(396)

and we denote by v[µ] its unique viscosity solution, for given h > 0 and N such thatNh = T , the previous discussion yields the semidiscrete approximation

vk[µ](x) = infα∈Rn

[12r ∑

r`=1

(vk+1[µ](xi−hα +

√hrσ`(tk))+ vk+1[µ](xi−hα−

√hrσ`(tk))

)+ 1

2 h|α|2 +hF(xi,µ(tk))],

vN [µ](x) = G(xi,µ(tN)).(397)

Using the interpolation operator defined in (377), we obtain the following fully-discrete scheme for (396), first proposed in [34] and thoroughly studied in [54],

vi,k = Sρ,h[µ](v·,k+1, i,k) for all i ∈ Gρ , k = 0, . . . ,N−1,vi,N = G(xi,µ(tN)), for all i ∈ Gρ ,

(398)

where Sρ,h[µ] : B(Gρ)×Zn×0, . . . ,N−1→ R is defined as

Sρ,h[µ]( f , i,k) := infα∈Rn

[12r

r

∑`=1

(I[ f ](xi−hα +

√hrσ`(tk))+ I[ f ](xi−hα−

√hrσ`(tk))

)+ 1

2 h|α|2 +hF(xi,µ(tk))

].

(399)

The solution of the scheme (398) is extended to a function vρ,h : Rn× [0,T ]→R asin (379).

Given µ ∈C([0,T ];P1(Rn)), in order to discretize the equation

∂tm− 12 tr(σ(t)σ>(t)∆m)−div

(∇v[µ]m

)= 0, in Rn× (0,T ),

m(0) = m0,(400)

the idea is the same as the one in the deterministic case, the difference being that wehave to take into account all the possible values of the random walk discretizing theBrowian motion W (t). For ε > 0, i ∈ Zn, `= 1, . . . ,r and k = 0, . . . ,N−1 let us set

Φε,`,+i,k [µ] := xi−h∇vε

ρ,h[µ](xi, tk)+√

rhσ`(tk),

Φε,`,−i,k [µ] := xi−h∇vε

ρ,h[µ](xi, tk)−√

rhσ`(tk),(401)

where vερ,h[µ] is a regularization of vρ,h[µ] (see (389)). Now, let us define mε

i,k[µ] ; i∈Zn, k = 0, . . . ,N recursively as

mεi,k+1[µ] :=

12r ∑

j∈Zn

r

∑`=1

[βi

(Φ

ε,`,+j,k [µ]

)+βi

(Φ

ε,`,−j,k [µ]

)]mε

j,k[µ],

mεi,0[µ] :=

∫Ei

m0(x)dx ,

(402)

where we recall that Ei is defined in (383). The scheme is extended to a functionmε

ρ,h[µ] ∈ L∞(Rn× (0,T )) as in (391).The fully discretization of (369) proposed in [49] reads

Find µ ∈C([0,T ];P1) such that mερ,h[µ] = µ . (403)

Using Brower fixed point theorem, it is proved that (403) admits at least one solu-tion. The main result in [49] is the convergence of solutions of (403) to a solutionof (369), where, for the same reasons than those in the first order case, the proof isvalid only in the scalar case n = 1.

Theorem 11. Suppose that n = 1 and that ρn = o(hn), hn = o(ε2n ), as εn→ 0.

Let mnn∈N be a sequence of solutions of (403) for the corresponding parametersρn,hn,εn and vn := vρn,hn [m

n]. Then, any limit point m in C([0,T ];P1(Rn)) of dis-crete solutions mn (there exists at least one) satisfies that (v[m], m) solves (369) andvn→ v[m] uniformly in compact subsets of Rn× (0,T ). In particular, if the contin-uous problem has a unique solution (v,m), then mn→ m in C([0,T ];P1(Rn)) andalso in L∞ (Rn× [0,T ])-weak-∗.

5.5 Numerical tests

We assume that the final cost G(x,m) is zero and that F(x,m) is given by

F(x,m) = 5(x− (1− sin(2πt))/2)2 +Vδ (x,m),

whereVδ (x,m) = φδ ∗ [φδ ∗m] (x) and φδ (x) =

1√2π

e−x2/(2δ 2).

The interpretation is that agents wants to be close to the curve t ∈ [0,T ] 7→ (1−sin(2πt))/2 ∈R, but at the same time they pay a cost Vδ (x,m) related to congestionat point x when the global distribution is m, i.e. the agents try to avoid places with ahigh concentration of the population. We consider a numerical domain Qb× [0,T ] =[0,1]× [0,2] and we choose as initial mass distribution:

m0(x) =ν(x)∫

Ων(x)dx

with ν(x) = e−(x−0.5)2/(0.1)2.

In figure 40 we show the result of an heuristic numerical resolution of the fully-discrete SL scheme when the game is deterministic, i.e. σ = 0. In figure 41 wesimulate the second order non-degenerate case with σ = 0.2. The regularizing ef-fect of the diffusion can be clearly observed. Finally, in figure 42 we consider an“intermediate” case when σ(t) := max(0,0.2−|t−1|), i.e. a degenerate diffusion.

Fig. 40: Test 1: Mass evolution mεi,k: the parameters of the method are ρ = 3.12 ·

10−3, h = ρ and ε = 0.15.

Fig. 41: Test 2: Mass evolution mεi,k: the parameters of the method are ρ = 6.35 ·

10−3, h = ρ and ε = 2√

h, τ = 10−3.

Fig. 42: Test 3: Mass evolution mεi,k: the parameters of the method are ρ = 6.35 ·

10−3, h = ρ and ε = 2√

h, τ = 10−3.

Appendix: Semi-discrete in time approximation revisited

In this section we review the semi-discrete in time approximation studied in [37],summarized in Section 5.3, and we improve some of the results therein providingalso a complete semi-discrete analysis of analogous results in [41, Section 4]. Firstlet us recall that for w : Rn→ R the super-differential D+w(x) at x ∈ Rn is definedas

D+w(x) :=

p ∈ Rn ; limsupy→x

w(y)−w(x)−〈p,y− x〉|y− x| ≤ 0

. (404)

We collect in the following Lemmas some useful properties of semiconcave func-tions, i.e. functions that satisfy (386) (see [39] for a very complete account of thissubject).

Lemma 1. For a function w : Rn→ R, the following assertions are equivalent:

(i) The function w is semiconcave, with constant c.

(ii) For all x, y ∈ Rn and p ∈ D+w(x), q ∈ D+w(y)

〈q− p,y− x〉 ≤ c|x− y|2. (405)

(iii) Setting In for the identity matrix, we have that ∆w ≤ cIn in the sense of distri-butions.

Lemma 2. Let w : Rn→ R be semiconcave. Then:

(i) w is locally Lipschitz.

(ii) If wn is a sequence of uniformly semiconcave functions (i.e. they share the samesemiconcavity constant) converging pointwise to w, then the convergence is locallyuniform and ∇wn(·)→ ∇w(·) a.e. in Rn.

5.5.1 Properties of the semi-discretization of the HJB equation

Recall that given h > 0 and N ∈N such that Nh = T , we set tk := kh for k = 0, . . . ,N.Let us define the following spaces:

KN :=

µ = (µ`)N`=0 : such that µ` ∈P1(Rn) for all `= 0, . . . ,N

,

Ak :=

α = (α`)N−1`=k : such that α` ∈ Rn

for k = 0, . . . ,N−1.

For µ ∈KN and k = 1, . . . ,N, we consider the following semi-discrete approxima-tion of v[µ] in (373)

vk[µ](x) := infα∈Ak

N−1

∑`=k

[ 12 |α`|2 +F(Xx,k

` [α],µ`)]h+G(Xx,k

N [α],µN)

,

where Xx,k`+1[α] := Xx,k

` [α]−hα` for `= k . . . ,N−1,

Xx,kk [α] := x

(C P)x,k

h [µ].

Classical arguments imply that (C P)x,kh [µ] admits at least a solution for all

(x,k). We denote by Ak[µ](x) ⊆ Ak the set of optimal solutions of (C P)x,kh [µ],

i.e. the set of discrete optimal controls. Note that vk[µ](x) can be equivalently de-fined with the discretized DPP (375). Recall also the extension vh[µ], defined inRn× [0,T ], of vk[µ](x) considered in (376). We have the following properties forvh[µ]:

Proposition 7. For all h > 0, we have:(i) For any t ∈ [0,T ], the function vh[µ](·, t) is Lipschitz continuous, with a Lipschitzconstant c > 0 independent of (µ,h,k).(ii) For all t ∈ [0,T ] the function vh[µ](·, t) is semiconcave uniformly in (h,µ, t).(iii) There exists a constant c > 0 (independent of (µ,h,x,k)) such that

max`=k,...,N−1

|α`| ≤ c for all α ∈Ak[µ](x).

(iv) For all x ∈ Rn, k = 0, . . . ,N−1 and α ∈Ak[µ](x), we have

α`+h∇F(

Xx,k` [α],µ`

)∈ D+vh[µ]

(Xx,k` [α], t`

)for `= k, . . . ,N−1.

(v) We have that vh[µ](·, t) is differentiable at x iff for k = [t/h] there exists α ∈Ak[µ](x) such that Ak[µ](x) = α. In that case, the following holds:

∇vh[µ](x, t) = αk +h∇F(x,µk).

(vi) Given (x, t) and α ∈Ak[µ](x), with k = [t/h], we have that for all s ∈ [tk+1,T ],the function vh[µ](·,s) is differentiable at Xx,k

` [α], with `= [s/h].

Proof. We only prove (iv) since the other statements are proved in [37]. For nota-tional convenience, we omit the µ argument and we prove the result for `= k, since

for ` = k + 1, . . . ,N the assertion follows from (v)-(vi). Let x,y ∈ Rn and τ ≥ 0.Since α ∈Ak[µ](x), we have

vk(x+ τy)≤N−1

∑`=k

[12 |α`|2 +F

(Xx+τy,k` [α],µ`

)]h+G

(Xx+τy,k

N [α],µN

),

with equality for τ = 0. Therefore,

vk(x+ τy)− vk(x) ≤ hN−1

∑`=k

[F(

Xx+τy,k` [α],µ`

)−F

(Xx,k` [α],µ`

)]+G

(Xx+τy,k

N [α],µN

)−G

(Xx,k` [α],µN

).

(406)

On the other hand, the optimality condition for α yields

αk = hN−1

∑`=k+1

∇F(

Xx,k` [α],µ`

)+∇G

(Xx,k` [α],µN

).

Combining with (406) and taking the limit as τ → 0, gives

limsupτ→0

vk(x+ τy)− vk(x)τ

−〈αk +h∇F(x,µk),y〉 ≤ 0,

which, by [39, Proposition 3.15 and Theorem 3.2.1], implies the result.

Given (x,k) and α ∈Ak[µ](x) we set

αk[µ](x) := αk. (407)

Proposition 7(iv) implies that

αk[µ](x) ∈ D+vh[µ](x, tk)−h∇F(x,uk). (408)

A straightforward computation shows that αk[µ](x) solves, for each (x,k), the prob-lem defined in the r.h.s. of (375). Moreover, by Proposition 7(v)-(vi), the followingrelation holds true

α` = α`[µ](

Xx,k` [α]

)for all `= k, . . . ,N−1. (409)

5.5.2 Semi-discretization of the continuity equation

Let αx,k[µ] ∈Ak be a measurable selection of the multifunction (x,k)→Ak[µ](x).Given this measurable selection, we set αk[µ](x) = α

x,kk [µ], as in (407). By (408) -

(409), there exists a measurable function (x,k)→ pk[µ](x)∈Rn such that pk[µ](x)∈D+vk[µ](x) and for all time iterations `= k, . . . ,N we have

α`[µ](

Xx,k` [αx,k[µ]]

)= p`[µ]

(Xx,k` [αx,k[µ]]

)−h∇F

(Xx,k` [αx,k[µ]],µ`

). (410)

Moreover, Proposition 7(v)-(vi) implies that for `= k+1, . . . ,N

p`[µ](

Xx,k` [αx,k

` [µ])= ∇v`[µ]

(Xx,k` [αx,k

` [µ]])

for all x ∈ Rn (411)

andpk[µ] (x) = ∇vk[µ] (x) for a.a. x ∈ Rn. (412)

Given (x,k1), the discrete flow Φk1,·[µ](x) ∈ R(N−k)×n is defined as

Φk1,k2 [µ](x) := x−hk2−1

∑`=k1

αx,k1` [µ] for all k2 ≥ k1. (413)

Equivalently, by (410), for all k1 ≤ k2 ≤ k3,

Φk1,k3 [µ](x) := x−h∑k3−1`=k1

α`[µ](

Xx,k1` [αx,k1 [µ]]

),

= Φk1,k2 [µ](x)−h∑k3−1`=k2

α`[µ](

Xx,k1` [αx,k1 [µ]]

).

(414)

In particular, for all k1 ≤ k2,

Φk1,k2+1[µ](x) = Φk1,k2 [µ](x)−hαk2 [µ](Φk1,k2 [µ](x)

). (415)

The following result is an important improvement of [37, Lemma 3.6].

Proposition 8. There exists a constant c > 0 (independent of µ and small enoughh) such that for all k = 1, ...,N and x,y ∈ Rn we have

|Φ0,k[µ](x)−Φ0,k[µ](y)| ≥ c|x− y|. (416)

Thus, Φ0,k[µ](·) is invertible in Φ0,k[µ](Rn) and the inverse ϒ0,k[µ](·) is 1/c-Lipschitz.

Proof. For notational convenience, let us set Φk = Φ0,k[µ](x) and Ψk = Φ0,k[µ](y).Expression (415) implies that

|Φk+1−Ψk+1|2 ≥ |Φk−Ψk|2−2h [αk[µ](Φk)−αk[µ](Ψk)] · (Φk−Ψk). (417)

By (410) we have (omitting the dependence on µ)

αk(Φk)−αk(Ψk) = pk(Φk)− pk(Ψk)−h [∇F(Φk)−∇F(Ψk)] .

Using the semiconcavity of vk[µ](·) and the fact that F has bounded second orderderivatives w.r.t. x, Lemma 1(iii) gives

[αk(Φk)−αk(Ψk)] · (Φk−Ψk)≤ c(1+h) |Φk−Ψk|2 , (418)

for some c > 0. By (417) and (418), there is c′ > 0 (independent of h small enough)such that

|Φk+1−Ψk+1|2 ≥ (1−hc′) |Φk−Ψk|2 .Therefore, for every k = 1, ...,N, we get

|Φk+1−Ψk+1|2 ≥ (1−hc′)k |x− y|2 ≥ (1−hc′)[T/h] |x− y|2 .

and the result follows from the convergence of (1−hc′)[T/h] to exp(−c′T ) as h ↓ 0.

As we already explained in Section 5.3, a natural semi-discretization of the so-lution m[µ] of (387) is obtained as the push-forward of m0 under the discrete flowΦ0,k[µ](·). For every k = 0, . . . ,N set

mk[µ] := Φ0,k[µ](·)]m0. (419)

By (414) we have

mk[µ] = Φ`,k[µ](·)]m`[µ] for all `= 1, . . . ,k, (420)

In particular, for all φ ∈Cb(Rn) we have∫Rn

φ(x)dmk+1[µ](x) =∫Rn

φ (x−hαk[µ](x))dmk[µ](x), (421)

which applied with φ ≡ 1 gives mk[µ](Rn) = 1 for k = 0, . . . ,N.We have the following Lemma, which improves [37, Lemma 3.7] since we now

prove, using Proposition 8, uniform bounds for the density of mk[µ]. Recall thedistance d1(µ,ν) between to probability measures with finite first order moments isdefined in (370).

Lemma 3. There exists c > 0 (independent of (µ,h)) such that:

(i) For all k1,k2 ∈ 1, ...,N, we have that

d1(mk1 [µ],mk2 [µ])≤ ch|k1− k2|= c|tk1 − tk2 |. (422)

(ii) For all k = 1, ...,N, mk[µ] is absolutely continuous (with density still denoted bymk[µ]), has a support in B(0,c) and ‖mk[µ]‖∞ ≤ c.

Proof. By Proposition 7(iii) we have

|Φ0,k1 [µ](x)−Φ0,k2 [µ](x)| ≤ ch|k1− k2|= c|tk1 − tk2 |. (423)

By definition of mk[µ](·), we have that for any 1-Lipschitz function φ : Rn→ R∫Rn φ(x)d

[mk1 [µ]−mk2 [µ]

](x) ≤ ∫Rn |Φ0,k1 [µ](x)−Φ0,k2 [µ](x)|dm0(x)

≤ ch|k1− k2|= c|tk1 − tk2 |.

On the other hand, since by (H1) we have supp(m0) ⊂ B(0,c1), Proposition 7(iii)implies that supp(mk[µ]) is contained in B(0,c′) for some c′ > 0. Moreover, for anyBorel set A and k = 1, ...,N, Proposition 8 and the fact that ‖m0‖∞ ≤ c imply theexistence of c′′ > 0 such that

mk[µ](A) = m0(ϒ0,k[µ](A))≤ ‖m0‖∞|ϒ0,k(A)| ≤ c′′|A|,

where |A| denotes the Lebesgue measure of the set A. Thus, mk[µ] is absolutelycontinuous and its density, still denoted by mk[µ], satisfies ‖mk[µ]‖∞ ≤ c′′. Theresult easily follows.

Recall that mk[µ] is extended to an element mh[µ](·) of C([0,T ];P1(Rn)) as in(382). The following result is a clear consequence of Lemma 3 and (382).

Proposition 9. There exists a constant c > 0 (independent of µ and small enoughh) such that:

(i) For all t1, t2 ∈ [0,T ], we have that

d1(mh[µ](t1),mh[µ](t2))≤ c|t1− t2|. (424)

(ii) For all t ∈ [0,T ], mh[µ](t) is absolutely continuous (with density denoted bymh[µ](·, t)), has a support in B(0,c) and ‖mh[µ](·, t)‖∞ ≤ c.

5.5.3 The semi-discrete scheme for the first order MFG problem

Recall that the semidiscretization of the MFG problem (371) can be written as

Find µ ∈C([0,T ];P1(Rn)) such that µ = mh[µ]. (425)

The following result is proved in [37].

Theorem 12. Under our assumptions we have that (425) admits at least one solu-tion mh ∈KN . Moreover, if the following monotonicity conditions hold true∫

Rn [F(x,m1)−F(x,m2)]d[m1−m2](x)> 0 for all m1 6= m2 ∈P1∫Rn [G(x,m1)−G(x,m2)]d[m1−m2](x)> 0 for all m1,m2 ∈P1, m1 6= m2.

(426)

then the solution is unique.

Remark 5. The monotonicity assumption (426) has been proposed in [93] and is alsoa sufficient condition to guarantee the uniqueness of a solution of (371).

Now, let us prove the convergence of solutions mh of (425).

Theorem 13. Under our assumptions, as h ↓ 0 every limit point of mh in C([0,T ];P1)(there exists at least one) solves (425). In particular, if (426) holds true, we havethat mh→m (the unique solution of (425)) in C([0,T ];P1) and in L∞ (Rn× [0,T ])-weak-∗.

Proof. Proposition 9 and Ascoli Theorem imply that mh has at least one limit pointm in C([0,T ];P1) as h ↓ 0. Now, setting vh = vh[mh] classical arguments (see e.g.[56, 57]) imply that vh→ v[m] uniformly on compact sets of Rn×(0,T ), where v[m]is the unique viscosity solution of (372) (with µ = m).

In order to conclude the proof we have to show that m = m[m] = Φ [m](0, t, ·)]m0(recall the notations introduced in subsection 5.3). By [41, Section 4] there exists aset A⊆ Rn, with L n(Rn \A) = 0, such that

A (x,0) := the set of optimal controls for v[m](x,0),

is a singleton α(x,0) for all x ∈ A. Proposition 7(iii) implies that the opti-mal controls αh(x, ·), associated with vh(x,0), are bounded in L∞([0,T ];Rn) (inparticular they are bounded in L2([0,T ];Rn)). Thus, up to subsequence, αh(x, ·)converges weakly in L2([0,T );Rn) to some α(x, ·) and thus the flow Φh(x, ·) :=x− ∫ ·0 αh(x,s)ds converge uniformly to some Φ(x, ·) := x− ∫ ·0 α(x,s)ds. Now, sincevh(x,0)→ v[m](x,0), by the uniqueness of the optimal control for x ∈ A, we obtainthat α = α(x,0). Using that m0 has compact support, by the dominated convergencetheorem, the convergence for x ∈ A of Φh(x, ·) to Φ(x, ·), which is the optimal tra-jectory for v[m](x,0), the optimal yields that for every 1-Lipschitz ϕ : Rn→ R andt ∈ [0,T ],∫

Rnϕ(x)d[mh(t)−m[m](t)](x)≤

∫Rn|Φh(x, t)− Φ(x, t)|m0(x)dx→ 0 as h ↓ 0.

Therefore m = m[m], which ends the proof. Finally, by Proposition 9(ii) we havethat mh→ m in L∞ (Rn× [0,T ])-weak-∗. The result follows.

References

1. R. Abgrall. Numerical discretization of the first-order hamilton-jacobi equation on triangularmeshes. Comm. Pure Appl. Math., 49:1339–1373, 1998.

2. R. Abgrall. Construction of simple, stable and convergent high order scheme for steady firstorder Hamilton-Jacobi equation. SIAM J. Sci. Comput., 31:2419–2446, 2009.

3. Y. Achdou, G. Barles, H. Ishii, and G.-L. Litvinov. Hamilton-Jacobi equations: approxima-tions, numerical analysis and applications, volume 2074 of Lecture Notes in Mathematics.Springer, Heidelberg; Fondazione C.I.M.E., Florence, 2013. Lecture Notes from the CIMESummer School held in Cetraro, August 29–September 3, 2011, Edited by Paola Loreti andNicoletta Anna Tchou, Fondazione CIME/CIME Foundation Subseries.

4. Y. Achdou, F. Camilli, and I. Capuzzo Dolcetta. Mean field games: Numerical methods forthe planning problem. SIAM J. Control Optim., 50:79–109, 2012.

5. Y. Achdou, F. Camilli, and I. Capuzzo Dolcetta. Mean field games: convergence of a finitedifference method. SIAM J. Numer. Anal., 51(5):2585–2612, 2013.

6. Y. Achdou and I. Capuzzo Dolcetta. Mean field games: Numerical methods. SIAM J. Numer.Anal., 48-3:1136–1162, 2010.

7. Y. Achdou and V. Perez. Iterative strategies for solving linearized discrete mean field gamessystems. Netw. Heterog. Media, 7(2):197–217, 2012.

8. Y. Achdou and A. Porretta. Convergence of a finite difference scheme to weak solutions ofthe system of partial differential equation arising in mean field games. SIAM J. Numer. Anal.,54(1):161–186, 2016.

9. T. Allsopp, A. Mason, and A. Philpott. Optimising yacht routes under uncertainty. In Pro-ceedings of the 2000 Fall National Conference of the Operations Reaseach Society of Japan,volume 176, page 183, 2000.

10. A. Altarovici, O. Bokanowski, and H. Zidani. A general hamilton-jacobi framework for non-linear state-constrained control problems. ESAIM Contr. Optim. Ca., 19(2):337–357, 2012.

11. L. Ambrosio, N. Gigli, and G. Savare. Gradient flows in metric spaces and in the space ofprobability measures. Second edition. Lecture notes in Mathematics ETH Zurich. BirkhauserVerlag, Bassel, 2008.

12. G. Aubert and P. Kornprobst. Mathematical problems in image processing, volume 147 of Ap-plied Mathematical Sciences. Springer, New York, second edition, 2006. Partial differentialequations and the calculus of variations.

13. J.-P. Aubin. Viability solutions to structured hamilton-jacobi equations under constraints.SIAM J. Control Optim., 49:1881–1915, 2011.

14. J.-P. Aubin and A. Cellina. Differential inclusions: set-valued maps and viability theory.Springer-Verlag New York, Inc., 1984.

15. J.-P. Aubin and H. Frankowska. Set-Valued Analysis. Birkhauser Boston, 1990.16. J.-P. Aubin and H. Frankowska. The viability kernel algorithm for computing value functions

of infinite horizon optimal control problems. J. Math. Anal. Appl., 201:555–576, 1996.17. M. Bardi and M. Falcone. An approximation scheme for the minimum time function. SIAM

J. Control Optim., 28(4):950–965, 1990.18. Martino Bardi and Italo Capuzzo-Dolcetta. Optimal control and viscosity solutions of

Hamilton-Jacobi-Bellman equations. Systems & Control: Foundations & Applications.Birkhauser Boston Inc., Boston, MA, 1997. With appendices by Maurizio Falcone and Pier-paolo Soravia.

19. G. Barles. Solutions de viscosite des equatiuons de Hamilton-Jacobi. Math. Appl. (Berlin)17, Springer-Verlag, Paris, 1994.

20. G. Barles and P.E. Souganidis. Convergence of approximation schemes for fully nonlinearsecond order equations. Asymptot. Anal., 4:271–283, 1991.

21. R.C. Barnard and P.R. Wolenski. Flow invariance on stratified domains. Set Valued Var. Anal.,pages 1–27, 2012.

22. E. N. Barron. Viscosity solutions and analysis in l∞. Proceeding of the NATO advanced studyInstitute, pages 1–60, 1999.

23. E. N. Barron and H. Ishii. The bellman equation for minimizing the maximum cost. NonlinearAnal-Theor., 13(9):1067–1090, 1989.

24. J.-D. Benamou and G. Carlier. Augmented lagrangian methods for transport optimization,mean field games and degenerate elliptic equations. J. Optim. Theory Appl., 167(1):1–26,2015.

25. O. Bokanowski, Y. Cheng, and C.-W. Shu. A discontinuous galerkin scheme for front propa-gation with obstacle. Numer. Math., 126(2):1–31, 2013.

26. O. Bokanowski, M. Falcone, R. Ferretti, L. Grune, D. Kalise, and H. Zidani. Value itera-tion convergence of ε-monotone schemes for stationary Hamilton-Jacobi equations. DiscreteContin. Dyn, Syst. Se. A, 35:4041–4070, 2015.

27. O. Bokanowski, M. Falcone, and S. Sahu. An efficient filtered scheme for some first or-der time-dependent hamilton–jacobi equations. SIAM Journal on Scientific Computing,38(1):A171–A195, 2016.

28. O. Bokanowski, N. Forcadel, and H. Zidani. Reachability and minimal times for state con-strained nonlinear problems without any controllability assumption. SIAM J. Control Optim.,48(7):4292–4316, 2010.

29. O. Bokanowski, N. Forcadel, and H. Zidani. Deterministic state-constrained optimal controlproblems without controllability assumptions. ESAIM Contr. Optim. Ca., 17(04):995–1015,2011.

30. O. Bokanowski, A. Picarelli, and H. Zidani. State constrained stochastic optimal controlproblems via reachability approach. SIAM J. Control Optim., 2016. To appear.

31. M.S. Branicky, V.S. Borkar, and S.K. Mitter. A unified framework for hybrid control: Modeland optimal control theory. IEEE Trans. Automat. Contr., 43(1):31–45, 1998.

32. M. Breuß, E. Cristiani, J.-D. Durou, M. Falcone, and O. Vogel. Perspective shape fromshading: ambiguity analysis and numerical approximations. SIAM J. Imaging Sci., 5(1):311–342, 2012.

33. Francis C., Y. Ledyaev, R. Stern, and P. Wolenski. Nonsmooth Analysis and Control Theory,volume 178. Springer, 1998.

34. F. Camilli and M. Falcone. An approximation scheme for the optimal control of diffusionprocesses. RAIRO Model. Math. Anal. Numer., 29(1):97–122, 1995.

35. F. Camilli and L. Grune. Numerical approximation of the maximal solutions for a class ofdegenerate Hamilton-Jacobi equations. SIAM J. Numer. Anal., 38(5):1540–1560, 2000.

36. F. Camilli and A. Siconolfi. Maximal subsolutions for a class of degenerate Hamilton-Jacobiproblems. Indiana Univ. Math. J., 48(3):1111–1131, 1999.

37. F. Camilli and F. J. Silva. A semi-discrete in time approximation for a first order-finite meanfield game problem. Netw. Heterog. Media, 7(2):263–277, 2012.

38. P. Cannarsa and T. Scarinci. Conjugate times and regularity of the minimum time functionwith differential inclusions. In Analysis and geometry in control theory and its applications,volume 11 of Springer INdAM Ser., pages 85–110. Springer, Cham, 2015.

39. P. Cannarsa and C. Sinestrari. Semiconcave Functions, Hamilton-Jacobi Equations, and Opti-mal Control. Progress in Nonlinear Differential Equations and Their Applications. Birkauser,2004.

40. I. Capuzzo-Dolcetta and P.-L. Lions. Hamilton-Jacobi equations with state constraints. Trans.Amer. Math. Soc., 318(2):643–683, 1990.

41. P. Cardaliaguet. Notes on Mean Field Games: from P.-L. Lions’ lectures at College de France.2013.

42. P. Cardaliaguet, M. Quincampoix, and P. Saint-Pierre. Optimal times for constrained nonlin-ear control problems without local controllability. Appl. Math. Optim, 36:21–42, 1997.

43. P. Cardaliaguet, M. Quincampoix, and P. Saint-Pierre. Numerical schemes for discontinuousvalue functions of optimal control. Set-valued Analysis, 8:111–126, 2000.

44. E. Carlini, M. Falcone, and R. Ferretti. An efficient algorithm for Hamilton-Jacobi equationsin high dimension. Comput. Vis. Sci., 7(1):15–29, 2004.

45. E. Carlini, R. Ferretti, and G. Russo. A weighted essentially nonoscillatory, large time-stepscheme for Hamilton-Jacobi equations. SIAM J. Sci. Comput., 27(3):1071–1091, 2005.

46. E. Carlini and F. J. Silva. Semi-lagrangian discretizations of fokker-planck equations. Inpreparation.

47. E. Carlini and F. J. Silva. Semi-lagrangian schemes for mean field game models. In Decisionand Control (CDC), 2013 IEEE 52nd Annual Conference on, pages 3115–3120, Dec 2013.

48. E. Carlini and F. J. Silva. A fully discrete semi-lagrangian scheme for a first order mean fieldgame problem. SIAM J. Numer. Anal., 52(1):45–67, 2014.

49. E. Carlini and F. J. Silva. A semi-lagrangian scheme for a degenerate second order mean fieldgame system. Discret. Contin. Dyn. S., 35(9):4269–4292, 2015.

50. F.H. Clarke and R.J. Stern. Hamilton-jacobi characterization of the state constrained value.Nonlinear Anal-Theor., 61(5):725–734, 2005.

51. B. Cockburn and C.-W. Shu. TVB Runge-Kutta local projection discontinuous Galerkinfinite element method for conservation laws III: One-dimensional systems,. J. Comput. Phys.,84:90–113, 1989.

52. M. G. Crandall and P.-L. Lions. Viscosity solutions of Hamilton-Jacobi equations. T. Am.Math. Soc., 277(1):1–42, 1983.

53. M. G. Crandall and P. L. Lions. Two approximations of solutions of Hamilton-Jacobi equa-tions. Comput. Methods Appl. Mech. Engrg., 195:1344–1386, 1984.

54. K. Debrabant and E. R. Jakobsen. Semi-Lagrangian schemes for linear and fully non-lineardiffusion equations. Math. Comp., 82(283):1433–1462, 2013.

55. S. Dharmatti and M. Ramaswamy. Hybrid control systems and viscosity solutions. SIAM J.Control Optim., 44(4):1259–1288, 2005.

56. I. Capuzzo Dolcetta. On a discrete approximation of the Hamilton-Jacobi equation of dy-namic programming. Appl. Math. Opt., 10:367–377, 1983.

57. I. Capuzzo Dolcetta and H. Ishii. Approximate solutions of the Bellman equation of deter-ministic control theory. Appl. Math. Opt., 11:161–181, 1984.

58. J.-D. Durou, M. Falcone, and M. Sagona. A survey of numerical methods for shape fromshading. Comput. Vis. Image Underst., 109(1):22–43, 2008.

59. C. L. Epstein and M. Gage. The curve shortening flow. In Wave motion: theory, modelling,and computation (Berkeley, Calif., 1986), volume 7 of Math. Sci. Res. Inst. Publ., pages 15–59. Springer, New York, 1987.

60. M. Falcone and R. Ferretti. Discrete time high-order scheme for viscosity solutions ofHamilton-Jacobi equation. Numer. Math., 67:315, 1994.

61. M. Falcone and R. Ferretti. Convergence analysis for a class of high-order semi-Lagrangianadvection schemes. SIAM J. Numer. Anal., 35(3):909–940, 1998.

62. M. Falcone and R. Ferretti. Semi-Lagrangian Approximation Schemes for Linear andHamilton-Jacobi Equations. MOS-SIAM Series on Optimization, 2013.

63. M. Falcone, T. Giorgi, and P. Loreti. Level sets of viscosity solutions: Some applications tofronts and rendez-vous problems. SIAM J. Appl. Math., 54:1335–1354, 1994.

64. R. Ferretti. Convergence of semi-Lagrangian approximations to convex Hamilton-Jacobiequations under (very) large Courant numbers. SIAM J. Numer. Anal., 40(6):2240–2253,2002.

65. R. Ferretti and H. Zidani. Monotone numerical schemes and feedback construction for hybridcontrol systems. J. Optim. Theory Appl., 165(2):507–531, 2015.

66. A. Festa and M. Falcone. An approximation scheme for an Eikonal equation with discontin-uous coefficient. SIAM J. Numer. Anal., 52(1):236–257, 2014.

67. U. S. Fjordholm. High-order accurate entropy stable numerical schemes for hyperbolic con-servation laws. PhD thesis, ETH Zurich Switzerland, 2013.

68. U. S. Fjordholm, S. Mishra, and E. Tadmor. Arbitrarily high order accurate entropy stableessentially non-oscillatory schemes for systems of conservation laws. SIAM J. Numer. Anal.,50:423–444, 2012.

69. H. Frankowska and M. Mazzola. Discontinuous solutions of Hamilton–Jacobi–Bellman equa-tion under state constraints. Calc. Var. Partial Differ. Equ., 46(3-4):725–747, 2013.

70. H. Frankowska and L.V. Nguyen. Local regularity of the minimum time function. J. Optim.Theory Appl., 164(1):68–91, 2015.

71. H. Frankowska and S. Plaskacz. Semicontinuous solutions of Hamilton-Jacobi-Bellman equa-tions with degenerate state constraints. J. Math. Anal. Appl., 251(2):818–838, 2000.

72. H. Frankowska and R. B. Vinter. Existence of neighboring feasible trajectories: Applicationsto dynamic programming for state-constrained optimal control problems. J. Optim. TheoryAppl., 104(1):20–40, 2000.

73. B. D. Froese and A. M. Oberman. Convergent filtered schemes for the monge-ampre partialdifferential equation. SIAM J. Numer. Anal., 51:423–444, 2013.

74. D. Gomes and J. Saude. Mean field games models—a brief survey. Dyn. Games Appl.,4(2):110–154, 2014.

75. S. Gottlieb and C.-W. Shu. Total variation diminishing Runge-Kutta schemes. Math. Comp.,67(221):73–85, 1998.

76. F. Gozzi and P. Loreti. Regularity of the minimum time function and minimum energy prob-lems: the linear case. SIAM J. Control Optim., 37(4):1195–1221, 1999.

77. O. Gueant. Mean field games equations with quadratic Hamiltonian: a specific approach.Math. Models Methods Appl. Sci., 22(9):1250022, 37, 2012.

78. O. Gueant. New numerical methods for mean field games with quadratic costs. Netw. Heterog.Media, 7:315 – 336, 2012.

79. A. Harten. High resolution schemes for hyperbolic conservation laws. J. Comput. phys.,49:357–393, 1983.

80. A. Harten. On a class of high resolution total-variation finite difference schemes. SIAM J.Numer. Anal., 21:1–23, 1984.

81. A. Harten, B. Engquist, S. Osher, and S.R. Chakravarty. Uniformly high order essentiallynon-oscillatory schemes. J. Comput. phys., 4:231–303, 1987.

82. C. Hermosilla and H. Zidani. Infinite horizon problems on stratifiable state-constraints sets.J. Differ. Equations, 258(4):1430–1460, 2015.

83. B. Horn. Obtaining shape from shading information. In The psychology of computer vision,pages 115–155. McGraw-Hill, New York, 1975.

84. H. Ishii. Uniqueness of unbounded viscosity solution of Hamilton-Jacobi equations. IndianaUniv. Math. Journal, 33(5):721–748, 1984.

85. H. Ishii and S. Koike. A new formulation of state constraint problems for first-order pdes.SIAM Journal on Control and Optimization, 34(2):554–571, 1996.

86. H. Ishii and M. Ramaswamy. Uniqueness results for a class of Hamilton-Jacobi equationswith singular coefficients. Comm. Partial Differential Equations, 20(11-12):2187–2213,1995.

87. V.Y. Kaloshin. A geometric proof of the existence of whitney stratifications. Mosc. Math. J,5(1):125–133, 2005.

88. A. Kurganov and E. Tadmor. New high-resolution semi-discrete central schemes forHamilton-Jacobi equations. J. Comput. Phys., 160:720–742, 2000.

89. A. B. Kurzhanski, I. M. Mitchell, and P. Varaiya. Optimization techniques for state-constrained control and obstacle problems. J. Optim. Theory Appl., 128:499–521, 2006.

90. A. B. Kurzhanski and P. Varaiya. Ellipsoidal techniques for reachability under state con-straints. SIAM J. Control Optim., 45:1369–1394, 2006.

91. A. Lachapelle, J. Salomon, and G. Turinici. Computation of mean field equilibria in eco-nomics. Math. Mod. Meth. Appl. S., 20-4:567–588, 2010.

92. J.-M. Lasry and P.-L. Lions. Jeux a champ moyen I. Le cas stationnaire. C. R. Math. Acad.Sci. Paris, 343:619–625, 2006.

93. J.-M. Lasry and P.-L. Lions. Jeux a champ moyen II. Horizon fini et controle optimal. C. R.Math. Acad. Sci. Paris, 343:679–684, 2006.

94. J.-M. Lasry and P.-L. Lions. Mean field games. Jpn. J. Math., 2:229–260, 2007.95. G. Leoni. A first course in Sobolev spaces, volume 105. American Mathematical Soc., 2009.96. O. Lepsky, C. Hu, and C.-W. Shu. Analysis of the discontinuous Galerkin method for

Hamilton-Jacobi equations. Appl. Num. Math., 33:423–434, 2000.97. F. Li and C.-W. Shu. Reinterpretation and simplified implementation of a discontinuous

Galerkin method for Hamilton-Jacobi equations. SIAM J. Sci. Comput. Anal., 18(11):1204–1209, 2005.

98. C.-T. Lin and E. Tadmor. L1-stability and error estimates for approximate Hamilton-Jacobisolutions. Numer. Math., 87(4):701–735, 2001.

99. P.-L. Lions. Cours au College de France. www.college-de-france.fr, 2007-2008.100. P.-L. Lions, E. Rouy, and A. Tourin. Shape-from-shading, viscosity solutions and edges.

Numer. Math., 64(3):323–353, 1993.101. P.-L. Lions and P. E. Souganidis. Convergence of MUSCL and filtered schemes for scalar

conservation laws and Hamilton-Jacobi equations. Numer. Math., 69(4):441–470, 1995.102. P. Loreti. Some properties of constrained viscosity solutions of Hamilton-Jacobi-Bellman

equations. SIAM J. Control Optim., 25:1244–1252, 1987.103. P. Loreti and E. Tessitore. Approximation and regularity results on constrained viscosity so-

lutions of Hamilton-Jacobi-Bellman equations. J. Math. Systems, Estimation Control, 4:467–483, 1994.

104. J. Lygeros. On reachability and minimum cost optimal control. Automatica, 40:917–927,2004.

105. J. Lygeros, C. Tomlin, and S. Sastry. Controllers for reachability specifications for hybridsystems. Automatica, 35:349–370, 1999.

106. K. Margellos and J. Lygeros. Hamilton-jacobi formulation for reach-avoid differentialgames. IEEE Trans. Automat Contr., 56:1849–1861, 2011.

107. M. Motta. On nonlinear optimal control problems with state constraints. SIAM J. ControlOptim., pages 1411–1424.

108. M. Motta and F. Rampazzo. Multivalued dynamics on a closed domain with absorbingboundary. applications to optimal control problems with integral constraints. Nonlinear Anal.,41:631–647, 2000.

109. C. Nour and R.J. Stern. The state constrained bilateral minimal time function. NonlinearAnal-Theor., 69(10):3549–3558, 2008.

110. A. M. Oberman and T. Salvador. Filtered schemes for Hamilton-Jacobi equations: a simpleconstruction of convergent accurate difference schemes. J. Comput. Phys., 284:367–388,2015.

111. S. Osher and J. A. Sethian. Fronts propagation with curvature dependent speed: algorithmsbased on Hamilton-Jacobi formulations. J. Comput. Phys., 79:12–49, 1988.

112. S. Osher and C.-W. Shu. High order essentially nonoscillatory schemes for Hamilton-Jacobiequations. SIAM J. Numer. Anal., 4:907–922, 1991.

113. R. T. Rockafellar. Proximal subgradients, marginal values, and augmented lagrangians innonconvex optimization. Math. Oper. Res., 6(3):424–436, 1981.

114. S. Sahu. High-order filtered scheme for front propagation problems. Bulletin of the BrazilianMathematical Society, 47(2):727–744, 2016.

115. J. A. Sethian. Evolution, implementation, and application of level set and fast marchingmethods for advancing fronts. J. Comput. Phys., 169(2):503–555, 2001.

116. H. Soner. Optimal control with state-space constraint I. SIAM J. Control Optim., 24(3):552–561, 1986.

117. P. Soravia. Estimates of convergence of fully discrete schemes for the Isaacs equation ofpursuit-evasion differential games via maximum principle. SIAM J. Control Optim., 36(1):1–11, 1998.

118. R.J. Stern. Characterization of the state constrained minimal time function. SIAM J. ControlOptim., 43(2):697–707, 2004.

119. L. Van den Dries and C. Miller. Geometric categories and o-minimal structures. Duke Math.J, 84(2):497–540, 1996.

120. P. Wolenski and Y. Zhuang. Proximal analysis and the minimal time function. SIAM J.Control Optim., 36(3):1048–1072, 1998.