optimal control of hamilton-jacobi-bellman equations

47
Introduction Main results Proofs Further results Optimal control of Hamilton-Jacobi-Bellman equations P. Jameson Graber Commands (ENSTA ParisTech, INRIA Saclay) 17 January, 2014 P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Upload: others

Post on 14-Jul-2022

11 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Optimal control of Hamilton-Jacobi-Bellmanequations

P. Jameson Graber

Commands (ENSTA ParisTech, INRIA Saclay)

17 January, 2014

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 2: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 3: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Objective

We consider an optimal control problem:

Problem

(1) For a given cost functional K and finite measure dm0, find

inf

∫ T

0

∫K (t, x , f (t, x))dxdt −

∫u(0, x)dm0(x)

over functions u, f such that f ∈ C ([0,T ]× TN) and u solves

−∂tu(t, x) + H(x ,Du(t, x)) = f (t, x), u(T , x) = uT (x).

(2) Find a PDE characterization of (weak) minimizers.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 4: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Motivation

Purpose: to steer an evolving set.Model: given a set A of admissible controls and a given regionΩT ⊂ RN , let Ωt be the backward reachable set at time t defined by

Ωt := y(t) : y = c(y , α), α ∈ A, y(T ) ∈ ΩT.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 5: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Model

Strategy: use obstacles.Obstacle construction: Let f ≥ 0 be continuous. Define Kτ , τ ∈ [0,T ] by

Kτ := x : f (τ, x) > 0.

Consider the modified “blocked” evolving set

Ωt = y(t) : y = c(y , α), α ∈ A, y(T ) ∈ ΩT , y(τ) /∈ Kτ ∀ τ ∈ [t,T ].

Following [Bokanowski, Forcadel, Zidani 2010], we have acharacterization: let uT ≥ 0 (continuous) be such that uT = 0 preciselyon ΩT .

Ωt = y(t) : uT (y(T )) +

∫ T

t

f (s, y(s)) ≤ 0, y = c(y , α), α ∈ A.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 6: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Model

Ωt = u(t, ·) ≤ 0 characterized by an optimal control problem:

u(t, x) := inf

uT (y(T )) +

∫ T

t

f (s, y(s))ds : y = c(y , α), α ∈ A, y(t) = x

.

u solves a Hamilton-Jacobi-Bellman equation:

−∂tu(t, x) + H(x ,Du(t, x)) = f (t, x), u(T , x) = uT (x).

Conclusion

Steering the front by obstacles is thus related to optimizing solutions toHJB equations.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 7: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Related model

A related open problem:

Optimal control via velocity

Given a cost functional C, find

inf

C(c)−

∫u(T , x)dm0(x)

,

where the infimum is taken over u, c satisfying

ut + c(t, x)|Du| = 0, u(0, x) = u0(x).

Find a characterization of the minimizers.

Very little appears to be known.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 8: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 9: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Mass transportation

In 2000, Benamou and Brenier considered a fluid mechanics formulationof the Monge-Kantorovich problem (MKP).For ρ0(x) ≥ 0 and ρT (x) ≥ 0 probability densities on Rd , we sayM : Rd → Rd transfers ρ0 to ρT if ∀A ⊂ Rd bounded,∫

x∈AρT (x)dx =

∫M(x)∈A

ρ0(x)dx .

The Lp MKP is to find

dp(ρ0, ρT )p = infM

∫|M(x)− x |pρ0(x)dx .

dp is called the Lp Kantorovich (or Wasserstein) distance.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 10: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

PDE characterization of MKP

Theorem (Benamou-Brenier, 2000)

The square of the L2 Kantorovich distance is equal to the infimum of

T

∫Rd

∫ T

0

ρ(t, x)|v(t, x)|2dxdt,

among all (ρ, v) satisfying

∂tρ+∇ · (ρv) = 0

for 0 < t < T and x ∈ Rd , and the initial-final condition

ρ(0, ·) = ρ0, ρ(T , ·) = ρT .

The optimality conditions are given by v(t, x) = ∇φ(t, x), where

∂tφ+1

2|∇φ|2 = 0.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 11: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Dolbeault-Nazaret-Savare distances

In 2009, Dolbeault, Nazaret, and Savare, building on Benamou-Brenier,introduced a new class of transport distances between measures, given byminimizing ∫ 1

0

∫Rd

|v(t, x)|2m(ρ(t, x))dxdt

subject to

∂t +∇ · (m(ρ)v) = 0, ρ(0, ·) = ρ0, ρ(1, ·) = ρ1.

In 2013, Cardaliaguet, Carlier, and Nazaret studied geodesics for thisclass of distances when m(ρ) = ρα, getting optimality conditions ∂tρ+∇ · ( 1

2ρα∇φ) = 0,

ρ > 0⇒ ∂tφ+ α4 ρ

α−1|∇φ|2 = 0,ρ ≥ 0, ∂tφ ≤ 0, ρ(0, ·) = ρ0, ρ(1, ·) = ρ1.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 12: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

MotivationBackground

Mean Field Games

In 2007, Lasry and Lions give an overview of the following system ofequations:

∂u∂t − ν∆u + H(x ,∇u) = V [m] in Q × (0,T )

∂m∂t + ν∆m +∇ ·

(∂H∂p (x ,∇u)m

)= 0 in Q × (0,T )

u|t=0 = V0[m(0)] on Q, m|t=T = m0 on Q.

There are two characterizations of the system:

1 asymptotic behavior of differential games with large number ofplayers,

2 optimal control of Fokker-Planck equation, or of Hamilton-Jacobiequation (by duality).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 13: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 14: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Assumptions

Hamiltonian structure. H = H(x , p) is Lipschitz in the first variable:

|H(x , p)− H(y , p)| ≤ L|x − y ||p|.

H(x , ·) is subadditive and positively homogeneous, hence convex, withlinear bounds

c0|p| ≤ H(x , p) ≤ c1|p|.

H(x , p) := sup−c(x , a) · p : a ∈ ACost function. k = k(t, x ,m) continuous and strictly increasing with

1

C|m|q−1 − C ≤ k(t, x ,m) ≤ C |m|q−1 + C .

K∗(t, x , ·) a primitive of k, K(t, x , ·) its Fenchel conjugate. We willassume that K(t, x , 0) = 0 and that K(t, x , f ) is increasing in f for f ≥ 0.Note that

1

C|f |p − C ≤ K(t, x , f ) ≤ C |f |p + C .

Final-initial conditions. uT : TN → R is Lipschitz, m0 ∈ L∞(TN)

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 15: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Comparison with the literature

1 This is the first study with linearly bounded Hamiltonian.

2 In [Cardaliaguet, 2013] the Hamiltonian is supposed to satisfy

1

rC|p|r − C2 ≤ H(x , p) ≤ C

r|p|r + C

for some r > N(q − 1) ∨ 1 and, for some θ ∈ [0, rN+1 ),

|H(x , p)− H(y , p)| ≤ C |x − y |(|p| ∨ 1)θ.

The Hamiltonian H(x , p) = c(x)|p|r is forbidden by this restriction.By contrast, H(x , p) = c(x)|p| is permitted under our assumptions.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 16: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Existence theorem

Relaxed set

The set K will be defined as the set of all pairs (u, f ) ∈ BV × Lp suchthat u ∈ L∞, u(T , ·) = uT in the sense of traces, and−∂tu + H(x ,Du) ≤ f in the sense of measures, by which we mean thatf − H(x ,Du) + ∂tu is a non-negative Radon measure.

Relaxed problem: Find inf(u,f )∈KA(u, f ).

Lemma

The relaxed problem has the same infimum as the original.

Theorem (PJG)

There exists a minimizer to the relaxed problem.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 17: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Characterization of minimizers

Heuristically, use Lagrange multipliers to get system (MFG)

−∂tu + H(x ,Du) = k(t, x ,m(t, x)) (state equation)u(T , x) = uT (x)

∂tm − div (m∂pH(x ,Du)) = 0 (adjoint equation)m(0, x) = m0(x)

which characterizes minimizers (u, f ) of the relaxed problem (the optimalchoice is f = k(t, x ,m)).Problems:

H not differentiable,

weak solutions otherwise hard to define.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 18: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Definition of weak solutions

A pair (u,m) ∈ BV ((0,T )× TN)× Lq((0,T )× TN) is called a weaksolution to the MFG system if it satisfies the following conditions.

1 u satisfies∫TN

u(0)m0dx =

∫TN

u(t)m(t)dx +

∫ t

0

∫TN

k(s, x ,m)mdxds,∫TN

u(t)m(t)dx =

∫TN

uTm(T )dx +

∫ T

t

∫TN

k(s, x ,m)mdxds,

for almost all t ∈ [0,T ]. We also have−∂tu + H(x ,Du) ≤ k(t, x ,m) in the sense of measures.Moreover, u(T , ·) = uT in the sense of traces

2 m satisfies the continuity equation

∂tm + div (mv) = 0 in (0,T )× TN , m(0) = m0

in the sense of distributions, where v ∈ L∞((0,T )× TN ;RN) is avector field such that v(t, x) ∈ c(x ,A) a.e.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 19: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

AssumptionsExistence of minimizersCharacterization of minimizers

Well-posedness

Theorem (PJG)

(i) There exists a weak solution (u,m) of (MFG). Moreover, if u ∈W 1,p,then the Hamilton-Jacobi equation

−∂tu − v · Du = f

holds pointwise almost everywhere in m > 0, where v is the boundedvector field appearing in the definition.(ii) If (u,m) and (u′,m′) are both weak solutions (MFG), then m = m′

almost everywhere while u = u′ almost everywhere in the set m > 0.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 20: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 21: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Smooth problem

Denote by K0 the set of maps u ∈ C 1([0,T ]× TN) such thatu(T , x) = uT (x). The “smooth problem” is to compute the infimumover K0 of the functional

A(u) =

∫ T

0

∫TN

K (t, x ,−∂tu + H(x ,Du))dxdt −∫TN

u(0, x)dm0(x).

Lemma

The “smooth problem” is equivalent to the original problem, i.e.

infu∈K0

A(u) = inff∈C([0,T ]×TN )

J (f ).

Proof: uses the superposition principle.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 22: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Dual problem

Define K1 to be the set of all pairs(m,w) ∈ L1((0,T )×TN)× L1((0,T )×TN ;RN) such that m ≥ 0 almosteverywhere, w(t, x) ∈ m(t, x)c(x ,A) almost everywhere, and

∂tm + div (w) = 0

m(0, ·) = m0(·)

in the sense of distributions. The “dual problem” is to compute theinfimum over K1 of

B(m,w) =

∫TN

uT (x)m(T , x)dx +

∫ T

0

∫TN

K∗(t, x ,m(t, x))dxdt.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 23: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Proof of duality

Lemma

The smooth and dual problems really are in duality, i.e.

infu∈K0

A(u) = − min(m,w)∈K1

B(m,w).

Moreover, the minimum on the right hand side is achieved by a pair(m,w) ∈ K1, of which m is unique, and which must satisfy the following:(t, x) 7→ K∗(t, x ,m(t, x)) ∈ L1((0,T )× TN) and m ∈ Lq((0,T )× TN).

Proof is an application of the Fenchel-Rockafellar duality Theorem. Let

X := C 1([0,T ]× TN ;R),

Y := C ([0,T ]× TN ;R)× C ([0,T ]× TN ;RN).

Let Λ : X → Y be given by Λ(u) = (∂tu,Du).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 24: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Proof of duality (cont)

Lemma

infu∈K0 A(u) = −min(m,w)∈K1B(m,w).

Then define the functionals F : X → R and G : Y → R by

F (u) =

−∫TN u(0, x)dm0(x) if u(T , ·) = uT (·)

∞ otherwise

G (a,b) =∫ T

0

∫TN K (t, x ,−a + H(x ,b))dxdt

One can show the lemma is equivalent to

max(m,w)∈Y ′

−F ∗(Λ∗(m,w))− G ∗(−(m,w))

= − min(m,w)∈Y ′

F ∗(Λ∗(m,w)) + G ∗(−(m,w)).

Apply Fenchel-Rockafellar to finish.P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 25: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 26: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Original and relaxed problems equivalent

Below we prove the following:

Proposition

inf(u,f )∈K

A(u, f ) = − min(m,w)∈K1

B(m,w).

Equivalently,inf

(u,f )∈KA(u, f ) = inf

u∈K0

A(u).

First part of the proof:Since for u ∈ K0 we have that (u,−ut + H(x ,Du)) ∈ K, it follows thatinf(u,f )∈KA(u, f ) ≤ infu∈K0 A(u).It remains to show other inequality.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 27: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

A technical lemma

Lemma

For u ∈ BV and f an integrable function, the following are equivalent:

−∂tu + H(x ,Du) ≤ f in the sense of measures, and

for every continuous vector field v = v(t, x) ∈ c(x ,A) we have

−∫ T

0

∫TN

φ∂tu + φv · Du ≤∫ T

0

∫TN

f φdtdx .

for every continuous function φ : [0,T ]× TN → [0,∞).

Remark: the crucial thing is that we only need to consider v continuous.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 28: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Proof of technical lemma

Lemma

−∂tu + H(x ,Du) ≤ f ⇔ −∫∫

φ(∂tu + v · Du) ≤∫∫

f φ ∀v cont.

One direction is obvious. For the other direction, let ε > 0. By Lusin’sTheorem, there is a compact set K with |Du|([0,T ]× TN \ K ) ≤ ε andDu/|Du| continuous on K . By continuity and convexity of c(·,A) weconstruct v(t, x) ∈ c(x ,A) by partition of unity so that−v · Du/|Du| ≥ H(x ,Du/|Du|)− ε. Use this to obtain the estimate∫ T

0

∫TN

φH(x ,Du) ≤ Cε+

∫ T

0

∫TN

φv · Du.

Then let ε→ 0.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 29: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Key lemma

Lemma

Suppose u ∈ BV ∩ L∞ satisfies −∂tu + H(x ,Du) ≤ f in the sense ofmeasures. Then for any m ∈ Lq satisfying the continuity equation∂tm + div(mv) = 0 for some vector field v with v(t, x) ∈ c(x ,A) a.e., wehave ∫

TN

u(0)m(0)dx ≤∫TN

u(t)m(t)dx +

∫ t

0

∫TN

fmdxds,∫TN

u(t)m(t)dx ≤∫TN

uTm(T )dx +

∫ T

t

∫TN

fmdxds,

for almost every t ∈ (0,T ).

Proof: obtain smooth approximations of m and mv through time scalingand convolution, apply previous lemma and integration by parts.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 30: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Proof of equivalence

Second half of the proof: Let (m,w) be a minimizer of the dual problem.It suffices to show that for (u, f ) ∈ K, we have A(u, f ) ≥ −B(m,w).This essentially follows from the last lemma. Indeed, we have

A(u, f ) =

∫ T

0

∫TN

K (t, x , f (t, x))dxdt −∫TN

u(0, x)m0(x)dx

≥∫ T

0

∫TN

fm − K∗(t, x ,m)dxdt −∫TN

u(0, x)m0(x)dx

≥ −∫ T

0

∫TN

K∗(t, x ,m)dxdt −∫TN

uT (x)m(T , x)dx = −B(m,w).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 31: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 32: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Upper bound lemma

Lemma

Suppose u is a continuous viscosity solution to the Hamilton-Jacobiequation

−∂tu + H(x ,Du) = f ≥ 0

in a region [0,T ]× U for some open set U in Euclidean space. Then forany 0 ≤ t < s ≤ T and for any 0 < β < 1, we have the followingestimate:

u(t, x)− u(s, y) ≤ C (1− β2)−N/2p‖f ‖p|t − s|α, ∀|x − y | ≤ βc0(s − t),

where α = 1− (N + 1)/p and the constant C depends on p,N, and c−10 .

Cf. [Cardaliaguet 2013]. This lemma gives us upper bounds.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 33: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Passing to the limit

Let (un, fn) be a minimizing sequence. WLOG, un and fn are Lipschitz,hence differentiable a.e., and fn ≥ 0. We have:

fn bounded in Lp by estimates on K , so fn → f weakly.

This implies un pointwise bounded.

The above implies H(x ,Dun) is bounded in L1, so Dun is as well.

The above implies ∂tun is bounded in L1, so un is bounded in BV .

(∂tun,Dun)→ (∂tu,Du) in the weak measure sense, while un → uin L1.

Conclusion: (u, f ) is the minimizer.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 34: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 35: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Minimizers are weak solutions

Suppose (u, f ) ∈ K is a minimizer of relaxed problem and (m,w) ∈ K1 isa minimizer of dual problem.

We know that −∂tu + H(x ,Du) ≤ f in the sense of measures, so∫ t

0

∫fm +

∫u(t)m(t)− u(0)m0 ≥ 0,∫ T

t

∫fm +

∫uTm(T )− u(t)m(t) ≥ 0.

The main point is to get∫ T

0

∫fm +

∫uTm(T )− u(0)m0 = 0, so

that the above inequalities become equality.

Everything else comes directly from the definition of K and K1 andthe properties of minimizers already proved.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 36: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Minimizers are weak solutions, part 2

(u, f ) ∈ K is a minimizer of relaxed problem and (m,w) ∈ K1 is aminimizer of dual problem.By the duality theorem,

0 =

∫ T

0

∫K (t, x , f ) + K∗(t, x ,m) +

∫uTm(T )− u(0)m0

≥∫ T

0

∫fm +

∫uTm(T )− u(0)m0 ≥ 0,

which implies the desired result.It also implies K (t, x , f (t, x)) + K∗(t, x ,m(t, x)) = f (t, x)m(t, x)a.e. sof (t, x) = k(t, x ,m).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 37: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Weak solutions are also minimizers

Suppose (u,m) is a weak solution. We set w = mv and f = k(·, ·,m).Then (u, f ) is a minimizer of the relaxed problem and (m,w) a minimizerof the dual problem. For example:

A(u′, f ′) =

∫∫K (t, x , f ′)−

∫u′(0)m0

≥∫∫

K (t, x , f ) + ∂f K (t, x , f )(f ′ − f )−∫

u′(0)m0

=

∫∫K (t, x , f ) + m(f ′ − f )−

∫u′(0)m0

=

∫∫K (t, x , f ) + mf ′ +

∫m(T )uT −m0u(0)− u′(0)m0

≥∫∫

K (t, x , f )−∫

m0u(0) = A(u, f ).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 38: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

DualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

Weak solutions are unique

Suppose (u1,m1) and (u2,m2) are both weak solutions.

1 m1 and m2 are both minimizers of dual problem, so m1 = m2 =: m.

2 f := k(·, ·,m), then both (u1, f ) and (u2, f ) are minimizers ofrelaxed problem.

3 u := maxu1, u2; we show −∂tu + H(x ,Du) ≤ f in the sense ofmeasures.

4 Then (u, f ) is a minimizer of relaxed problem, so∫∫K (t, x , f )−

∫ui (0)m0 =

∫∫K (t, x , f )−

∫u(0)m0.

5 Combine with previous inequalities and use u ≥ ui to get u = ui inm > 0.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 39: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Table of Contents

1 IntroductionMotivationBackground

2 Main resultsAssumptionsExistence of minimizersCharacterization of minimizers

3 ProofsDualityEquivalence of original and relaxed problemsA priori boundsPassing to the limitExistence of weak solutionsUniqueness of weak solutions

4 Further resultsWell-posedness of weak solutionsLong time average behavior

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 40: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Recent work

The following is joint work with Pierre Cardaliaguet. We study (i) −∂tφ+ H(x ,Dφ) = f (x ,m)(ii) ∂tm − div (mDpH(x ,Dφ)) = 0(iii) φ(T , x) = φT (x),m(0, x) = m0(x)

and the corresponding ergodic problem (i) λ+ H(x ,Dφ) = f (x ,m(x))(ii) −div(mDpH(x ,Dφ)) = 0(iii) m ≥ 0,

∫Td m = 1

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 41: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Assumptions

1 (Initial-final conditions) m0 ∈ C (Td), m0 > 0, φT : Td → RLipschitz.

2 (Conditions on the Hamiltonian) H : Td × Rd → R is continuous inboth variables, convex and differentiable in the second variable, withDpH continuous in both variables. Also,

1

rC|p|r − C ≤ H(x , p) ≤ C

r|p|r + C .

3 (Conditions on the coupling) Let f be continuous on Td × (0,∞),strictly increasing in the second variable, satisfying

1

C|m|q−1 − C ≤ f (x ,m) ≤ C |m|q−1 + C ∀ m ≥ 1.

4 The relation holds between the growth rates of H and of F :

r > maxd(q − 1), 1.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 42: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Optimal control problemsOptimal control of HJ equation: find the infimum of

A(φ) =

∫ T

0

∫Td

F∗(x ,−∂tφ+ H(x ,Dφ))dxdt −∫Tdφ(0, x)dm0(x)

over the set of maps φ ∈ C1([0,T ]× Td ) such that φ(T , x) = φT (x).Optimal control of transport equation: find the infimum of

B(m,w) =

∫TdφTm(T )dx +

∫ T

0

∫Td

mH∗(x ,−

w

m

)+ F (x ,m)dxdt

over (m,w) ∈ L1((0,T )× Td )× L1((0,T )× Td ;Rd ), m ≥ 0 a.e.,∫Td m(t, x)dx = 1

a.e. t ∈ (0,T ), and∂tm + div (w) = 0, m(0, ·) = m0(·)

in the sense of distributions.

Theorem (Cardaliaguet, PJG)

The optimal control problems above are in duality, i.e.

infφ∈K0

A(φ) = − min(m,w)∈K1

B(m,w).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 43: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Relaxed problem

K will be defined as the set of all pairs (φ, α) ∈ BV × L1 such thatDφ ∈ Lr ((0,T )× Td), φ(T , ·) ≤ φT in the sense of traces,α+ ∈ Lp((0,T )× Td), φ ∈ L∞((t,T )× Td) for every t ∈ (0,T ), and

−∂tφ+ H(x ,Dφ) ≤ α.

in the sense of distribution. Find

inf(φ,α)∈K

A(φ, α) = inf(φ,α)∈K

∫ T

0

∫Td

F ∗(x , α(t, x))dxdt−∫Td

φ(0, x)m0(x)dx .

Theorem (Cardaliaguet, PJG)

The relaxed problem has a minimizer. Moreover,

inf(φ,α)∈K

A(φ, α) = − min(m,w)∈K1

B(m,w) = infφ∈K0

A(φ).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 44: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Definition of weak solution

A pair (φ,m) ∈ BV ((0,T )× Td)× Lq((0,T )× Td) is called a weaksolution if it satisfies the following conditions.

1 Dφ ∈ Lr and the maps mf (x ,m), mH∗ (x ,−DpH(x ,Dφ)) andmDpH(x ,Dφ) are integrable,

2 φ satisfies −∂tφ+ H(x ,Dφ) ≤ f (x ,m) in the sense of distributions,the boundary condition φ(T , ·) ≤ φT in the sense of trace and thefollowing equality∫∫

m (H(x ,Dφ)− 〈Dφ,DpH(x ,Dφ)〉 − f (x ,m)) dxdt

=

∫(φTm(T )− φ(0)m0)dx

3 m satisfies the continuity equation

∂tm − div (mDpH(x ,Dφ)) = 0 in (0,T )× Td , m(0) = m0

in the sense of distributions.P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 45: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Well-posedness

Theorem (Cardaliaguet, PJG)

(i) If (m,w) ∈ K1 is a minimizer of the dual problem and (φ, α) ∈ K is aminimizer of the relaxed problem, then (φ,m) is a weak solution andα(t, x) = f (x ,m(t, x)) almost everywhere.

(ii) Conversely, if (φ,m) is a weak solution, then there exist functionsw , α such that (φ, α) ∈ K is a minimizer of the relaxed problem and(m,w) ∈ K1 is a minimizer of the dual problem.

(iii) If (φ,m) and (φ′,m′) are both weak solutions, then m = m′ almosteverywhere while φ = φ′ almost everywhere in the set m > 0.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 46: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Main result

Let φf ∈ C 1 on Td .Let (φT ,mT ) be a “good” solution of −∂tφ+ H(x ,Dφ) = f (x ,m)

∂tm − div (mDpH(x ,Dφ)) = 0φ(T , x) = φf (x),m(0, x) = m0(x).

and (λ, φ, m) be a solution of the ergodic MFG system. Define

ψT (s, x) = φT (sT , x), µT (s, x) = mT (sT , x) ∀(s, x) ∈ (0, 1)×Td .

Theorem (Cardaliaguet, PJG)

As T → +∞,

(µT ) converges to m in Lθ((0, 1)× Td) for any θ ∈ [1, q),

ψT/T converges to the map s → λ(1− s) in Lθ((δ, 1)× Td) for anyθ ≥ 1 and any δ ∈ (0, 1).

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations

Page 47: Optimal control of Hamilton-Jacobi-Bellman equations

IntroductionMain results

ProofsFurther results

Well-posedness of weak solutionsLong time average behavior

Thank you!

P. Cardaliaguet and P. Graber, “Mean field games systems of firstorder,” submitted to ESAIM: Control, Optimization, and Calculus ofVariations. Available on the arXiv.

P. Graber, “Optimal control of first-order Hamilton-Jacobi equationswith linearly bounded Hamiltonians,” to appear in AppliedMathematics and Optimization. Also available on the arXiv.

P. Jameson Graber Optimal control of Hamilton-Jacobi-Bellman equations