A Mean Field Game with Interactions for EpidemicModels

Optimal Stochastic Control Approach

Josu DoncelINRIA

joint work with N. Gast and B. Gaujal

Groupe de Travail COS (Contrôle Optimal Stochastique)LAAS-CNRS, Toulouse

September 10, 2015

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 1 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 2 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 3 / 26

Introduction

Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.

Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.

Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass

N players⇒ continuous playersSimplification of games and equilibria in the continuous limit

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26

Introduction

Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.

Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.

Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass

N players⇒ continuous playersSimplification of games and equilibria in the continuous limit

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26

Introduction

Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.

Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.

Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass

N players⇒ continuous playersSimplification of games and equilibria in the continuous limit

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26

Introduction (cont.)

EDP approach to mean-field games: HJB and FP equations

⇒ Optimal stochastic control approach

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 5 / 26

SIRV Model

SIRV dynamics:Susceptible⇒ Infected: if it meets an infected with rate βInfected⇒ Recovered: with rate γSusceptible⇒ Vaccinated: with rate b(t)

Susc Infec Reco

Vac

b(t)

β I(t) γ

Some applications:MedicineBiologyComputer networks: virus and adverts

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 6 / 26

SIRV Model

SIRV dynamics:Susceptible⇒ Infected: if it meets an infected with rate βInfected⇒ Recovered: with rate γSusceptible⇒ Vaccinated: with rate b(t)

Susc Infec Reco

Vac

b(t)

β I(t) γ

Some applications:MedicineBiologyComputer networks: virus and advertsJosu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 6 / 26

SIRV Model

When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)

0 2 4 6 8 10Time

0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of

Indi

vidu

als

SusceptibleInfectedRecoveryVaccinated

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26

SIRV Model

When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)

0 2 4 6 8 10Time

0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of

Indi

vidu

als

SusceptibleInfectedRecoveryVaccinated

Classic MFG: interactions given only by the controlMass dynamics depend on the control, Brownian motion...

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26

SIRV Model

When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)

0 2 4 6 8 10Time

0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of

Indi

vidu

als

SusceptibleInfectedRecoveryVaccinated

Classic MFG: interactions given only by the controlMass dynamics depend on the control, Brownian motion...

Our model: mass dynamics depend also on S(t) · I(t)⇒ Mean-Field Game with Interactions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26

SIRV Model (cont.)

Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI

Obj: choose b(t) to minimize cost

Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine

Objective:Compare cost of centralized and decentralized vaccination policies

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26

SIRV Model (cont.)

Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI

Obj: choose b(t) to minimize cost

Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine

Objective:Compare cost of centralized and decentralized vaccination policies

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26

SIRV Model (cont.)

Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI

Obj: choose b(t) to minimize cost

Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine

Objective:Compare cost of centralized and decentralized vaccination policies

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26

Related work

No much literature of MFG with interactions

(Laguzet and Turinici, 2015)Approximation:P(X (t) = infec) = P(X (t) = infec | no vac) andP(X (t) = vac) = P(X (t) = vac | no infec)

Our solution: No approximation

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 9 / 26

Related work

No much literature of MFG with interactions

(Laguzet and Turinici, 2015)Approximation:P(X (t) = infec) = P(X (t) = infec | no vac) andP(X (t) = vac) = P(X (t) = vac | no infec)

Our solution: No approximation

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 9 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 10 / 26

Decentralized Control

X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)

Susc Infec Reco

Vac

bi(t)

β I(t) γ

Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost

E

(∫ T

0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt

)(1)

Continuous Time Markov Decision Process

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26

Decentralized Control

X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)

Susc Infec Reco

Vac

bi(t)

β I(t) γ

Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost

E

(∫ T

0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt

)(1)

Continuous Time Markov Decision Process

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26

Decentralized Control

X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)

Susc Infec Reco

Vac

bi(t)

β I(t) γ

Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost

E

(∫ T

0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt

)(1)

Continuous Time Markov Decision Process

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26

Decentralized Control (cont.)

PropositionFor any b(t), the solution of (1) is of threshold type

0 2 4 6 8 10Time

1

0

1

2

3

4

Vacc

inati

on R

ate

MDP solutionb(t)

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 12 / 26

Mean-Field Equilibrium

Assumption: Homogeneous individuals⇒ solve (1) equallySymmetric MFE

Definition (Mean-Field Equilibrium):A vaccination policy is a symmetric MFE if and only if it minimizes (1)and it coincides with b(t)

Fixed point problem

Solution of (1) of threshold type⇒ MFE requirements:b(t) of threshold typeThresholds of b(t) and of solution of (1) coincide

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 13 / 26

Mean-Field Equilibrium

Assumption: Homogeneous individuals⇒ solve (1) equallySymmetric MFE

Definition (Mean-Field Equilibrium):A vaccination policy is a symmetric MFE if and only if it minimizes (1)and it coincides with b(t)

Fixed point problem

Solution of (1) of threshold type⇒ MFE requirements:b(t) of threshold typeThresholds of b(t) and of solution of (1) coincide

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 13 / 26

Mean-Field Equilibrium

TheoremThere exists a unique MFE and it is of threshold type.

Sketch of the proof: Monotonicity of MDP equations

time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

beq

(t)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 14 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 15 / 26

Centralized control

Definition: Global Cost

CGlo(b(t)) =∫ T

0(cV b(t) S(t) + cI I(t))dt

Definition: Social Optimum

bopt(t) = argminb(t)CGlo(b(t))

bopt(t) is of threshold type?

time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

bop

t (t)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26

Centralized control

Definition: Global Cost

CGlo(b(t)) =∫ T

0(cV b(t) S(t) + cI I(t))dt

Definition: Social Optimum

bopt(t) = argminb(t)CGlo(b(t))

bopt(t) is of threshold type?

time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

bop

t (t)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26

Centralized control

Definition: Global Cost

CGlo(b(t)) =∫ T

0(cV b(t) S(t) + cI I(t))dt

Definition: Social Optimum

bopt(t) = argminb(t)CGlo(b(t))

bopt(t) is of threshold type?

time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

bop

t (t)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26

Centralized control (cont.)

TheoremGlobal optimum is of threshold type

Sketch of the proof: Policy improvement

0 0.5 1 1.5 2 2.50

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0 0.5 1 1.5 2 2.50

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Left policy⇒ less cost

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 17 / 26

Centralized control (cont.)

TheoremGlobal optimum is of threshold type

Sketch of the proof: Policy improvement

0 0.5 1 1.5 2 2.50

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0 0.5 1 1.5 2 2.50

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Left policy⇒ less cost

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 17 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 18 / 26

Mechanism Design

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.5

1

1.5

beq(t)bopt(t)

teqtopt

Question:Can we get that CGlo(bopt(t)) = CGlo(beq(t))?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 19 / 26

Mechanism Design

Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))

⇒ Change the model!

Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV

Individual vaccination cost: c′V = cV + p

Example: Hospitalc′

V : price to sell the medicine to each individualcV : medicine production price

p is positive, negative or zero?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26

Mechanism Design

Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))

⇒ Change the model!

Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV

Individual vaccination cost: c′V = cV + p

Example: Hospitalc′

V : price to sell the medicine to each individualcV : medicine production price

p is positive, negative or zero?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26

Mechanism Design

Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))

⇒ Change the model!

Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV

Individual vaccination cost: c′V = cV + p

Example: Hospitalc′

V : price to sell the medicine to each individualcV : medicine production price

p is positive, negative or zero?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26

Mechanism Design

Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))

⇒ Change the model!

Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV

Individual vaccination cost: c′V = cV + p

Example: Hospitalc′

V : price to sell the medicine to each individualcV : medicine production price

p is positive, negative or zero?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 21 / 26

Dynamics and Thresholds

N° Susceptible

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

N°

Infe

cted

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

OPT: NOEQ: NO

OPT: MAXEQ: MAX

OPT: MAXEQ: NO

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26

Dynamics and Thresholds

N° Susceptible

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

N°

Infe

cted

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

bopt(t) beq(t)

OPT: NOEQ: NO

OPT: MAXEQ: MAX

OPT: MAXEQ: NO

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26

Dynamics and Thresholds

N° Susceptible

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

N°

Infe

cted

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

bopt(t) beq(t)

OPT: NOEQ: NO

OPT: MAXEQ: MAX

OPT: MAXEQ: NO

ConclusionExcept in the trivial cases, teq < topt

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26

Varying cV and Pricing Mechanism

cV

0.2 0.4 0.6 0.8 1 1.2

Thr

esho

lds

0

0.05

0.1

0.15

0.2

MFE

OPT

cV

0.2 0.4 0.6 0.8 1 1.2

Cos

t

0.35

0.4

0.45

0.5

0.55

0.6

0.65

MFE

OPT

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26

Varying cV and Pricing Mechanism

cV

0.2 0.4 0.6 0.8 1 1.2

Cos

t

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

p

MFE

OPT

cV

0.2 0.4 0.6 0.8 1 1.2

Thr

esho

lds

0

0.05

0.1

0.15

0.2

MFE

OPT

cV = 0.8 ⇒ p = 0.16 (20% of cV )p: between 0% and 40% of cV

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26

Varying cV and Pricing Mechanism

cV

0.2 0.4 0.6 0.8 1 1.2

Cos

t

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

p

MFE

OPT

cV

0.2 0.4 0.6 0.8 1 1.2

Thr

esho

lds

0

0.05

0.1

0.15

0.2

MFE

OPT

cV = 0.8 ⇒ p = 0.16 (20% of cV )p: between 0% and 40% of cV

Conclusion: p < 0Vaccination to individuals must becheaper!

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26

Outline

1 Introduction

2 Decentralized Control

3 Centralized Control

4 Pricing Technique

5 Numerical Experiments

6 Conclusions

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 24 / 26

Conclusions

MFG⇒ Optimal stochastic control

Simple model: interactions and control

Susc Infec Reco

Vac

bi(t)

β I(t) γ

MFE is unique and of threshold type, as well as the global optimum

Pricing mechanism

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26

Conclusions

MFG⇒ Optimal stochastic control

Simple model: interactions and control

Susc Infec Reco

Vac

bi(t)

β I(t) γ

MFE is unique and of threshold type, as well as the global optimum

Pricing mechanism

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26

Conclusions

MFG⇒ Optimal stochastic control

Simple model: interactions and control

Susc Infec Reco

Vac

bi(t)

β I(t) γ

MFE is unique and of threshold type, as well as the global optimum

Pricing mechanism

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26

Thank you

Questions?

Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 26 / 26