A Mean Field Game with Interactions for EpidemicModels
Optimal Stochastic Control Approach
Josu DoncelINRIA
joint work with N. Gast and B. Gaujal
Groupe de Travail COS (Contrôle Optimal Stochastique)LAAS-CNRS, Toulouse
September 10, 2015
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 1 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 2 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 3 / 26
Introduction
Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.
Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.
Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass
N players⇒ continuous playersSimplification of games and equilibria in the continuous limit
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26
Introduction
Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.
Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.
Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass
N players⇒ continuous playersSimplification of games and equilibria in the continuous limit
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26
Introduction
Definition (Mean-Field Games)Mathematical models for the study of the behaviour of a very largenumber of rational agents in interaction.
Theorem (Lasry and Lions, 2006)All Nash equilibrium converges as N →∞ to a Mean Field equilibrium.The equilibrium is unique under monotonicity conditions.
Assumptions:A1 Homogeneous playersA2 Individual object action do not affect in the dynamics of the mass
N players⇒ continuous playersSimplification of games and equilibria in the continuous limit
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 4 / 26
Introduction (cont.)
EDP approach to mean-field games: HJB and FP equations
⇒ Optimal stochastic control approach
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 5 / 26
SIRV Model
SIRV dynamics:Susceptible⇒ Infected: if it meets an infected with rate βInfected⇒ Recovered: with rate γSusceptible⇒ Vaccinated: with rate b(t)
Susc Infec Reco
Vac
b(t)
β I(t) γ
Some applications:MedicineBiologyComputer networks: virus and adverts
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 6 / 26
SIRV Model
SIRV dynamics:Susceptible⇒ Infected: if it meets an infected with rate βInfected⇒ Recovered: with rate γSusceptible⇒ Vaccinated: with rate b(t)
Susc Infec Reco
Vac
b(t)
β I(t) γ
Some applications:MedicineBiologyComputer networks: virus and advertsJosu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 6 / 26
SIRV Model
When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)
0 2 4 6 8 10Time
0.0
0.2
0.4
0.6
0.8
1.0
Prop
ortio
n of
Indi
vidu
als
SusceptibleInfectedRecoveryVaccinated
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26
SIRV Model
When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)
0 2 4 6 8 10Time
0.0
0.2
0.4
0.6
0.8
1.0
Prop
ortio
n of
Indi
vidu
als
SusceptibleInfectedRecoveryVaccinated
Classic MFG: interactions given only by the controlMass dynamics depend on the control, Brownian motion...
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26
SIRV Model
When N →∞:S(t) = −β · S(t) · I(t)− b(t) · S(t)I(t) = β · S(t) · I(t)− γ · I(t)R(t) = γ · I(t)V (t) = b(t) · S(t)
0 2 4 6 8 10Time
0.0
0.2
0.4
0.6
0.8
1.0
Prop
ortio
n of
Indi
vidu
als
SusceptibleInfectedRecoveryVaccinated
Classic MFG: interactions given only by the controlMass dynamics depend on the control, Brownian motion...
Our model: mass dynamics depend also on S(t) · I(t)⇒ Mean-Field Game with Interactions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 7 / 26
SIRV Model (cont.)
Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI
Obj: choose b(t) to minimize cost
Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine
Objective:Compare cost of centralized and decentralized vaccination policies
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26
SIRV Model (cont.)
Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI
Obj: choose b(t) to minimize cost
Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine
Objective:Compare cost of centralized and decentralized vaccination policies
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26
SIRV Model (cont.)
Vaccination policy: b(t) ∈ [0,bmax ]Vaccination cost: cVInfection cost: cI
Obj: choose b(t) to minimize cost
Example: HospitalDecentralized⇒ each individual chooses how to vaccinateCentralized⇒ central agent decides when people take medicine
Objective:Compare cost of centralized and decentralized vaccination policies
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 8 / 26
Related work
No much literature of MFG with interactions
(Laguzet and Turinici, 2015)Approximation:P(X (t) = infec) = P(X (t) = infec | no vac) andP(X (t) = vac) = P(X (t) = vac | no infec)
Our solution: No approximation
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 9 / 26
Related work
No much literature of MFG with interactions
(Laguzet and Turinici, 2015)Approximation:P(X (t) = infec) = P(X (t) = infec | no vac) andP(X (t) = vac) = P(X (t) = vac | no infec)
Our solution: No approximation
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 9 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 10 / 26
Decentralized Control
X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)
Susc Infec Reco
Vac
bi(t)
β I(t) γ
Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost
E
(∫ T
0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt
)(1)
Continuous Time Markov Decision Process
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26
Decentralized Control
X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)
Susc Infec Reco
Vac
bi(t)
β I(t) γ
Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost
E
(∫ T
0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt
)(1)
Continuous Time Markov Decision Process
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26
Decentralized Control
X (t) ∈ {S, I,R,V} state of object i ⇒ bi(t)
Susc Infec Reco
Vac
bi(t)
β I(t) γ
Generic player i : given b(t), choose vaccination policy bi(t) tominimize his expected cost
E
(∫ T
0(cV bi(t) P(X (t) = S) + cI P(X (t) = I)) dt
)(1)
Continuous Time Markov Decision Process
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 11 / 26
Decentralized Control (cont.)
PropositionFor any b(t), the solution of (1) is of threshold type
0 2 4 6 8 10Time
1
0
1
2
3
4
Vacc
inati
on R
ate
MDP solutionb(t)
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 12 / 26
Mean-Field Equilibrium
Assumption: Homogeneous individuals⇒ solve (1) equallySymmetric MFE
Definition (Mean-Field Equilibrium):A vaccination policy is a symmetric MFE if and only if it minimizes (1)and it coincides with b(t)
Fixed point problem
Solution of (1) of threshold type⇒ MFE requirements:b(t) of threshold typeThresholds of b(t) and of solution of (1) coincide
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 13 / 26
Mean-Field Equilibrium
Assumption: Homogeneous individuals⇒ solve (1) equallySymmetric MFE
Definition (Mean-Field Equilibrium):A vaccination policy is a symmetric MFE if and only if it minimizes (1)and it coincides with b(t)
Fixed point problem
Solution of (1) of threshold type⇒ MFE requirements:b(t) of threshold typeThresholds of b(t) and of solution of (1) coincide
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 13 / 26
Mean-Field Equilibrium
TheoremThere exists a unique MFE and it is of threshold type.
Sketch of the proof: Monotonicity of MDP equations
time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
beq
(t)
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 14 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 15 / 26
Centralized control
Definition: Global Cost
CGlo(b(t)) =∫ T
0(cV b(t) S(t) + cI I(t))dt
Definition: Social Optimum
bopt(t) = argminb(t)CGlo(b(t))
bopt(t) is of threshold type?
time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
bop
t (t)
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26
Centralized control
Definition: Global Cost
CGlo(b(t)) =∫ T
0(cV b(t) S(t) + cI I(t))dt
Definition: Social Optimum
bopt(t) = argminb(t)CGlo(b(t))
bopt(t) is of threshold type?
time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
bop
t (t)
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26
Centralized control
Definition: Global Cost
CGlo(b(t)) =∫ T
0(cV b(t) S(t) + cI I(t))dt
Definition: Social Optimum
bopt(t) = argminb(t)CGlo(b(t))
bopt(t) is of threshold type?
time0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
bop
t (t)
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 16 / 26
Centralized control (cont.)
TheoremGlobal optimum is of threshold type
Sketch of the proof: Policy improvement
0 0.5 1 1.5 2 2.50
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0 0.5 1 1.5 2 2.50
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Left policy⇒ less cost
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 17 / 26
Centralized control (cont.)
TheoremGlobal optimum is of threshold type
Sketch of the proof: Policy improvement
0 0.5 1 1.5 2 2.50
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0 0.5 1 1.5 2 2.50
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Left policy⇒ less cost
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 17 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 18 / 26
Mechanism Design
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.5
1
1.5
beq(t)bopt(t)
teqtopt
Question:Can we get that CGlo(bopt(t)) = CGlo(beq(t))?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 19 / 26
Mechanism Design
Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))
⇒ Change the model!
Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV
Individual vaccination cost: c′V = cV + p
Example: Hospitalc′
V : price to sell the medicine to each individualcV : medicine production price
p is positive, negative or zero?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26
Mechanism Design
Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))
⇒ Change the model!
Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV
Individual vaccination cost: c′V = cV + p
Example: Hospitalc′
V : price to sell the medicine to each individualcV : medicine production price
p is positive, negative or zero?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26
Mechanism Design
Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))
⇒ Change the model!
Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV
Individual vaccination cost: c′V = cV + p
Example: Hospitalc′
V : price to sell the medicine to each individualcV : medicine production price
p is positive, negative or zero?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26
Mechanism Design
Observation:For a fixed system parameters, except in the trivial cases, teq < topt .Therefore, CGlo(bopt(t)) < CGlo(beq(t))
⇒ Change the model!
Different cost of vaccination for decentralized and centralized problemPopulation vaccination cost: cV
Individual vaccination cost: c′V = cV + p
Example: Hospitalc′
V : price to sell the medicine to each individualcV : medicine production price
p is positive, negative or zero?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 20 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 21 / 26
Dynamics and Thresholds
N° Susceptible
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
N°
Infe
cted
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
OPT: NOEQ: NO
OPT: MAXEQ: MAX
OPT: MAXEQ: NO
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26
Dynamics and Thresholds
N° Susceptible
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
N°
Infe
cted
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
bopt(t) beq(t)
OPT: NOEQ: NO
OPT: MAXEQ: MAX
OPT: MAXEQ: NO
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26
Dynamics and Thresholds
N° Susceptible
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
N°
Infe
cted
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
bopt(t) beq(t)
OPT: NOEQ: NO
OPT: MAXEQ: MAX
OPT: MAXEQ: NO
ConclusionExcept in the trivial cases, teq < topt
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 22 / 26
Varying cV and Pricing Mechanism
cV
0.2 0.4 0.6 0.8 1 1.2
Thr
esho
lds
0
0.05
0.1
0.15
0.2
MFE
OPT
cV
0.2 0.4 0.6 0.8 1 1.2
Cos
t
0.35
0.4
0.45
0.5
0.55
0.6
0.65
MFE
OPT
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26
Varying cV and Pricing Mechanism
cV
0.2 0.4 0.6 0.8 1 1.2
Cos
t
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
p
MFE
OPT
cV
0.2 0.4 0.6 0.8 1 1.2
Thr
esho
lds
0
0.05
0.1
0.15
0.2
MFE
OPT
cV = 0.8 ⇒ p = 0.16 (20% of cV )p: between 0% and 40% of cV
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26
Varying cV and Pricing Mechanism
cV
0.2 0.4 0.6 0.8 1 1.2
Cos
t
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
p
MFE
OPT
cV
0.2 0.4 0.6 0.8 1 1.2
Thr
esho
lds
0
0.05
0.1
0.15
0.2
MFE
OPT
cV = 0.8 ⇒ p = 0.16 (20% of cV )p: between 0% and 40% of cV
Conclusion: p < 0Vaccination to individuals must becheaper!
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 23 / 26
Outline
1 Introduction
2 Decentralized Control
3 Centralized Control
4 Pricing Technique
5 Numerical Experiments
6 Conclusions
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 24 / 26
Conclusions
MFG⇒ Optimal stochastic control
Simple model: interactions and control
Susc Infec Reco
Vac
bi(t)
β I(t) γ
MFE is unique and of threshold type, as well as the global optimum
Pricing mechanism
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26
Conclusions
MFG⇒ Optimal stochastic control
Simple model: interactions and control
Susc Infec Reco
Vac
bi(t)
β I(t) γ
MFE is unique and of threshold type, as well as the global optimum
Pricing mechanism
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26
Conclusions
MFG⇒ Optimal stochastic control
Simple model: interactions and control
Susc Infec Reco
Vac
bi(t)
β I(t) γ
MFE is unique and of threshold type, as well as the global optimum
Pricing mechanism
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 25 / 26
Thank you
Questions?
Josu Doncel (INRIA) MFG with Interactions for Epidemic Models 10 Sept 2015 26 / 26