Population Games and Evolutionary Dynamics

William H. Sandholm

September 9, 2009

1

3

4

Chaos under the replicator dynamic

ii

CONTENTS IN BRIEF

Preface xv

1 Introduction 1

I Population Games 19

2 Population Games 21

3 Potential Games, Stable Games, and Supermodular Games 51

II Deterministic Evolutionary Dynamics 121

4 Revision Protocols and Evolutionary Dynamics 123

5 Deterministic Dynamics: Families and Properties 143

6 Best Response and Projection Dynamics 181

III Convergence and Nonconvergence of Deterministic Dynamics 225

7 Global Convergence of Evolutionary Dynamics 227

8 Local Stability under Evolutionary Dynamics 279

9 Nonconvergence of Evolutionary Dynamics 329

iii

IV Stochastic Evolutionary Models 375

10 Stochastic Evolution and Deterministic Approximation 377

11 Stationary Distributions and Infinite Horizon Behavior 409

12 Limiting Stationary Distributions and Stochastic Stability 465

Bibliography 557

iv

CONTENTS

Preface xv

1 Introduction 11.1 Population Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Modeling Interactions in Large Populations . . . . . . . . . . . . . . . 21.1.2 Definitions and Classes of Population Games . . . . . . . . . . . . . . 5

1.2 Evolutionary Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.1 Knowledge, Rationality, and Large Games . . . . . . . . . . . . . . . 71.2.2 Foundations for Evolutionary Dynamics . . . . . . . . . . . . . . . . 81.2.3 Deterministic Evolutionary Dynamics . . . . . . . . . . . . . . . . . . 101.2.4 Orders of Limits for Stochastic Evolutionary Models . . . . . . . . . 121.2.5 Stationary Distributions and Stochastic Stability . . . . . . . . . . . . 12

1.3 Remarks on History, Motivation, and Interpretation . . . . . . . . . . . . . . 131.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

I Population Games 19

2 Population Games 212.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.1 Population Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.1.1 Populations, Strategies, and States . . . . . . . . . . . . . . . . . . . . 222.1.2 Payoffs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.1.3 Best Responses and Nash Equilibria . . . . . . . . . . . . . . . . . . . 232.1.4 Prelude to Evolutionary Dynamics . . . . . . . . . . . . . . . . . . . . 24

2.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.2.1 Random Matching in Normal Form Games . . . . . . . . . . . . . . . 24

v

2.2.2 Congestion Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.2.3 Two Simple Externality Models . . . . . . . . . . . . . . . . . . . . . . 28

2.3 The Geometry of Population Games and Nash Equilibria . . . . . . . . . . . 282.3.1 Drawing Two-Strategy Games . . . . . . . . . . . . . . . . . . . . . . 292.3.2 Displacement Vectors and Tangent Spaces . . . . . . . . . . . . . . . 302.3.3 Orthogonal Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.3.4 Drawing Three-Strategy Games . . . . . . . . . . . . . . . . . . . . . 352.3.5 Tangent Cones and Normal Cones . . . . . . . . . . . . . . . . . . . . 372.3.6 Normal Cones and Nash Equilibria . . . . . . . . . . . . . . . . . . . 40

2.A Affine Spaces, Tangent Spaces, and Projections . . . . . . . . . . . . . . . . . 432.A.1 Affine Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432.A.2 Affine Hulls of Convex Sets . . . . . . . . . . . . . . . . . . . . . . . . 442.A.3 Orthogonal Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . 452.A.4 The Moreau Decomposition Theorem . . . . . . . . . . . . . . . . . . 49

2.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3 Potential Games, Stable Games, and Supermodular Games 513.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513.1 Full Potential Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

3.1.1 Full Population Games . . . . . . . . . . . . . . . . . . . . . . . . . . . 523.1.2 Definition and Characterization . . . . . . . . . . . . . . . . . . . . . 523.1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.1.4 Nash Equilibria of Full Potential Games . . . . . . . . . . . . . . . . . 573.1.5 The Geometry of Nash Equilibrium in Full Potential Games . . . . . 633.1.6 Efficiency in Homogeneous Full Potential Games . . . . . . . . . . . 653.1.7 Inefficiency Bounds for Congestion Games . . . . . . . . . . . . . . . 67

3.2 Potential Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.2.1 Motivating Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.2.2 Definition, Characterizations, and Examples . . . . . . . . . . . . . . 723.2.3 Potential Games and Full Potential Games . . . . . . . . . . . . . . . 753.2.4 Passive Games and Constant Games . . . . . . . . . . . . . . . . . . . 77

3.3 Stable Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793.3.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813.3.3 Invasion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 853.3.4 Global Neutral Stability and Global Evolutionary Stability . . . . . . 863.3.5 Nash Equilibrium and Global Neutral Stability in Stable Games . . . 90

vi

3.4 Supermodular Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943.4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943.4.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 973.4.3 Best Response Monotonicity in Supermodular Games . . . . . . . . . 993.4.4 Nash Equilibria of Supermodular Games . . . . . . . . . . . . . . . . 100

3.A Multivariate Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1013.A.1 Univariate Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1013.A.2 The Derivative as a Linear Map . . . . . . . . . . . . . . . . . . . . . . 1013.A.3 Differentiation as a Linear Operation . . . . . . . . . . . . . . . . . . . 1043.A.4 The Product Rule and the Chain Rule . . . . . . . . . . . . . . . . . . 1043.A.5 Homogeneity and Euler’s Theorem . . . . . . . . . . . . . . . . . . . 1063.A.6 Higher Order Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 1073.A.7 The Whitney Extension Theorem . . . . . . . . . . . . . . . . . . . . . 1083.A.8 Vector Integration and the Fundamental Theorem of Calculus . . . . 1083.A.9 Potential Functions and Integrability . . . . . . . . . . . . . . . . . . . 109

3.B Affine Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1103.B.1 Linear Forms and the Riesz Representation Theorem . . . . . . . . . 1103.B.2 Dual Characterizations of Multiples of Linear Forms . . . . . . . . . 1113.B.3 Derivatives of Functions on Affine Spaces . . . . . . . . . . . . . . . . 1133.B.4 Affine Integrability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

3.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

II Deterministic Evolutionary Dynamics 121

4 Revision Protocols and Evolutionary Dynamics 1234.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1234.1 Revision Protocols and Mean Dynamics . . . . . . . . . . . . . . . . . . . . . 124

4.1.1 Revision Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1244.1.2 Mean Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1264.1.3 Target Protocols and Target Dynamics . . . . . . . . . . . . . . . . . . 128

4.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1294.3 Evolutionary Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1334.A Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . 135

4.A.1 Basic Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1354.A.2 Existence, Uniqueness, and Continuity of Solutions . . . . . . . . . . 1364.A.3 Ordinary Differential Equations on Compact Convex Sets . . . . . . 139

vii

4.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

5 Deterministic Dynamics: Families and Properties 1435.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1435.1 Principles for Evolutionary Modeling . . . . . . . . . . . . . . . . . . . . . . 1455.2 Desiderata for Revision Protocols and Evolutionary Dynamics . . . . . . . . 146

5.2.1 Limited Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1465.2.2 Incentives and Aggregate Behavior . . . . . . . . . . . . . . . . . . . 148

5.3 Families of Evolutionary Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 1525.4 Imitative Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

5.4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1565.4.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1575.4.3 Biological Derivations of the Replicator Dynamic . . . . . . . . . . . 1625.4.4 Extinction and Invariance . . . . . . . . . . . . . . . . . . . . . . . . . 1645.4.5 Monotone Percentage Growth Rates and Positive Correlation . . . . 1655.4.6 Rest Points and Restricted Equilibria . . . . . . . . . . . . . . . . . . . 168

5.5 Excess Payoff Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1695.5.1 Definition and Interpretation . . . . . . . . . . . . . . . . . . . . . . . 1695.5.2 Incentives and Aggregate Behavior . . . . . . . . . . . . . . . . . . . 171

5.6 Pairwise Comparison Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . 1735.6.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1735.6.2 Incentives and Aggregate Behavior . . . . . . . . . . . . . . . . . . . 1745.6.3 Desiderata Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

5.7 Multiple Revision Protocols and Combined Dynamics . . . . . . . . . . . . . 1765.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

6 Best Response and Projection Dynamics 1816.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1816.1 The Best Response Dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

6.1.1 Definition and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 1826.1.2 Construction and Properties of Solution Trajectories . . . . . . . . . . 1846.1.3 Incentive Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

6.2 Perturbed Best Response Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 1906.2.1 Revision Protocols and Mean Dynamics . . . . . . . . . . . . . . . . . 1916.2.2 Perturbed Optimization: A Representation Theorem . . . . . . . . . 1926.2.3 Logit Choice and the Logit Dynamic . . . . . . . . . . . . . . . . . . . 1956.2.4 Perturbed Incentive Properties via Virtual Payoffs . . . . . . . . . . . 200

viii

6.3 The Projection Dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2026.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2026.3.2 Solution Trajectories . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2036.3.3 Incentive Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2066.3.4 Revision Protocols and Connections with the Replicator Dynamic . . 206

6.A Differential Inclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2096.A.1 Basic Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2096.A.2 Differential Equations Defined by Projections . . . . . . . . . . . . . . 211

6.B The Legendre Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2126.B.1 Legendre Transforms of Functions on Open Intervals . . . . . . . . . 2136.B.2 Legendre Transforms of Functions on Multidimensional Domains . . 216

6.C Perturbed Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2176.C.1 Proof of the Representation Theorem . . . . . . . . . . . . . . . . . . 2176.C.2 Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

6.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221

III Convergence and Nonconvergence of Deterministic Dynamics 225

7 Global Convergence of Evolutionary Dynamics 2277.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2277.1 Potential Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

7.1.1 Potential Functions as Lyapunov Functions . . . . . . . . . . . . . . . 2297.1.2 Gradient Systems for Potential Games . . . . . . . . . . . . . . . . . . 235

7.2 Stable Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2397.2.1 The Projection and Replicator Dynamics in Strictly Stable Games . . 2397.2.2 Integrable Target Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 2427.2.3 Impartial Pairwise Comparison Dynamics . . . . . . . . . . . . . . . 2537.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

7.3 Supermodular Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2567.3.1 The Best Response Dynamic in Two-Player Normal Form Games . . 2587.3.2 Stochastically Perturbed Best Response Dynamics . . . . . . . . . . . 261

7.4 Dominance Solvable Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2657.4.1 Dominated and Iteratively Dominated Strategies . . . . . . . . . . . 2667.4.2 The Best Response Dynamic . . . . . . . . . . . . . . . . . . . . . . . . 2667.4.3 Imitative Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267

7.A Limit and Stability Notions for Deterministic Dynamics . . . . . . . . . . . . 269

ix

7.A.1 ω-Limits and Notions of Recurrence . . . . . . . . . . . . . . . . . . . 2697.A.2 Stability of Sets of States . . . . . . . . . . . . . . . . . . . . . . . . . . 270

7.B Stability Analysis via Lyapunov Functions . . . . . . . . . . . . . . . . . . . 2717.B.1 Lyapunov Stable Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2717.B.2 ω-Limits and Attracting Sets . . . . . . . . . . . . . . . . . . . . . . . 2727.B.3 Asymptotically Stable and Globally Asymptotically Stable Sets . . . 274

7.C Cooperative Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . 2747.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

8 Local Stability under Evolutionary Dynamics 2798.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2798.1 Non-Nash Rest Points of Imitative Dynamics . . . . . . . . . . . . . . . . . . 2808.2 Local Stability in Potential Games . . . . . . . . . . . . . . . . . . . . . . . . . 2818.3 Evolutionarily Stable States . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

8.3.1 Single-Population Games . . . . . . . . . . . . . . . . . . . . . . . . . 2848.3.2 Multipopulation Games . . . . . . . . . . . . . . . . . . . . . . . . . . 2898.3.3 Regular Taylor ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290

8.4 Local Stability via Lyapunov Functions . . . . . . . . . . . . . . . . . . . . . 2918.4.1 The Replicator and Projection Dynamics . . . . . . . . . . . . . . . . 2918.4.2 Target and Pairwise Comparison Dynamics: Interior ESS . . . . . . . 2928.4.3 Target and Pairwise Comparison Dynamics: Boundary ESS . . . . . 294

8.5 Linearization of Imitative Dynamics . . . . . . . . . . . . . . . . . . . . . . . 2988.5.1 The Replicator Dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . 2998.5.2 General Imitative Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 303

8.6 Linearization of Perturbed Best Response Dynamics . . . . . . . . . . . . . . 3058.6.1 Deterministically Perturbed Best Response Dynamics . . . . . . . . . 3058.6.2 The Logit Dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306

8.A Matrix Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3088.A.1 Rank and Invertibility . . . . . . . . . . . . . . . . . . . . . . . . . . . 3088.A.2 Eigenvectors and Eigenvalues . . . . . . . . . . . . . . . . . . . . . . 3098.A.3 Similarity, (Block) Diagonalization, and the Spectral Theorem . . . . 3118.A.4 Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3138.A.5 The Real Jordan Canonical Form . . . . . . . . . . . . . . . . . . . . . 3148.A.6 The Spectral Norm and Singular Values . . . . . . . . . . . . . . . . . 3158.A.7 Hines’s Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316

8.B Linear Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 3178.B.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318

x

8.B.2 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3208.B.3 Stability and Hyperbolicity . . . . . . . . . . . . . . . . . . . . . . . . 321

8.C Linearization of Nonlinear Differential Equations . . . . . . . . . . . . . . . 3238.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325

9 Nonconvergence of Evolutionary Dynamics 3299.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3299.1 Conservative Properties of Evolutionary Dynamics . . . . . . . . . . . . . . 330

9.1.1 Constants of Motion in Null Stable Games . . . . . . . . . . . . . . . 3309.1.2 Preservation of Volume . . . . . . . . . . . . . . . . . . . . . . . . . . 335

9.2 Games with Nonconvergent Evolutionary Dynamics . . . . . . . . . . . . . 3379.2.1 Circulant Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3389.2.2 Continuation of Attractors for Parameterized Games . . . . . . . . . 3419.2.3 Mismatching Pennies . . . . . . . . . . . . . . . . . . . . . . . . . . . 3449.2.4 The Hypnodisk Game . . . . . . . . . . . . . . . . . . . . . . . . . . . 348

9.3 Chaotic Evolutionary Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . 3529.4 Survival of Dominated Strategies . . . . . . . . . . . . . . . . . . . . . . . . . 3559.A Three Classical Theorems on Nonconvergent Dynamics . . . . . . . . . . . . 365

9.A.1 Liouville’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3659.A.2 The Poincare-Bendixson and Bendixson-Dulac Theorems . . . . . . . 368

9.B Attractors and Continuation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3699.B.1 Attractors and Repellors . . . . . . . . . . . . . . . . . . . . . . . . . . 3699.B.2 Continuation of Attractors . . . . . . . . . . . . . . . . . . . . . . . . . 371

9.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372

IV Stochastic Evolutionary Models 375

10 Stochastic Evolution and Deterministic Approximation 37710.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37710.1 The Stochastic Evolutionary Process . . . . . . . . . . . . . . . . . . . . . . . 37810.2 Finite Horizon Deterministic Approximation . . . . . . . . . . . . . . . . . . 380

10.2.1 Kurtz’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38010.2.2 Deterministic Approximation of the Stochastic Evolutionary Process 383

10.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38610.3.1 Discrete-Time Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 38610.3.2 Finite-Population Adjustments . . . . . . . . . . . . . . . . . . . . . . 387

xi

10.A The Exponential and Poisson Distributions . . . . . . . . . . . . . . . . . . . 38810.A.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38810.A.2 The Poisson Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . 391

10.B Probability Models and their Interpretation . . . . . . . . . . . . . . . . . . . 39310.B.1 Countable Probability Models . . . . . . . . . . . . . . . . . . . . . . 39310.B.2 Uncountable Probability Models and Measure Theory . . . . . . . . 39410.B.3 Distributional Properties and Sample Path Properties . . . . . . . . . 396

10.C Countable State Markov Chains and Processes . . . . . . . . . . . . . . . . . 40010.C.1 Countable State Markov Chains . . . . . . . . . . . . . . . . . . . . . 40010.C.2 Countable State Markov Processes: Definition and Construction . . . 40110.C.3 Countable State Markov Processes: Transition Probabilities . . . . . 404

10.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406

11 Stationary Distributions and Infinite Horizon Behavior 40911.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40911.1 Irreducibile Evolutionary Processes . . . . . . . . . . . . . . . . . . . . . . . . 411

11.1.1 Full Support Revision Protocols . . . . . . . . . . . . . . . . . . . . . 41111.1.2 Stationary Distributions and Infinite Horizon Behavior . . . . . . . . 41411.1.3 Reversibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415

11.2 Stationary Distributions for Two-Strategy Games . . . . . . . . . . . . . . . . 41511.2.1 Birth and Death Processes . . . . . . . . . . . . . . . . . . . . . . . . . 41611.2.2 The Stationary Distribution of the Evolutionary Process . . . . . . . . 41911.2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420

11.3 Waiting Times and Infinite Horizon Prediction . . . . . . . . . . . . . . . . . 42511.3.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42611.3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429

11.4 Model Adjustments for Finite Populations . . . . . . . . . . . . . . . . . . . . 43111.4.1 Finite-Population Games . . . . . . . . . . . . . . . . . . . . . . . . . 43111.4.2 Clever Payoff Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 43211.4.3 Committed Agents and Imitative Protocols . . . . . . . . . . . . . . . 434

11.5 Exponential Protocols and Potential Games . . . . . . . . . . . . . . . . . . . 43711.5.1 Finite-Population Potential Games . . . . . . . . . . . . . . . . . . . . 43711.5.2 Exponential Revision Protocols . . . . . . . . . . . . . . . . . . . . . . 44111.5.3 Reversibility and Stationary Distributions . . . . . . . . . . . . . . . . 443

11.A Long Run Behavior of Markov Chains and Processes . . . . . . . . . . . . . 44711.A.1 Communication, Recurrence, and Irreducibility . . . . . . . . . . . . 44711.A.2 Periodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449

xii

11.A.3 Hitting Times and Hitting Probabilities . . . . . . . . . . . . . . . . . 45011.A.4 The Perron-Frobenius Theorem . . . . . . . . . . . . . . . . . . . . . . 45411.A.5 Stationary Distributions for Markov Chains . . . . . . . . . . . . . . . 45411.A.6 Reversible Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . 45611.A.7 Stationary Distributions and Reversibility for Markov Processes . . . 45711.A.8 Convergence in Distribution . . . . . . . . . . . . . . . . . . . . . . . 45911.A.9 Ergodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462

11.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463

12 Limiting Stationary Distributions and Stochastic Stability 46512.0 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46512.1 Definitions of Stochastic Stability . . . . . . . . . . . . . . . . . . . . . . . . . 468

12.1.1 Small Noise Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46812.1.2 Large Population Limits . . . . . . . . . . . . . . . . . . . . . . . . . . 46912.1.3 Double Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47212.1.4 Double Limits: A Counterexample . . . . . . . . . . . . . . . . . . . . 474

12.2 Exponential Protocols and Potential Games . . . . . . . . . . . . . . . . . . . 47712.2.1 Direct Exponential Protocols: The Small Noise Limit . . . . . . . . . 47812.2.2 Direct Exponential Protocols: The Large Population Limit . . . . . . 48012.2.3 Direct Exponential Protocols: Double Limits . . . . . . . . . . . . . . 48612.2.4 Imitative Exponential Protocols with Committed Agents . . . . . . . 487

12.3 Two-Strategy Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48912.3.1 Noisy Best Response Protocols and their Cost Functions . . . . . . . 48912.3.2 The Small Noise Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . 49212.3.3 The Large Population Limit . . . . . . . . . . . . . . . . . . . . . . . . 49412.3.4 Double Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49512.3.5 Stochastic Stability: Examples . . . . . . . . . . . . . . . . . . . . . . . 49812.3.6 Risk Dominance, Stochastic Dominance, and Stochastic Stability . . 50112.3.7 Imitative Protocols with Mutations . . . . . . . . . . . . . . . . . . . . 50612.3.8 Imitative Protocols with Committed Agents . . . . . . . . . . . . . . 509

12.4 Small Noise Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51112.4.1 Noisy Best Response Protocols and Cost Functions . . . . . . . . . . 51112.4.2 Limiting Stationary Distributions via Trees . . . . . . . . . . . . . . . 51412.4.3 Two-Strategy Games and Risk Dominance . . . . . . . . . . . . . . . 51612.4.4 The Radius-Coradius Theorem . . . . . . . . . . . . . . . . . . . . . . 52212.4.5 Half Dominance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524

12.5 Large Population Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527

xiii

12.5.1 Convergence to Recurrent States of the Mean Dynamic . . . . . . . . 52812.5.2 Convergence to Stable Rest Points of the Mean Dynamic . . . . . . . 530

12.A Trees, Escape Paths, and Stochastic Stability . . . . . . . . . . . . . . . . . . . 53212.A.1 The Markov Chain Tree Theorem . . . . . . . . . . . . . . . . . . . . . 53312.A.2 Limiting Stationary Distributions via Trees . . . . . . . . . . . . . . . 53412.A.3 Limiting Stationary Distributions via Trees on Recurrent Classes . . 53712.A.4 Radius-Coradius Theorems . . . . . . . . . . . . . . . . . . . . . . . . 54012.A.5 Lenient Transition Costs and Weak Stochastic Stability . . . . . . . . 542

12.B Stochastic Approximation Theory . . . . . . . . . . . . . . . . . . . . . . . . . 54512.B.1 Convergence to the Birkhoff Center . . . . . . . . . . . . . . . . . . . 54512.B.2 Sufficient Conditions for Convergence to Stable Rest Points . . . . . 547

12.N Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551

Bibliography 557

xiv

CHAPTER

ONE

Introduction

This book describes an approach to modeling recurring strategic interactions in largepopulations of small, anonymous agents. The approach is built upon two basic elements.The first, called a population game, describes the strategic interaction that is to occur re-peatedly. The second, called a revision protocol, specifies the myopic procedure that agentsemploy to decide when and how to choose new strategies. Starting with a populationgame and a revision protocol, one can derive dynamic processes, both deterministic andstochastic, that describe how the agents’ aggregate behavior changes over time. Theseprocesses are known as evolutionary game dynamics.

This introductory chapter begins the work of adding substance to this austere accountof evolutionary game theory, providing motivations for and overviews of the analyses tocome. For the most part, the chapter is written with an eye toward modeling in economicsand other social sciences, though it also discusses the biological origins of the field. Butthese perspectives should not be viewed as constraints, as the methods presented inthis book have ready applications in other disciplines that require models of interactingpopulations of humans, animals, or machines.

Section 1.1 introduces the notion of a population game by presenting applications,offering informal definitions, and discussing connections with normal form games. Itthen previews our treatment of population games in Chapters 2–3. Section 1.2 describesdynamic models of behavior in recurrent play of population games, and contrasts thisdynamic approach with the equilibrium approach traditionally used in game theory. Thissection also offers an overview of our presentation of evolutionary dynamics in Chapters4–12. Section 1.3 concludes with some remarks on motivations for and interpretations ofevolutionary game theory. References relevant to the discussions in the text can be foundin the Notes at the end of the chapter, where the references from the Preface can also befound.

1

1.1 Population Games

[O]nly after the theory for moderate numbers of participants has been satisfactorilydeveloped will it be possible to decide whether extremely great numbers of partic-ipants simplify the situation. . . We share the hope. . . that such simplifications willindeed occur. . .

von Neumann and Morgenstern (1944, p. 14)

We shall now take up the “mass-action” interpretation of equilibrium points. . . It isunnecessary to assume that the participants have full knowledge of the total structureof the game, or the ability and inclination to go through any complex reasoningprocesses. But the participants are supposed to accumulate empirical information onthe relative advantages of the various pure strategies at their disposal.

To be more detailed, we assume that there is a population (in the sense of statistics)of participants for each position of the game. Let us also assume that the “averageplaying” of the game involves n participants selected at random from the n popula-tions, and that there is a stable average frequency with which each pure strategy isemployed by the “average member” of the appropriate population.

Nash (1950b, p. 21)

There are many situations, however, in which an individual is, in effect, competing notagainst an individual opponent but against the population as a whole, or some sectionof it. Such cases can loosely be described as “playing the field”. . . [S]uch contestsagainst the field are probably more widespread and important than pairwise contests.

Maynard Smith (1982, p. 23)

1.1.1 Modeling Interactions in Large Populations

One can imagine many economic, social, and technological environments in whichlarge collections of small agents make strategically interdependent decisions:

• Network congestion. Drivers commute over a highway network. The delay eachdriver experiences depends not only on the route he selects, but also on the con-gestion created by other agents along this route.• Public goods and externalities. A local government maintains a collection of public

recreation facilities. The benefit that a family obtains from using a facility dependson the quality of the facility and the number of other families that use it.• Industrial organization. Software developers choose whether to copyright their

products or make them freely available under a public license; they can also chooseto work within an existing open-source framework. The latter options entail a lossof control, but allow products to improve through the accumulation of uncoordi-nated individual efforts.

2

• The emergence of conventions, norms, and institutions. Firms in a developingeconomy choose among various business practices: whether to accept credit orrequire cash, whether to fight or acquiesce to corrupt officials, whether to rewardmerit or engage in nepotism. Through historical precedent and individual de-cisions, conventions about business conduct are formed. Whether or not theseconventions are efficient, they enable firms to form accurate expectations abouthow trading partners will act.• Cultural integration and assimilation. The behavior of immigrants settling in a

new country is influenced by traditions imported from their home country, and, tothe extent that interactions with the incumbent population require coordination,by the practices of these incumbents as well. At the same time, the choices ofthe incumbents are influenced by the need to coordinate with the immigrants. Theinterplay of these forces determines how behavior in the society as a whole evolves.• Language and communication. Agents without a common language interact re-

peatedly, attempting to communicate their intentions to the others on each occa-sion. Whether these attempts are ultimately understood, and whether they enablethe agents to coordinate on mutually beneficial behavior, depends on meaningsdetermined by the aggregate communication pattern.• Task allocation and decentralized control. The employees of a large firm provide

services to customers in a number of distinct locations. To take advantage of theground-level information possessed by the employees, the firm has them allocatethemselves among the customers requiring service, providing incentives to ensurethat the employees’ choices further the firm’s objectives.• Markets and bargaining. Large numbers of buyers and sellers participate in a

centralized exchange. Each individual specifies acceptable terms of trade in thehopes of obtaining the greatest benefit from his initial endowment.

While the environments listed above are quite varied, they have certain basic featuresin common. First, each environment contains a large number of agents capable of makingindependent decisions. Second, each agent is small, in that his choices have only a minorimpact on other agents’ outcomes. Third, agents are anonymous: an agent’s outcomefrom the interaction depends on his own strategy and the distribution of others’ strategies;further individuation of the opponents is not required.

Simultaneous interactions exhibiting these three properties can be modeled usingpopulation games. The participants in a population game form a society consisting of oneor more populations of agents. Agents in a given population are assumed to be identical:an agent’s population determines his role in the game, the strategies available to him,

3

and his preferences. These preferences are described by a payoff function that conditionson the agent’s own strategy and the distribution of strategies in each population. Thepopulations may have either finite numbers of agents or continua of agents, whicheveris more convenient. When populations are continuous, the payoffs to each strategy areassumed to depend on the society’s aggregate behavior in a continuous fashion, reflectingthe idea that very small changes in aggregate behavior do not lead to large changes in theconsequences of playing any given strategy.

Aggregate behavior in a population game is described by a social state, which specifiesthe empirical distribution of strategy choices in each population. For simplicity, we assumethroughout this book that there are a finite number of populations, and that members ofeach population choose from a finite strategy set. Doing so ensures that the social stateis finite-dimensional, expressible as a vector with a finite number of components; ifpopulations are continuous the set of social states is a polytope.

Except when they are revising (see the next section), agents in population games areassumed to play pure strategies. One of the main reasons for introducing randomizedstrategies is moot here: when the populations are continuous, and payoffs are continuousin the social state, pure strategy Nash equilibria always exist. This guarantee may beone of the “simplifications” that von Neumann and Morgenstern had in mind whenthey looked ahead to the study of games with large numbers of players. But this factis not as essential to our theory as it is to traditional approaches to analyzing games:while traditional approaches are grounded on the assumption of equilibrium play, weemphasize the process through which agents adjust their behavior in response to theircurrent strategic environment.

From our point of view, a more important simplification provided by the populationgames framework is the description of behavior using social states—that is, by distri-butions of agents’ choices. To understand the advantages this approach brings, let uscontrast it with the standard framework for modeling simultaneous-move interactions:normal form games. To define a normal form game, we must specify a (finite or infinite)set of players, and to describe behavior in such a game, we must stipulate each player’sstrategy choice. If there are many players, these tasks can be laborious.

One can view a population game as a normal form game that satisfies certain restric-tions on the diversity and anonymity of the players. But it is nevertheless preferable towork directly with population games, and so to avoid the extra work involved in individ-uating the players. Moreover, describing behavior using empirical strategy distributionsshifts our attention from questions like “who chose strategy i?” to ones like “how manychose strategy i?” and “what happens to the payoff of strategy j players if some agents

4

switch to strategy i?” This is just where our attention should be: as we will see in Chapter3, the answers to the last question determine the incentive structure of a population game.

Much research in evolutionary game theory has focused on population games of aparticularly simple sort: those generated when populations of agents are matched toplay a normal form game. Indeed, Nash informally introduced population games of thissort in 1949 in proposing the “mass action” interpretation of his equilibrium concept, adevelopment that seems to have gone unnoticed for the next 45 years (see the Notes). Thispopulation-matching interpretation of normal form games has a clear appeal, and we willuse normal form games as a ready source of simple examples throughout the book.

At the same time, if our main interest is in large population interactions rather thanin normal form games, then focusing only on matching in normal form games is quiterestrictive. Maynard Smith observes that matching is a rather special sort of interactionin large populations. Instead, interactions in which each agent’s payoffs are determineddirectly from all agents’ behavior—what Maynard Smith terms “playing the field”—seemto us to be the rule rather than the exception. Only a few of the applications listed atthe onset are most naturally modeled using matching; some, like congestion in highwaynetworks, require payoffs to depend nonlinearly on the population state, and so aremathematically inconsistent with a random matching approach. While one might expectthat moving from linear to nonlinear payoffs might lead to intractable models, we will seethat it does not: the dynamics we study are nonlinear even when payoffs in the underlyinggame are not, so allowing nonlinear payoffs does not lead to a qualitative increase in thecomplexity of the analysis.

1.1.2 Definitions and Classes of Population Games

Our formal introduction to population games begins in Chapter 2, which offers def-initions of population states, payoff functions, and various other items mentioned andunmentioned above. To reduce the degree of abstraction, and to illustrate the range ofpossible applications, the chapter presents a number of basic examples of populationgames; among these are congestion games, which provide a general but tractable model ofnetwork congestion and of related sorts of multilateral externalities. Finally, by showinghow low-dimensional population games can be represented in pictures, Chapter 2 ushersin the geometric methods of analysis that are emphasized throughout the book.

From a purely formal point of view, a population game is defined by an arbitrarycollection of continuous, real-valued functions on an appropriate domain. While somebasic results, including existence of Nash equilibrium, can be proved at this level ofgenerality, obtaining more specific conclusions requires us to focus on classes of games

5

defined by certain structural properties. In Chapter 3, we introduce three important classesof population games: potential games, stable games, and supermodular games. Each of theseclasses of games includes a number of important examples, and each is characterizedby restrictions on the nature of the externalities that users of each strategy impose onone another. The structure imposed by these restrictions makes analyses of games inthese classes relatively simple. For instance, in games from all three classes, one canprove existence of Nash equilibrium without recourse to fixed point theorems. But moreimportant to us is the impact of this structure on disequilibrium behavior: we will see inChapter 7 that in potential games, stable games, and supermodular games, broad classesof evolutionary dynamics are assured of converging to Nash equilibrium from arbitraryinitial conditions.

1.2 Evolutionary Dynamics

The state of equilibrium. . . is therefore stable; i.e., if either of the producers, misledas to his true interest, leaves it temporarily, he will be brought back to it by a seriesof reactions, constantly declining in amplitude, and of which the dotted lines of thefigure give a representation by their arrangement in steps.

Cournot (1838)

We repeat most emphatically that our theory is thoroughly static. A dynamic theorywould unquestionably be more complete and preferable.

von Neumann and Morgenstern (1944, p. 44–45)

An equilibrium would be just an extreme state of rare occurrence if it were not stable—that is, if there were no forces which tended to restore equilibrium as soon as smalldeviations from it occurred.

Besides this stability “in the small”, one may consider stability “in the large”—that is,the ability of the system to reach an equilibrium from any initial position.

Beckmann, McGuire, and Winsten (1956, p. 70)

An obvious weakness of the game-theoretic approach to evolution is that it placesgreat emphasis on equilibrium states, whereas evolution is a process of continuous, orat least periodic, change. It is, of course, mathematically easier to analyse equilibriathan trajectories of change. There are, however, two situations in which game theorymodels force us to think about change as well as constancy. The first is that a gamemay not have an ESS, and hence the population cycles indefinitely. . .

The second situation. . . is when, as is often the case, a game has more than one ESS.Then, in order to account for the present state of the population, one has to allow forinitial conditions—that is, for the state of the ancestral population.

Maynard Smith (1982, p. 8)

6

After modeling a strategic interaction using a population game, one would like to usethe game as the basis for predicting how agents in the interaction will behave. Tradi-tionally, most predictions in game theory have been based on equilibrium analysis: oneintroduces some notion of equilibrium play, and then finds all behaviors in the game thatagree with the equilibrium notion. While the equilibrium approach is standard practicein applications of game theory, the quotations above, drawn from seminal works in eco-nomic theory, game theory, transportation science, and theoretical biology, all emphasizethat this approach is incomplete, and should be complimented by an analysis of dynam-ics. The latter quotes go further: they point out that local stability analysis, which checkswhether equilibrium play will be restored after small disturbances in behavior, is onlya first step, as it begs the question of how equilibrium is established in the first place.These concerns are most pronounced in settings with large numbers of players, where theinterplayer coordination of beliefs and actions associated with equilibrium seems mostdifficult to achieve.

The majority of this book studies dynamic models of behavior in large populationgames, and so is an attempt to provide some answers to the questions raised above.

1.2.1 Knowledge, Rationality, and Large Games

The fundamental solution concept of noncooperative game theory is Nash equilibrium:the requirement that each agent choose a strategy that is optimal given the choices ofthe others. There are many other solution concepts for games, but most of them arerefinements of Nash’s definition, and are called upon to reduce the set of predictions ingames with multiple Nash equilibria.

Despite the central role of the Nash equilibrium concept, the traditional, rationalisticjustification for applying this concept is not especially convincing. This justification isbased on three assumptions about the players in the game. First, each player is assumedto be rational, acting to maximize his payoffs given what he knows. Second, players haveknowledge of the game they are playing: they know what strategies are available, and whatpayoffs result from every strategy profile. Third, the players have equilibrium knowledge:they are able to anticipate correctly what their opponents will do. If all players expect acertain strategy profile to be played, and if each player is rational and understands thepayoff consequences of switching strategies, then each player is content to play his partin the strategy profile if and only if that profile is a Nash equilibrium.

Of the three assumptions listed above, the equilibrium knowledge assumption is thehardest to accept. Certainly, shared expectations about play can be put into place bya disinterested moderator who guides the players to a particular strategy profile. But

7

without such guidance, it is hard to explain how players can introspectively anticipatehow others will act, particularly in games with large numbers of participants.

In fact, when we consider games with many players or strategies, even apparentlyinnocuous conditions for equilibrium play may be called into question. Under the tradi-tional interpretation of equilibrium play in a traffic network, a driver choosing a route towork has a complete mental account of all of the routes he could take, and he is able toanticipate the delay that would arise on each route for any possible profile of choices byhis fellow drivers. Evidently, the assumption of knowledge of the game, while seeminglyinnocent, may actually be quite bold when the game is large.

This discussion suggests that in large games, even the force of seemingly weak solu-tion concepts, ones that do not require equilibrium knowledge, should not be taken forgranted. For instance, a basic tenet of traditional game-theoretic analysis holds that astrictly dominated strategy—a strategy that performs worse than some single alternativestrategy regardless of how opponents behave—should not be chosen. This requirement isuncontroversial when players have full knowledge of the game. But if players are unableor unwilling to keep the entire game in mind, they may well not notice that one strategyis dominated by another. While one might expect that an accumulation of experiencemight ensure that players eventually avoid dominated strategies, we will see that this isnot necessarily so: in Section 9.4, we will present a set of seemingly mild conditions onplayers’ updating rules that are enough to ensure that strictly dominated strategies mustsurvive in perpetuity in some games.

Obtaining a convincing rationalistic justification for equilibrium play in games seemsan impossible task. But in settings where the same game is played many times, the pos-sibilities become brighter, as one can replace introspection with repetition as the basis forcoordination of behavior. In large population settings, repetition may be enough to coor-dinate behavior even when agents’ information and abilities are quite limited. But whiledynamic approaches can support and even refine traditional game-theoretic predictions,they also can also lead to predictions of cyclical or more complex nonstationary behavior,possibilities that are ignored by traditional analyses.

1.2.2 Foundations for Evolutionary Dynamics

There are a variety of approaches one could take to studying disequilibrium dynamicsin games, depending on the number of players involved, the information the players areexpected to possess, and the importance of the interaction to the players. Evolutionary gametheory, the approach studied in this book, considers the dynamics of behavior in large,strategically interacting populations. This approach posits that agents only occasionally

8

switch strategies, and then use simple myopic rules to decide how to act. While theseassumptions are certainly not appropriate for every application, they seem natural whenthe interaction in question is just one among many the agent faces, so that the sporadicapplication of a rule of thumb is a reasonable way for the agent to proceed.

While it is possible to proceed directly with a description of aggregate behavior dy-namics, we find it preferable to begin by specifying when and how individual agents makedecisions. We accomplish these tasks using a modeling device called a revision protocol.A revision protocol is a function that takes the strategies’ payoffs and utilization levels asinputs; it returns as outputs the overall rate of switching strategies, and the probabilitieswith which each alternative strategy will be chosen.

In defining a revision protocol, we implicitly specify the informational burden thatthe agents must bear. Starting in Chapter 4, we show that revision protocols come inmany different varieties, from ones that embody exact myopic optimization, to othersthat require each agent to know nothing more than his own current payoff. In all cases,though, the protocol only relies on information about current strategic conditions: his-torical information, as well as counterfactual information about strategies’ performancesunder other conditions, are not considered.

Also implicit in the definition of a revision protocol is the method to be used to identifyalternative strategies. One can place protocols into two broad categories according to thiscriterion. Under imitative protocols, an agent obtains a candidate strategy by observing thestrategy of a randomly chosen member of his population. Under direct protocols, agents areassumed to choose candidate strategies directly; a strategy’s popularity does not directlyinfluence the probability with which it is considered. (Agents may also meander amongdifferent protocols as time passes, in which case they are said to employ a hybrid protocol.)After obtaining a candidate strategy, an agent can evaluate its current payoff by brieflyexperimenting with it. If the strategy is currently in use, its current payoff can also bedetermined by observing the outcomes of an opponent who employs it.

We will see in Chapter 5 that the aggregate behavior dynamics generated by imitativeprotocols are very similar to those studied in mathematical biology. Thus, the replicatordynamic, introduced in the mathematical biology literature to model natural selection, alsodescribes the aggregate behavior of agents who use certain imitative protocols. In contrast,dynamics based on direct selection—for instance, the best response dynamic, which is basedon optimal myopic choices—behave rather differently than those studied in biology. Directselection allows unused strategies to be introduced to the population, which is impossibleunder pure imitation (or under biological reproduction without mutations). For its part,imitation generates dynamics with relatively simple functional forms and behavior. But

9

the simple forms and special properties of imitative dynamics are rather special: we willfind that when agents use hybrid protocols, their aggregate behavior agrees in broadqualitative terms with what one would see under direct selection of alternative strategies.

1.2.3 Deterministic Evolutionary Dynamics

Suppose that one or more large populations of agents recurrently play a populationgame, with each agent occasionally updating his choice of strategies using a fixed revisionprotocol. Since there are many agents, and since the stochastic elements of the agents’updating procedures are idiosyncratic, one expects these stochastic influences to be aver-aged away, leaving aggregate behavior to evolve in an essentially deterministic fashion.In Chapter 4, we explain how such a deterministic evolutionary process can be describedby an ordinary differential equation, and how this equation can be derived from the pop-ulation game and revision protocol. A formal justification for our study of this differentialequation, which we call the mean dynamic, is deferred until Chapter 10; see Section 1.2.4below.

The mean dynamic specifies the rate of change in the use of each strategy i. It isthe difference of two terms: an inflow term, which captures agents’ switches from otherstrategies to strategy i, and an outflow term, which captures agents’ switches from strategyi to other strategies. Of course, the exact specification of the dynamic depends on theprimitives of the model: namely, the protocol the agents employ and the game they play.

Taking a slightly different point of view, we note that any fixed revision protocol definesa map that takes population games as inputs, and returns specific instances of meandynamics as outputs. We call this map from population games to ordinary differentialequations an evolutionary dynamic.

Chapters 5 and 6 introduce a variety of families of evolutionary dynamics and investi-gate their properties. Each family of dynamics is generated from a set of revision protocolssharing certain qualitative features: for instance, being based on imitation, or on myopicoptimization. The main results in Chapters 5 and 6 establish properties of evolutionarydynamics that hold regardless of the population game at hand.

One of the basic issues considered in these chapters is the relationship between therest points of the dynamics and the payoff structure of the underlying game. We showin Chapter 5 that the rest points of imitative dynamics include all Nash equilibria of thegame being played. Further analyses reveal that under many direct and hybrid dynamics,the sets of rest points and Nash equilibria are identical. This latter property, which we callNash stationarity, provides a first link between the population dynamics and traditionalequilibrium analyses. A second issue we address is the connection between out-of-

10

equilibrium dynamics and incentives in the underlying game. Many of the dynamics westudy satisfy a monotonicity property called positive correlation, which requires strategies’growth rates to be positively correlated with their payoffs. This property and its relativesare a basic ingredients in our later analyses of dynamic stability.

Properties like Nash stationarity only provide a weak justification for the prediction ofNash equilibrium play. To obtain a more convincing defense of this prediction, one mustaddress the questions of stablilty raised at the start of this section.

Chapter 8 considers local stability of equilibrium: whether equilibrium will be reachedif play begins at a nearby social state. It is easy to see that some Nash equilibria areunlikely to be locally stable. For instance, if play begins near the mixed equilibrium ofa coordination game, then myopic adjustment will lead the population away from thisequilibrium. Additional restrictions beyond Nash’s condition are thus needed to obtaingeneral stability results. A main finding in Chapter 8 is that the notion of an evolutionarilystable state (ESS), introduced by Maynard Smith and Price for model of evolution inpopulations of mixed strategists, provides a general sufficient condition for local stabilityunder the pure-strategist dynamics studied here.

Local stability results only offer a partial justification of equilibrium predictions. To beconvinced that equilibrium will be reached, one must look instead for global convergenceresults, which establish that equilibrium is attained from any initial state. We offer suchresults in Chapter 7, where we prove that in certain classes of games—namely, the classesof potential games, stable games, and supermodular games, introduced in Chapter 3—there are classes of dynamics that converge to equilibrium from all or almost all initialconditions.

These stability results provide strong support for the Nash prediction in some settings,but they say little about behavior in games outside of the classes the results cover. Whileone might hope that these results are nevertheless representative of behavior in mostgames, this seems not to be so. In Chapter 9, we present a variety of examples in whichdeterministic dynamics enter cycles far from Nash equilibrium, and others in which thedynamics display chaotic behavior. Thus, if we take the model of myopic adjustmentdynamics seriously, we must sometimes accept these more complicated limit behaviorsas more credible predictions than equilibrium play. Moreover, as we hinted earlier, thepossibility of nonconvergent behavior has some counterintuitive consequences: we showin Section 9.4 that under “typical” evolutionary dynamics, we can always find simplegames in which strictly dominated strategies survive.

11

1.2.4 Orders of Limits for Stochastic Evolutionary Models

The deterministic dynamics studied in Chapters 4–9 form the largest portion of theliterature on evolutionary game dynamics. But there is another major branch of the litera-ture that focuses on the infinite horizon behavior of stochastic evolutionary dynamics, andwhose central aim is to obtain unique predictions of play in games with multiple equilib-ria. In Chapter 10, we bring together these two literatures by showing how deterministicand stochastic evolutionary game dynamics can be derived from a single foundation.

To accomplish this, we consider population games played by large but finite popu-lations of agents. Together, a finite-population game and a revision protocol define astochastic evolutionary process—in particular, a Markov process—on the now finite setof population states. What methods are most useful for studying this process depends onthe parameter values—the population size N and the time horizon T—that are relevant tothe application at hand.

This point is made most clearly by taking limits. To begin, suppose that we fix the timehorizon T and take the population size N to infinity. The main result of Chapter 10 showsthat once N is large enough, the stochastic evolutionary process is very likely to behave ina nearly deterministic way, mirroring a solution trajectory of the relevant mean dynamicthrough time T. For intuition, observe that the stochastic aspects of the evolutionaryprocess—the random arrivals of revision opportunities, and the randomizations amongcandidate strategies during these opportunities—are idiosyncratic. When the populationsize becomes very large, the idiosyncratic noise is averaged away. Thus, the behavior ofthe process is driven by the expected changes in the state, which are precisely what themean dynamic captures.

Alternatively, suppose we fix the population size N and take the time horizon T toinfinity. In this case we are studying the infinite horizon behavior of a finite-state Markovprocess. If this process is irreducible—that is, if all states are mutually accessible, as istrue, for instance, under any revision protocol that always places positive probability oneach alternative strategy—then results from probability theory tell us that every socialstate will be visited infinitely often, and that the proportion of time spent in each stateover the infinite horizon does not depend on the initial state. This invariance propertyopens the door to obtaining unique predictions of play.

1.2.5 Stationary Distributions and Stochastic Stability

The proportion of time that an irreducible Markov process spends in each state isdescribed by the process’s stationary distribution. By computing this distribution, we can

12

obtain an exact description of the infinite horizon behavior of our evolutionary process.Chapter 11 presents some basic results on stationary distributions of stochastic evolution-ary models, and studies in detail those cases—namely, the models that generate reversibleMarkov processes—in which the stationary distribution can be computed exactly.

These analyses provide a first illustration of a fundamental idea. Even if the underlyinggame has multiple equilibria, it is nevertheless the case that in the infinite horizon, astochastic evolutionary dynamic may spend the vast majority of periods in the vicinityof a single state. But we also argue that the amount of time necessary for this analysis tobecome relevant can be exceedingly long, limiting the range of possible applications.

Once one moves beyond reversible cases, obtaining an exact expression for the station-ary distribution is typically impossible. One way of circumventing this difficulty is to takecertain parameters of the evolutionary process to their limiting values. If one takes thenoise level that parameterizes the agents’ revision protocol to zero, or the population sizeto infinity, one can sometimes describe the limiting stationary distribution, even thoughthe stationary distributions for fixed parameter values cannot be computed explicitly.There are many interesting cases in which the limiting stationary distribution places all ofits mass on a single state, providing an especially tidy prediction of infinite horizon play.

States that retain mass in the limiting stationary distribution are said to be stochasticallystable. In Chapter 12 we offer a complete presentation of stochastic stability theory. Wedescribe the main analytical techniques used to determine the stochastically stable states,and we present key selection results. We also establish formal connections between thestates that are stochastically stable in the large population limit and the recurrent statesof the relevant mean dynamic. In doing so, we provide another link between stochasticstability analysis and the deterministic dynamics studied in earlier chapters.

1.3 Remarks on History, Motivation, and Interpretation

Evolutionary game theory dates from the early 1970s, when John Maynard Smithintroduced the ESS concept as a way of understanding ritualized animal conflict (see theNotes). In pursuing this line of research, Maynard Smith and other biologists borrowed thestandard game-theoretic definitions of mathematicians and economists, but reinterpretedthese definitions to suit their own purposes. In a biological model, a strategy is notsomething that an animal chooses, but a behavior that an animal is hardwired to performby virtue of its genetic endowment. Similarly, payoffs are not a numerical representationof a preference relation, but instead represent “Darwinian fitnesses”: they describe howthe use of the strategy improves an animal’s prospects for survival and reproduction.

13

Thus, the development of a biological approach to game theory was marked not only bythe invention of new solution concepts and methods of analysis, but also by a recasting ofexisting definitions.

In the late 1980s, economists surmised that this biological paradigm could be used toaddress problems with the foundations of traditional game theory. During the heyday ofthe equilibrium refinements literature, game theorists introduced solution concepts thatseemed to impose ever larger demands on the reasoning abilities of the players whosebehavior they purported to describe. This development raised the criticism that thesolution concepts could only describe the behavior of hyperrational players. If, though,it could be shown that the behavior of populations of preprogrammed creatures could bedescribed by means of these same solution concepts, then this would seem to provide away of dispelling the hyperrationality critique.

In view of this motivation, it is not surprising that many early efforts by economists inevolutionary game theory adopted the biological paradigm wholesale, with populationsof agents hardwired to choose certain strategies, and payoffs interpreted as Darwinianfitnesses. These analyses often found that behavior in these populations, as describedby ESS and other biological solution concepts, agreed in a formal sense with the mostdemanding of the equilibrium refinements introduced by economists.

While these links between the biological and economic approaches to game theorymight provide some sense of reassurance, it is important to be mindful of the limitationsof this line of reasoning. In many economic applications of game theory, the game inquestion is played by, say, two or three players. Nash equilibrium and its refinementsoffer predictions about the mixed strategies these players will employ. For their part, theevolutionary models introduced by biologists concern interactions among large popula-tions of animals, and their solution concepts describe the effects of natural selection onpopulation shares of different genetic types. The fact that economists’ predictions aboutrational agents’ choices of mixed strategies formally agree with biologists’ predictionsabout population shares under natural selection is certainly intriguing. But there is noobvious logic by which this agreement enables either analysis to justify the other.

Economists working on population dynamics soon realized that to obtain a modelingtool useful for typical social science environments, the Darwinian assumptions and inter-pretations of evolutionary game theory would need to be replaced with ones reflectinghuman decision making. This new vision of evolutionary game theory brought back theoriginal interpretation of the game itself: strategies returned to being objects of choice,and payoffs resumed being descriptions of individual preferences. But other parts of thebiologists’ approach—in particular, the very idea of population dynamics, with changes

14

in strategies’ population shares driven by differences in current payoffs—were retained.Of course, some reinterpretation was necessary here, as the “evolution” of populationshares would no longer be driven by relative rates of reproduction, but rather by con-scious decisions to switch from one strategy to another. More intriguing agreementswere discovered between the biological and economic approaches. For instance, Taylorand Jonker introduced the replicator dynamic in 1978 to provide a dynamic foundationfor Maynard Smith’s static approach to natural selection. But in the 1990s, economistsfound that this dynamic could also be derived from models of imitation in populations ofeconomic agents—see Section 5.4.

This history has created some understandable confusion about the meaning of the term“evolutionary game theory”. From the origins of the field through the earliest contribu-tions of economists, this phrase referred exclusively to game-theoretic models of naturalselection. But since the mid 1990s, it has also encompassed models of myopic behaviorin games played by large populations of active decision makers. In this interpretation,which predominates in this book, the word “evolution” should not be understood as adirect reference to Darwinian survival of the fittest, but instead should be taken in itsbroader sense, referring to a process of gradual change.

As the interpretation of evolutionary game theory has changed, so too has its rolewithin economic modeling. As we noted earlier, the initial flush of interest in evolutionarygame theory in the economics community stemmed from the theory’s potential to providelow-rationality foundations for high-rationality solution concepts. This motivation helpsexplain why economists working in evolutionary game theory have devoted so mucheffort to the study of random matching in normal form games. If the point of the theoryis to motivate solution concepts for such games, then there is not much reason to movebeyond the random matching setting.

As we have emphasized here, predictions about the aggregate behavior of large pop-ulations do not directly address how two players would play a two-player normal formgame. For such predictions to play more than a metaphorical role, they must do so inapplications that specifically concern the behavior of populations. But once such appli-cations are brought to the fore, restricting attention to matching in normal form gamesbegins to feel artificial: if one is interested in modeling behavior in large populations, thegames one should write down are those one believes populations actually play.

15

1.N Notes

Preface: General references on evolutionary game theory. Books: Maynard Smith (1982),Hofbauer and Sigmund (1988, 1998), Bomze and Potscher (1989), Cressman (1992, 2003),Weibull (1995), Vega-Redondo (1996), Samuelson (1997), Fudenberg and Levine (1998),Young (1998b). Surveys: van Damme (1991, Chapter 9), Hines (1987), Kandori (1997),Mailath (1998), Weibull (2002), Hofbauer and Sigmund (2003), Sandholm (2008b),

References on omitted topics. Evolution in extensive form games. Cressman (1996a, 2000,2003), Cressman and Schlag (1998), Chamberland and Cressman (2000), Binmore et al.(1995a), Binmore and Samuelson (1999), Ponti (2000) (deterministic models); Samuelson(1994, 1997), Noldeke and Samuelson (1993), Hart (2002), Kuzmics (2004) (stochasticmodels). Local interaction models: Blume (1993, 1995, 1997), Ellison (1993, 2000), Kosfeld(2002), Miekisz (2004); Herz (1994), Ely (2002), Eshel et al. (1998), Anderlini and Ianni(1996), Goyal and Janssen (1997), Alos-Ferrer and Weidenholzer (2006, 2007, 2008), Goyal(2007), Vega-Redondo (2007), Jackson (2008); Nowak and May (1992, 1993), Nowak (2006),Szabo and Fath (2007), Hauert (2007). Continuous strategy sets: Eshel (1983), Bomze (1990,1991), Friedman and Yellin (1997), Oechssler and Riedel (2001, 2002), Eshel and Sansone(2003), Cressman (2005, 2009), Cressman and Hofbauer (2005), Cressman et al. (2006),Friedman and Ostrov (2008), Norman (2008), Hofbauer et al. (2009). Stochastic evolutionvia diffusion processes: Foster and Young (1990), Fudenberg and Harris (1992), Cabrales(2000), Imhof (2005), Benaım et al. (2008), Hofbauer and Imhof (2009).

References on heuristic learning. Original references: Brown (1949, 1951), Robinson(1951). Books: Fudenberg and Levine (1998), Young (2004), Cesa-Bianchi and Lugosi(2006). Survey: Hart (2005). Brief overview: Sandholm (2008d).

References on applications. Markets: Hopkins and Seymour (2002), Lahkar (2007), Drosteet al. (2002), Sethi (1999), Ben-Shoham et al. (2004), Agastya (2004), Vega-Redondo (1997),Alos-Ferrer et al. (2000), Ania et al. (2002), Alos-Ferrer and Ania (2005), Alos-Ferrer et al.(2006), Kandori et al. (2008), Friedman and Ostrov (2008). Bargaining and hold-up prob-lems: Young (1993b, 1998b,a), Burke and Young (2001), Ellingsen and Robles (2002), Troger(2002), Binmore et al. (2003), Abreu and Sethi (2003), Dawid and MacLeod (2008), Robles(2008). Signaling and cheap talk: Noldeke and Samuelson (1993, 1997), Jacobsen et al.(2001), Robson (1990), Matsui (1991), Warneryd (1993), Blume et al. (1993), Kim and Sobel(1995), Bhaskar (1998), Banerjee and Weibull (2000), Trapa and Nowak (2000), Skyrms(2002), Pawlowitsch (2008), Demichelis and Weibull (2008). Public good provision: Sethiand Somanathan (1996), Myatt and Wallace (2008b,a, 2009). Implementation and de-centralized control: Cabrales (1999), Cabrales and Ponti (2000), Sandholm (2002, 2005b,2007b), Cabrales and Serrano (2007), Mathevet (2007), Arslan et al. (2007), Marden et al.

16

(2009a). Residential segregation: Young (1998b, 2001), Mobius (2000), Zhang (2004a,b),Dokumacı and Sandholm (2007a). Preference evolution: Guth and Yaari (1992), Guth(1995), Huck and Oechssler (1999), Kockesen et al. (2000), Sethi and Somanathan (2001),Ok and Vega-Redondo (2001), Ely and Yilankaya (2001), Dekel et al. (2007), Heifetz et al.(2007), Herold and Kuzmics (2008). Cultural evolution: Bisin and Verdier (2001), Sand-holm (2001b), Kuran and Sandholm (2008) Montgomery (2009). Applications in biology:Maynard Smith (1982), Hofbauer and Sigmund (1988) (books); Hammerstein and Selten(1994), Dugatkin and Reeve (1998) (surveys).

Section 1.1. Noncooperative games with a continuum of agents were introduced bySchmeidler (1973), who proves that pure strategy Nash equilibria exist when each player’spayoffs only depend on his own action and on the aggregate behavior of all players.Schmeidler (1973) uses a normal form approach, in that he names and describes the be-havior of each agent separately. Mas-Colell (1984) introduces a simpler formulation basedon empirical distributions of strategies rather than on strategy profiles; his formulationcan be understood as a population game in which both the set of populations and the setof strategies may be continuous. Mas-Colell (1984) also points out the overlap betweenhis model and the distributional approach to Bayesian games with a continuum of types(Milgrom and Weber (1985)). This connection can be used as the basis for studying evo-lution in Bayesian games; see Ely and Sandholm (2005). The literature on noncooperativegames with large numbers of players is surveyed in Khan and Sun (2002); see Balder(2002) and Carmona and Podczeck (2009) for recent results.

Nash’s “mass action” interpretation of his equilibrium concept appears in his disserta-tion (Nash (1950b)), but not in either of its published versions (Nash (1950a, 1951)). It waslater rediscovered by Leonard (1994), Weibull (1994, 1996), and Bjornerstedt and Weibull(1996); see also Hofbauer (2000).

Section 1.2. There are many good general references on traditional game-theoreticanalysis: see Osborne (2004) at the undergraduate level, and Fudenberg and Tirole (1991),Myerson (1991), Ritzberger (2002), and van Damme (1991) at the graduate level. For anoverview of the use of rationality and knowledge assumptions in game theory, see Dekeland Gul (1997).

While evolutionary game theory models assume that agents are myopic, one canimagine large population interactions whose participants are better modeled as forward-looking dynamic optimizers, who attempt to maximize discounted flows of future payoffs.The perfect foresight dynamics of Matsui and Matsuyama (1995) introduce forward-lookingbehavior to large population models while retaining the assumption of inertia from theevolutionary approach. Recent work on perfect foresight dynamics includes Hofbauer

17

and Sorger (1999), Oyama (2002), Matsui and Oyama (2006), Kojima (2006a), Kojima andTakahashi (2008), Takahashi (2008), Oyama et al. (2008), and Oyama and Tercieux (2009).

Section 1.3. While papers by John Maynard Smith in the early 1970s (Maynard Smith(1972, 1974), Maynard Smith and Price (1973)), are usually viewed as marking the birthof evolutionary game theory, Maynard Smith (1982, p. 2) himself notes the appearanceof game-theoretic ideas in earlier studies of the evolution of sex ratios: they appearedimplicitly in the work on Fisher (1930), and explicitly in the work of Hamilton (1967). Butit was only with Maynard Smith’s work that game theory began to take on its role as astandard framework for biological modeling.

18