1

Presenter: Chih-Yuan Chou

GA-BASED ALGORITHMS FOR FINDING EQUILIBRIUM

2

AGENDA• “The evolution of strategies in the iterated prisoner's

dilemma” (Axelrod, 1987).

• “Genetic algorithm learning and evolutionary games” (Riechmann, 2001).

• “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

3

AGENDA• “The evolution of strategies in the iterated prisoner's

dilemma” (Axelrod, 1987).

• “Genetic algorithm learning and evolutionary games” (Riechmann, 2001).

• “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

4

PRISONER'S DILEMMA

5

ITERATED PRISONER'S DILEMMA• The same two individuals may meet more than once.

• An individual can recognize a previous interactant and remember some aspects of the prior outcomes.

• A strategy is a decision rule which specifies the probability of cooperation or defection as a function of the history of the interaction so far.

6

5-STAGE SIMULATION• 1. Initialize the population by generate random strings

of C's and D's.

• 2. each individual player uses the strategy defined by its chromosome to play in iterated Prisoner's Dilemma with other strategies

• 3. The relatively successful individuals are selected to have more offspring.

7

5-STAGE SIMULATION• 4. crossover and mutation

• 5. The new population will display patterns of behavior that are more like those of the successful individuals of the previous generation.

8

SIMULATION RESULTS• Cooperate after three mutual cooperations.

• Defect after three mutual defections.

• Defect when the other player defects out of the blue.

• Continue to cooperate after cooperation has been restored.

9

TIT FOR TAT• Cooperate on the first move

• Do whatever the other player did on the preceding move

10

AGENDA• “The evolution of strategies in the iterated prisoner's

dilemma” (Axelrod, 1987).

• “Genetic algorithm learning and evolutionary games” (Riechmann, 2001).

• “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

11

CANONICAL GA

12

ECONOMIC GA• Definition: An economic genetic algorithm is a genetic

algorithm with a state-dependent fitness function.

• The fitness of an economic agent does not only depend on her own strategy, but also on the strategies of all other agents involved in the model.

13

GENETIC POPULATIONS AS NEAR NASH EQUILIBRIUM• A population can be described by counting how often

each of the different possible strategies is used by the members of the population.

• The population tends over time to move to a Nash equilibrium without fully reaching it.

14

EVOLUTIONARILY STABLE STATES(ESS)• a strategy is evolutionarily stable if, relative to its

population, it performs better than any ‘invading’ strategy.

• ESS are based on the notion that invading ‘mutant’ strategies are somehow rejected or eliminated from the population.

15

SET UP A CONCEPT OF EVOLUTIONARY STABILITY FOR GA• A genetic population is evolutionarily stable if the process of

the genetic algorithm rejects an invasion by one or more strategies from the genetic population.

• Invasion itself can either take the form of a totally new strategy entering the population or it can simply mean a change in the frequency of the strategies already contained within the population.

• A more precise definition of an evolutionarily stable population might be: A population is evolutionarily stable if it is resistant against changes in its composition.

16

SET UP A CONCEPT OF EVOLUTIONARY STABILITY FOR GA• If a genetic population is evolutionarily stable, it will

recover from an invasion within a finite number of steps of the GA, which means that in the long run the population will not lastingly be changed.

17

CONCLUSION OF THIS PAPER• Economic genetic algorithm learning can be shown to

be a specific form of an evolutionary game.

• The results of this paper demonstrate that research in the dynamics of economic GA learning models can be equipped with the whole tool box of evolutionary game theory.

18

AGENDA• “The evolution of strategies in the iterated prisoner's

dilemma” (Axelrod, 1987).

• “Genetic algorithm learning and evolutionary games” (Riechmann, 2001).

• “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004).

19

MOTIVATION• The conventional NE search algorithms and

coevolutionary algorithms can not differentiate real NE and “local NE traps”

20

CONTRIBUTIONS OF THIS PAPER• Prove that any conventional iterative NE search

algorithms applied to games with local optima can misidentify NE by following a local optimization path.

• Introduce the concept of “local NE trap” that attracts the conventional iterative NE search algorithms based on local optimization tools.

• Develop an enhanced version of coevolutionary programming to overcome the problem.

21

LOCAL NE TRAP• Definition 1 (Nash Equilibrium):

is an NE if

.

• Definition 2 (Local NE):

is a local NE if

where .

22

LOCAL NE TRAP• Definition 3 (Local NE Trap):

is a local NE trap if

but

23

LOCAL NE TRAP

24

LOCAL NE TRAP

25

MODEL OF ITERATIVE NE SEARCH ALGORITHMS• 1) Initialize each player’s strategy choice set.

• 2) Given other players’ decisions, solve each player’s profit maximization problem. Do this step for all players.

• 3) Repeat Step 2) until the NE condition is satisfied or the maximum iteration number is reached.

26

MODEL OF ITERATIVE NE SEARCH ALGORITHMS• Proposition: Every local NE, including any local NE

trap, can be the solution of any conventional iterative NE search algorithm.

27

EXAMPLE

28

EXAMPLE

29

COEVOLUTIONARY PROGRAMMING• Each generation of the genetic algorithm-based

coevolutionary programming consists of two parts:

• The random matching

• The evolutionary process of each player

30

COEVOLUTIONARY PROGRAMMING

31

COEVOLUTIONARY PROGRAMMING

32

WHOLE PROCESS OF COEVOLUTIONARY PROGRAMMING• 1) Set the basic parameters of maximum generation number, population

size, crossover rate, and mutation rate.

• 2) Initialize the strategy population of each player.

• 3) Choose one strategy from the strategy population of each player randomly from among the strategies that have not already been assigned profits. Input the strategy information to the tournament. The result of the tournament will decide profit values for these chosen strategies.

• 4) Repeat Step 3) until every strategy is assigned a profit value.

• 5) Apply the evolutionary operators to each strategy population. Keep the best strategy of the current generation alive (elitism).

• 6) Repeat Steps 3)–5) until the maximum generation number is reached.

33

HYBRID COEVOLUTIONARY PROGRAMMING• “best rival matching and fine tuning.”

• The chosen individual will be matched against the best strategy of the other populations in the current generation.

34

THE ALGORITHMIC STEPS• 1) Randomly choose a strategy from the strategy population

of a player.

• 2) Take the best strategies of all other rivals from the previous generation.

• 3) Apply a local hill climber for fine tuning of the chosen strategy of the player.

• 4) Repeat Steps 1)–3) as many times as set by the best rival matching rate. For example, with a best rival matching rate of 0.2, 20% of strategies in each population will be chosen.

• 5) Repeat Steps 1)–4) for every other player in turn.

35

THE ALGORITHMIC STEPS

36

SIMULATION• This paper compared three algorithms:

• Conventional NE search algorithm

• Simple coevolutionary programming

• Hybrid coevolutionary programming

37

GAME CONFIGURATION• The profit functions for player A and player B:

38

GAME CONFIGURATION• NE and local NE trap of the numerical game example

39

CONVENTIONAL NE SEARCH• Find multiple solutions including NE and local NE trap

without being able to differentiate the NE from the local NE traps.

40

LOCAL ITERATIVE SEARCH• Iterative NE search result

41

COEVOLUTIONARY PROGRAMMING

42

HYBRID COEVOLUTIONARY PROGRAMMING

43

CONCLUSION OF THIS PAPER

44

CONCLUSION OF THIS PAPER• Prove that any iterative NE search algorithms based on

local optimization can not differentiate between the real NE and the “local NE traps”

• Suggested an enhancement of coevolutionary programming to solve games with local optima.

• Hybrid coevolutionary programming converged to the NE, if it existed, and did not converge to any local NE traps.

45

REFERENCE• “The evolution of strategies in the iterated prisoner's dilemma” Axelrod,

1987.

• “Genetic algorithm learning and evolutionary games” Riechmann, 2001.

• “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” Son, 2004.