multiagent social learning in large repeated games jean oh
Post on 13-Dec-2015
219 Views
Preview:
TRANSCRIPT
Selfish solutions can be suboptimal.
If short-sighted,
Motivation Approach Theoretical Empirical Conclusion
far
“Discovery of strategies that support mutually desirable outcomes”
3
A={ resource1, resource2… resourcem }
N={ } …
statet
agent1agent2 agentn
Strategy of agent iCost ci(si, s-i) ? Strategy profile
e1 e2
e4
e3
Multiagent resource selection problem
strategy
Individual objective: to find a path that minimizes cost
4
Congestion cost depends on:the number of agents that have chosen the same resource.
• Individual objective: to minimize congestion cost• “Selfish solutions” can be arbitrarily suboptimal [Roughgarden 2007].• Important subject in transportation science, computer networks, and
algorithmic game theory.
Congestion game!
“Selfish solution”: the cost of every path becomes more or less indifferent; thus no one wants to deviate from current path. (a.k.a. Nash equilibrium, Wardrop’s first principle)
social welfare:average cost of
all agents
5
Constant cost: 1
n agents
Metro vs. Driving[Pigou 1920, Roughgarden 2007]
Example: Inefficiency of selfish solution
Depends on # of drivers: 1
Optimal average cost [n/2 1 + n/2 ½]/n = ¾
Objective: minimize average cost
Centraladministrator
Stationary algorithms(e.g. no regret, fictious play)
n
1 1
Selfish solution Average cost = 1
metro
drivi
ng
2
n
2
n
Nonlinear cost function?
# of agents
cost
6
If a few agents take alternative route, everyone else is better off. Just a few altruistic agents to sacrifice, any volunteers?
Excellent! as long as it’s not me.
7
Coping with the inefficiency of selfish solution
• Increase resource capacity [Korilis 1999]
• Redesign network structure [Roughgarden 2001a]
• Algorithmic mechanism design [Ronen 2000,Calliese&Gordon 2008]
• Centralization [Shenker 1995, Chakrabarty 2005, Blumrosen 2006]
• Periodic policy under “homo-egualis” principle [Nowé et al. 2003]– Taking the worst-performing agent into consideration (to avoid inequality)
• Collective Intelligence (COIN) [Wolpert & Tumer 1999]– WLU: Wonderful Life Utility!
• Altruistic Stackelberg strategy [Roughgarden 2001b]
– (Market) leaders make first moves, hoping to induce desired actions from the followers
– LLF (centralized + selfish) agents• “Explicit coordination is necessary to achieve system
optimal solution in congestion games” [Milchtaich 2004]
Braess’ paradox
Related work
Can self-interested agents support mutually
beneficial solution without external
intervention?
8
Explicit threat: grim-trigger
We’ll be mutually beneficial
I’ll punish you with minimax value
forever
As long as you stay If you deviateWhatever you do from then on
Minimax value: as good as [i] can get when the rest of the world turns against [i].
• Computational intractability• May require centralization: “significant coordination overhead”• Existing algorithms limited to 2-player games (Stimpson 2001, Littman & Stone 2003, Sen et al. 2003, Crandall 2005)
NP-complete(Borgs et al. 2008)
NP-hard(Meyers 2006)
Complete monitoring
Related work: strategies that support mutually beneficial solutions
Congestion cost
Coordinationoverhead
Agenda: to find more efficient strategies that can support mutually beneficial solutions.
[Nash equilibrium of a repeated game] when everyone adopts grim-trigger
10
Assumptions
The other agents are _______________.1. opponents
2. sources of uncertainty
3. sources of knowledge
The agents are _________ in their ability.1. symmetric
2. asymmetric
“sources of knowledge”
“asymmetric”
may be
IMPRES
“Learn to act more rationallyby using strategy given by others”
“Learn to act more rationallyby giving strategy to others”stop
Go
Intuition: social learning
IMPRES
Independent: non-zero probability of collision
12
Agent i
Agent k
congestion cost
path
Agent j
Inner-layer
Overview: 2-layered decision making
Meta-layer
Agent iAgent j Agent k
Environment
IMPRES
-solitary-subscriber-strategist
1. whose strategy?
2. which path?
3. Learn strategies using cost
“Take route 2”
- Meta-layer- Inner-layer
13
Meta-learning: which strategy?
LOOP:• p selectPath(a); take path p; find out congestion cost c• Update Q value of action a using cost c: Q(a) (1-)Q(a) + (MaxCost - c)• new action randomPick(strategist lookup table L); A A {}• Update meta-strategy s
• a select action according to meta-strategy s; if a = -strategist, L L {i}
Aa
TaQ
TaQ
as
Aa
,)'(
exp
)(exp
)(
'
IMPRES
A = {-strategist, -solitary }Q 0 0s 0.5 0.5
how to select action from A
Current meta-action a
-subscriber0
Environment
path
coststrategy Agent i
how to select path from P = {p1,…}
strategy …
Strategist lookuptable L
More probability mass to low cost actions
14
Inner-learning: which path?• f: number of subscribers (to this strategy)
when f = 0, no inner-learning : joint strategy for f agents
1. path p; take path p; observe # of agents on edges of p
2. Predict traffic on each edge generated by others3. Select best joint strategy for f agents (exploration
with small probability) symmetric network congestion games
4. Shuffle joint strategy correlated strategy: probability distribution over all possible
joint actions
IMPRES
e1 e2
e4
e3
f = 2 f = 0
drive metro
drive 0 0.5
metro 0.5 0(drive, metro)
15
Motivation Approach Theoretical Empirical Conclusion
IMPRES
• Mechanics of the algorithm– Meta-layer: which strategy?– Inner-layer: which path?
• Structure of learned strategy– IMPRES vs. Grim-trigger– Main theoretical results
• Empirical results
16
Non-stationary strategy:strategy that depends on past plays
-subscriberstrategy
-solitarystrategy
exploitexplore
Cost(C) Cost(I)
Cost(C) ≥ Cost(I)
Cost(C) Cost(I)
Cost(I) Cost(C)
An IMPRES strategy
Correlated Strategy (C)
IndependentStrategy (I)
Motivation Approach Theoretical Empirical Conclusion
IMPRES: any correlated strategy that is better than independent strategy can
be supported.
Grim-trigger: any correlated strategy that is better than minimax can be supported.
17
Mutually beneficialstrategy
Independentstrategy
Cost(C) Cost(I)
Cost(C) ≥ Cost(I)
Cost(C) Cost(I)
Cost(I) Cost(C)
An IMPRES strategy
Grim-trigger vs. IMPRES
Mutually beneficialstrategy
Minimaxstrategy
Other playersobey Whatever
A grim-trigger strategy
Observe a deviator
Perfect monitoring Imperfect monitoring Intractable Tractable
Coordination overhead (centralization) Efficient coordination Deterministic Stochastic
Strategies that can support mutually beneficial outcome
Independentstrategy
Minimaxstrategy
exploitexplore
18
Mutually beneficialstrategy
Rational agents can support mutually beneficial outcome with
explicit threat.
General belief
Motivation Approach Theoretical Empirical Conclusion
Minimaxstrategy
Explicit threat
independentstrategy
Implicit threat
“Rationally bounded IMPRES”
“without”
Main result
Empirical evaluation
Motivation Approach Theoretical Empirical Conclusion
Selfish solutionsCongestion cost: arbitrarily suboptimal Coordination overhead: none
Con
gest
ion
cost
Mutually beneficial solutions Congestion cost: optimalCoordination overhead: significant
Coordination overhead
IMPRES
Quantifying “mutually beneficial” and “efficient”
(1-to-n centralization)
20
Evaluation criteria
1. Individual rationality: minimax-safety2. Average congestion cost of all agents
(social welfare); for problem p3. Coordination overhead (size of
subgroups) relative to a 1-to-n centrally administrated system.
4. Agent demographic (based on meta-strategy), e.g. percentage of solitaries, strategists, and subscribers.
Cost (solutionp)
Cost (optimump)
overhead (solutionp)
overhead (maxp)
21
• Number of agents n = 100; (n = 2 ~ 1000)• All agents use IMPRES (self-play)• Number of iterations = 20,000 ~ 50,000• Averaged over 10-30 trials • Learning parameters:
Experimental setup
Parameter Value Description
Learning step size; use bigger step size for actions tried less often.
T T0=10; T 0.95T Temperature in update eq.
k 10 Max number of actions in meta-layer
)10
1,01.0max(
iatrials
22
metro
driving
Metro vs. Driving (n=100)
metro
driving
Agent demographic
The lower, the better
Free riders:always driving
# of agents
cost
24
C(s): congestion cost of solution s
C(s)
C(optimum)
Selfish solution Optimum IMPRES
(3,3)(3,1)(3,1.2)
For this problem:
Polynomial cost functions, average number of paths=5
Optimal baseline[Meyers 2006]
Selfish base
line
[Fabrikant 2
004]
Selfish solution: the cost of every path becomes more or less
indifferent; thus no one wants to deviate from current path.
y=x
(data is based on average cost after 20,000 iterations)
C(selfish solution)
C(optimum)
25
o(s): coordination overhead of solution s
C(s)
C(optimum)
O(solution)
O(1-to-n solution)
Polynomial cost functions, average number of paths=5
1-to-n solution
eso )( Average communication bandwidth
Congestion cost
Optimum
better
worse
Coordination overhead
26
On dynamic population
1 agent in every ith round, randomly selected, replaced with new one
40 problems with mixed convex cost functions, average number of paths=5
Optimal baseline
(data is based on average cost after 50,000 iterations)
Selfish base
line
C(s)
C(optimum)
C(selfish solution)
C(optimum)
27
Summary of experiments
• Symmetric network congestion games– Well-known examples– Linear, polynomial, exponential, & discrete cost functions– Scalability
• number of alternative paths (|S| = 2 ~ 15)• Population size (n = 2 ~ 1000)
– Robustness under dynamic population assumption
• 2-player matrix games• Inefficiency of solution based on 121 problems:
– Selfish solutions: 120% higher than optimum– IMPRES solutions: 30% higher than optimum
25% coord. overhead of 1-to-n model
Motivation Approach Theoretical Empirical Conclusion
limitation
28
Contributions
• Discovery of social norm (strategies) that can support mutually beneficial solutions
• Investigated “social learning” in multiagent context• Proposed IMPRES: 2-layered learning algorithm
– significant extension to classical reinforcement learning models
– the first algorithm that learns non-stationary strategies for more than 2 players under imperfect monitoring
• Demonstrated IMPRES agents self-organize:– Every agent is individually rational (minimax-safety)– Social welfare is improved by approx. 4 times from selfish
solutions– Efficient coordination (overhead within 25% of 1-to-n model)
Motivation Approach Theoretical Empirical Conclusion
29
Future work
• Short-term goals: more asymmetry– Strategists – give more incentive– Individual threshold (sightseers vs. commuters)– Tradeoffs between multiple criteria (weight)– Free rider problem
• Long-term goals:– Establish the notion of social learning in artificial
agent learning context• Learning by copying actions of others• Learning by observing consequences of other agents
Motivation Approach Theoretical Empirical Conclusion
30
Conclusion
Rationally bounded agents adopting social learning can support mutually beneficial outcomes without the explicit notion of threat.
top related