mechanismdesignforpredic2on: combinatorial*predic2on

Mechanism Design for Predic2on: Combinatorial Predic2on Markets

@ CS286r Fall 2012, Harvard David Pennock

MicrosoD Research

An Example Predic2on

•  A random variable, e.g. Will US go into recession in 2012?

(Y/N)

An Example Predic2on Market

•  A random variable, e.g.

•  Turned into a financial instrument payoff = realized value of variable

$1 if $0 if I am entitled to:

Will US go into recession in 2012? (Y/N)

Recession in 2012

No Recession in 2012

2012

Sep

tem

ber 3

0 8:

02 p

.m. E

T"

2012

Sep

tem

ber 3

0 8:

02 p

.m. E

T"

Between 6.0% and 8.0% chance

http://www.predictwise.com/maps/2012president

Predic2on Markets Versus...

• model it - baseline • model it - baseline++ • poll a crowd - mTurk • pay a crowd - probSports contest • pay a crowd - Vegas market • pay a crowd - TradeSports market

• guess

Design for Predic2on

•  Goals for trade –  Efficiency (gains) –  Inidiv. ra2onality –  Budget balance –  Revenue –  Comp. complexity

•  Equilibrium –  General, Nash, ...

Design for Predic2on

•  Goals for trade –  Efficiency (gains) –  Inidiv. ra2onality –  Budget balance –  Revenue –  Comp. complexity

•  Equilibrium –  General, Nash, ...

•  Goals for predic2on –  Info aggrega2on –  1. Liquidity –  2. Expressiveness –  Bounded budget –  Indiv. ra2onality –  Comp. complexity

•  Equilibrium –  Ra2onal expecta2ons

Competes with:��experts, scoring rules, opinion pools, ML/stats, polls, Delphi

Why Liquidity?

Why Liquidity?

Low liquidity takes the predic2on out of markets http://blog.oddhead.com/2010/07/08/why-automated-market-makers/

Between 0.2% and 99.8% chance

Why Expressiveness?

Why Expressiveness?

•  Call op2on and put op2ons are redundant •  Range bets require four trades (“bu_erfly spread”)

•  Bid to buy call op2on @strike 15 can’t match with ask to sell @strike 10

•  Can’t set own strike •  Bo_om line: Lacks expressiveness

Why Expressiveness?

•  Dem Pres, Dem Senate, Dem House Dem Pres, Dem Senate, GOP House Dem Pres, GOP Senate, Dem House Dem Pres, GOP Senate, GOP House...

•  Dem Pres Dem House Dem wins >=270 electoral votes Dem wins >=280 electoral votes...

Industry Standard

•  Ignore rela2onships: Treat them as independent markets

•  Las Vegas sports beeng Kentucky horseracing Wall Street stock op2ons High Street spread beeng

A Be_er Way (Or,... Bringing trading into digital age) •  Expressiveness

– Linear programming – Bossaerts, Fine, Ledyard: Combined Value Trading –  http://bit.ly/multipm

•  Expressiveness + Liquidity – Automated market maker – Always quote a price on anything – Downside: requires subsidy/risk

Example: Liquidity and Expressiveness

Geeng Greedy

•  Design a market for informa2on on exponen8ally many things

•  “Combinatorial predic2on market”

Combinatorial securi2es: More informa2on, more fun

•  Payoff is func2on of common variables, e.g. 50 states elect Dem or Rep


•  Dem will win California


•  Dem will lose FL but win elec2on •  Dem will win >8 of 10 Northeastern states •  Same party will win OH & PA


•  There will be a path of blue from Canada to Mexico

Some Coun2ng

•  57 “states”: 48 + DC + Maine (3), Nebraska (5) •  257 = 144 quadrillion possible outcomes •  2257 ∼ 10144115188075848064 dis2nct predic2ons More than a googol, less than a googolplex

•  NOT independent

A research methodology

Design Build Analyze

HSX NF TS

WSEX FX PS

Examples

Design

•  Predic2on markets –  Dynamic parimutuel –  Combinatorial bids –  Combinatorial

outcomes –  Shared scoring rules –  Linear programming

backbone •  Ad auc2ons •  Spam incen2ves

Build Analyze

•  Computational complexity

•  Does money matter? •  Equilibrium analysis •  Wisdom of crowds:

Combining experts •  Practical lessons

•  Predictalot •  Yoopick •  Y!/O Buzz •  Centmail •  Pictcha •  Yootles

New Example: PredictWiseQ

•  You read the paper •  Now play the game! http://PredictWiseQ.com

•  Crazy combo predic2on market for elec2on 2012 •  Implements constraint gen market maker •  Join the CS286r group! http://bit.ly/predictwiseq-hcs

•  Earn the highest “WiseQ” in the class •  Or browse results http://www.predictwise.com/WiseQResults

http://PredictWiseQ.com

A tractable combinatorial market maker using constraint genera2on

MIROSLAV DUDÍK, SEBASTIEN LAHAIE, DAVID M. PENNOCK

Thanks: David Rothschild, Dan Osherson, Arvid Wang, Jake Abernethy, Rafael Frongillo, Rob Schapire

Automated Market Maker Exchange Market Maker

Independent Tractable No risk No info propaga2on Industry standard

Tractable Exponen2al loss bound No info propaga2on

Combinatorial NP-‐hard No risk Full info propaga2on Major liquidity problem

#P-‐hard Linear/Const loss bound Full info propaga2on

•  Info propaga2on Reward traders for information, not computational power

Automated Market Maker Exchange Market Maker

Independent Tractable No risk No info propaga2on Industry standard

Tractable Exponen2al loss bound No info propaga2on

Our approach Tractable Good loss bound Some info propaga<on

Combinatorial NP-‐hard No risk Full info propaga2on Major liquidity problem

#P-‐hard Linear/Const loss bound Full info propaga2on

•  Info propaga2on Reward traders for information, not computational power

Consistent pricing

0

1

0 1

A&B’&C

&B&

C

Independent markets

Consistent pricing

0

1

0 1

A&B’&C

&B&

C

Independent markets

Prices p

Consistent pricing

0

1

0 1

A&B’&C

&B&

C

Independent markets

Consistent pricing

0

1

0 1

A&B’&C

&B&

C B = 0.6

A = 0.8 C = 0.9

Independent markets

Consistent pricing

0

1

0 1

0.6 B = 0.6

0.8 A = 0.8

A&B’&C

&B&

C

0.9 C = 0.9

Consistent pricing

0

1

0 1

0.6 B = 0.6

0.4

0.8 A = 0.8

0.8

A&B’&C

&B&

C

0.9 C = 0.9

0.9

Approximate pricing

0

1

0 1

0.6 B = 0.6

0.4

0.8 A = 0.8

0.8

A&B’&C

&B&

C

0.9 C = 0.9

0.9

Approximate pricing

0

1

0 1

0.6 B = 0.6

0.4

Prices p 0.8 A = 0.8

0.8

A&B’&C

&B&

C

0.9 C = 0.9

0.9

Approximate pricing

0

1

0 1

0.5 B = 0.5

0.5

Buy NotB


0.8

A&B’&C

&B&

C

0.9 C = 0.9

0.9

Approximate pricing

0

1

0 1

0.5 B = 0.5

0.5


0.8

A&B’&C

&B&

C

0.9 C = 0.9

0.9

Approximate pricing

0

1

0 1

0.8

0.5

A = 0.8

B = 0.55

0.5 0.8

Prices p

A&B’&C

&B&

C

0.9 C = 0.9

0.9

For Elec2on

•  Create 50 states – ini2alize with prior •  Create all groups of 2 – init as indep •  For conjunc2ons of 3 or more, group with it opposite disjunc2on: A&B&C, A’|B’|C’

•  Each group is indep MM – fast •  In parallel: Generate, find, and fix constraints

Arbitrage and Constraints

•  Possibility of risk-‐free profit:

•  Execute trades: – Buy x shares of A – Buy x shares of B – Sell x shares of A ∪ B

Prob[A] + Prob[B] ≥ Prob[A ∪ B]

Price[A] + Price[B] − Price[A ∪ B] ≤ 0

September 26, 2012 Microsoft Research, New York City

Constraints

•  Clique lower bound P(L1|...|Lm) ≥ΣC P(Li) –ΣC P(Li&Lj)

•  Spanning tree upper bound P(L1|...|Lm) ≤Σ P(Li) –ΣT P(Li&Lj)

•  Threshold constraints TBA •  Choosing constraints is key!

– Depends on bets (unlike Monte Carlo) – An art

Does it work?

Tested on over 300K complex predic2ons from Princeton study

Budget

10 States

Does it work?


Budget

budget

score

−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●

●●

●●

●●

●

●

●

●

●

●

●

AggregateBrier

−0.14

−0.10

−0.06

−0.02

● ●●

●

●

●

●●

●●●

●

●

●

●

●

●

2−ConjunctionsBrier

−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●●

●

●●

●●

●

●

●

●

●

●

●


−0.6−0.5−0.4−0.3−0.2

●●

● ●●● ●●

●●

●

●●

●

●

●

●

AggregateLogarithmic

10^0 10^1 10^2 10^3

−0.6−0.5−0.4−0.3−0.2−0.1

● ●●

●

●●

●●

●● ●●

●

●

●

●

●

2−ConjunctionsLogarithmic

−0.5

−0.4

−0.3

−0.2

●●

● ●●● ●●

●

●

●

●

●

●

●

●

●


LocalIndependentTreeTree,Clique

●

●

budget

score

−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●

●●

●●

●●

●

●

●

●

●

●

●

AggregateBrier

−0.14

−0.10

−0.06

−0.02

● ●●

●

●

●

●●

●●●

●

●

●

●

●

●


−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●●

●

●●

●●

●

●

●

●

●

●

●


−0.6−0.5−0.4−0.3−0.2

●●

● ●●● ●●

●●

●

●●

●

●

●

●


10^0 10^1 10^2 10^3

−0.6−0.5−0.4−0.3−0.2−0.1

● ●●

●

●●

●●

●● ●●

●

●

●

●

●


−0.5

−0.4

−0.3

−0.2

●●

● ●●● ●●

●

●

●

●

●

●

●

●

●



●

●

budget

score

−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●

●●

●●

●●

●

●

●

●

●

●

●

AggregateBrier

−0.14

−0.10

−0.06

−0.02

● ●●

●

●

●

●●

●●●

●

●

●

●

●

●


−0.10

−0.08

−0.06

−0.04

10^0 10^1 10^2 10^3

●●

●●●

●

●●

●●

●

●

●

●

●

●

●


−0.6−0.5−0.4−0.3−0.2

●●

● ●●● ●●

●●

●

●●

●

●

●

●


10^0 10^1 10^2 10^3

−0.6−0.5−0.4−0.3−0.2−0.1

● ●●

●

●●

●●

●● ●●

●

●

●

●

●


−0.5

−0.4

−0.3

−0.2

●●

● ●●● ●●

●

●

●

●

●

●

●

●

●



●

●

Log

Scor

e

50 States

Does it work?


Revenue

No really, does it work?

•  Built this thing for real •  WiseQ Game for Elec2on 2012 http://PredictWiseQ.com

Predictalot alpha

Further reading

Gory details What is (and what good is) a combinatorial predic<on market? http://bit.ly/combopm Guest post on Freakonomics http://bit.ly/combopmfreak Our paper http://research.microsoft.com/apps/pubs/default.aspx?id=167977

Play PredictWiseQ

•  You read the paper •  Now play the game! http://PredictWiseQ.com

•  Crazy combo predic2on market for elec2on 2012 •  Implements constraint gen market maker •  Join the CS286r group! http://bit.ly/predictwiseq-hcs

•  Earn the highest “WiseQ” in the class •  Or browse results http://www.predictwise.com/WiseQResults

mechanism*design*for*predic2on:* combinatorial*predic2on

Documents

mechanismdesignforpredic2on: combinatorial*predic2on