prediction markets and the wisdom of crowds

Download Prediction Markets and the Wisdom of Crowds

Post on 13-Jan-2016

50 views

Category:

Documents

1 download

Embed Size (px)

DESCRIPTION

Prediction Markets and the Wisdom of Crowds. David Pennock , Yahoo! Research Joint with: - PowerPoint PPT Presentation

TRANSCRIPT

  • Prediction Markets and the Wisdom of Crowds David Pennock, Yahoo! Research

    Joint with:Yiling Chen, Varsha Dani, Lance Fortnow, Ryan Fugger, Brian Galebach, Arpita Ghosh, Sharad Goel, Mingyu Guo, Joe Kilian, Nicolas Lambert, Omid Madani, Mohammad Mahdian, Eddie Nikolova, Daniel Reeves, Sumit Sanghai, Mike Wellman, Jenn Wortman

    Research

    Bet = Credible OpinionWhich is more believable? More Informative?Betting intermediariesLas Vegas, Wall Street, Betfair, Intrade,...Prices: stable consensus of a large number of quantitative, credible opinionsExcellent empirical track recordObama will win the 2008 US Presidential electionI bet $100 Obama will win at 1 to 2 odds

    Research

    A Prediction MarketTake a random variable, e.g.

    Turn it into a financial instrument payoff = realized value of variable$1 if$0 ifI am entitled to:Bin Laden captured in 2008? (Y/N)Bin Laden caught 08Bin Laden caught 08

    Research

    http://intrade.com

    Research

    OutlineThe Wisdom of CrowdsThe Wisdom of MarketsPrediction Markets: Examples & ResearchDoes Money Matter?Combinatorial BettingStorySurveyResearchResearch

    Research

    A WOC StoryProbabilitySports.comThousands of probability judgments for sporting eventsAlice: Jets 67% chance to beat PatriotsBob: Jets 48% chance to beat PatriotsCarol, Don, Ellen, Frank, ...Reward: Quadratic scoring rule: Best probability judgments maximize expected scoreSurveyStoryResearchOpinion1/7

    Research

    IndividualsMost individuals are poor predictors2005 NFL SeasonBest: 3747 pointsAverage: -944Median: -2751,298 out of 2,231 scored below zero (takes work!)

    Research

    IndividualsPoorly calibrated (too extreme)Teams given < 20% chance actually won 30% of the timeTeams given > 80% chance actually won 60% of the time

    Research

    The CrowdCreate a crowd predictor by simply averaging everyones probabilitiesCrowd = 1/n(Alice + Bob + Carol + ... )2005: Crowd scored 3371 points (7th out of 2231) !Wisdom of fools: Create a predictor by averaging everyone who scored below zero 2717 points (62nd place) !(the best fool finished in 934th place)

    Research

    The Crowd: How Big?More: http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/ http://www.overcomingbias.com/2007/02/how_and_when_to.html

    Research

    Can We Do Better?: ML/StatsMaybe NotCS experts algorithmsOther expert weightsCalibrated expertsOther averaging fns (geo mean, RMS, power means, mean of odds, ...)Machine learning (NB, SVM, LR, DT, ...)Maybe SoBayesian modeling + EMNearest neighbor (multi-year)[Dani et al. UAI 2006]

    Research

    Can we do better?: Markets

  • Prediction Markets:Examples & Research

    Research

    The Wisdom of CrowdsBacked in dollarsWhat you can say/learn % chance thatObama winsGOP wins TexasYHOO stock > 30Duke wins tourneyOil prices fallHeat index risesHurricane hits FloridaRains at place/timeWhere IEM, Intrade.comIntrade.comStock options marketLas Vegas, BetfairFutures marketWeather derivativesInsurance companyWeatherbill.com

    Research

    Prediction MarketsWith Money Without

    Research

    The Widsom of CrowdsBacked in PointsHSX.comNewsfutures.comInklingMarkets.comForesight ExchangeCasualObserver.netFTPredict.comYahoo!/OReilly Tech BuzzProTrade.comStorageMarkets.comTheSimExchange.comTheWSX.comAlexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowdshttp://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets

  • http://tradesports.comhttp://betfair.comScreen capture 2007/05/18Screen capture 2008/05/07

    Research

    Example: IEM 1992[Source: Berg, DARPA Workshop, 2002]

    Research

    Example: IEM[Source: Berg, DARPA Workshop, 2002]

    Research

    Example: IEM[Source: Berg, DARPA Workshop, 2002]

  • Does it work? Yes, evidence from real markets, laboratory experiments, and theory Racetrack odds beat track experts [Figlewski 1979] Orange Juice futures improve weather forecast [Roll 1984] I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]HP market beat sales forecast 6/8 [Plott 2000]Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC03][Schmidt 2002]Laboratory experiments confirm information aggregation [Plott 1982;1988;1997][Forsythe 1990][Chen, EC01]Theory: rational expectations [Grossman 1981][Lucas 1972]Market games work [Servan-Schreiber 2004][Pennock 2001][Thanks: Yiling Chen]

  • Prediction Markets:Does Money Matter?

    Research

    The Wisdom of CrowdsWith Money WithoutIEM: 237 CandidatesHSX: 489 Movies

    Research

    The Wisdom of CrowdsWith Money Without

    Research

    Real markets vs. market gamesHSXFX, F1P6forecast sourceavg log scoreF1P6 linear scoring-1.84F1P6 F1-style scoring-1.82betting odds-1.86F1P6 flat scoring-2.03F1P6 winner scoring-2.32

    Research

    Does money matter? Play vs real, head to headExperiment2003 NFL SeasonProbabilitySports.com Online football forecasting competitionContestants assess probabilities for each gameQuadratic scoring rule~2,000 experts, plus:NewsFutures (play $)Tradesports (real $)Used last trade pricesResults:Play money and real money performed similarly6th and 8th respectivelyMarkets beat most of the ~2,000 contestantsAverage of experts came 39th (caveat)

    Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, David Pennock and Brian Galebach

    Research

    Research

    Does money matter? Play vs real, head to headStatistically: TS ~ NF NF >> AvgTS > Avg

    Probability-Football Avg

    TradeSports(real-money)

    NewsFutures(play-money)

    DifferenceTS - NF

    Mean Absolute Error

    = lose_price

    [lower is better]

    0.443

    (0.012)

    0.439

    (0.011)

    0.436

    (0.012)

    0.003

    (0.016)

    Root Mean Squared Error

    = Average( lose_price2 )

    [lower is better]

    0.476

    (0.025)

    0.468

    (0.023)

    0.467

    (0.024)

    0.001

    (0.033)

    Average Quadratic Score

    = 100 - 400*( lose_price2 )

    [higher is better]

    9.323

    (4.75)

    12.410

    (4.37)

    12.427

    (4.57)

    -0.017

    (6.32)

    Average Logarithmic Score

    = Log(win_price)

    [higher (less negative) is better]

    -0.649

    (0.027)

    -0.631

    (0.024)

    -0.631

    (0.025)

    0.000

    (0.035)

    Research

    DiscussionAre incentives for virtual currency strong enough?Yes (to a degree)Conjecture: Enough to get what people already know; not enough to motivate independent researchReduced incentive for information discovery possibly balanced by better interpersonal weightingStatistical validations show HSX, FX, NF are reliable sources for forecastsHSX predictions >= expert predictionsCombining sources can help

    Research

    A Problem w/ Virtual CurrencyPrinting MoneyAlice 1000Betty 1000Carol 1000

    Research

    A Problem w/ Virtual CurrencyPrinting MoneyAlice 5000Betty 1000Carol 1000

    Research

    YootlesA Social CurrencyAlice 0Betty 0Carol 0

    Research

    YootlesA Social CurrencyI owe you 5Alice -5Betty 0Carol 5

    Research

    YootlesA Social Currencycredit: 5credit: 10I owe you 5Alice -5Betty 0Carol 5

    Research

    YootlesA Social Currencycredit: 5credit: 10I owe you 5I owe you 5Alice -5Betty 0Carol 5

    Research

    YootlesA Social Currencycredit: 5credit: 10I owe you 5I owe you 5Alice 3995Betty 0Carol 5

    Research

    YootlesA Social CurrencyFor tracking gratitude among friendsA yootle says thanks, I owe you one

  • Combinatorial Betting

    Research

    Combinatorics ExampleMarch Madness

    Research

    Combinatorics ExampleMarch MadnessTypical today Non-combinatorialTeam wins Rnd 1Team wins TourneyA few other propsEverything explicit (By def, small #)Every bet indep: Ignores logical & probabilistic relationshipsCombinatorialAny propertyTeam wins Rnd k Duke > {UNC,NCST} ACC wins 5 games2263 possible props (implicitly defined)1 Bet effects related bets correctly; e.g., to enforce logical constraints

  • Expressiveness:Getting InformationThings you can say today:(63% chance that) Obama winsGOP wins TexasYHOO stock > 30 Dec 2007Duke wins NCAA tourneyThings you cant say (very well) today:Oil down, DOW up, & Obama winsObama wins election, if he wins OH & FLYHOO btw 25.8 & 32.5 Dec 2007#1 seeds in NCAA tourney win more than #2 seeds

  • Expressiveness:Processing InformationIndependent markets today:Horse race win, place, & show poolsStock options at different strike pricesEvery game/proposition in NCAA tourneyAlmost everything: Stocks, wagers, intrade, ...Information flow (inference) left up to tradersBetter: Let traders focus on predicting whatever they want, however they want: Mechanism takes care of logical/probabilistic inferenceAnother advantage: Smarter budgeting

    Research

    Market CombinatoricsPermutationsA > B > C.1A > C > B.2B > A > C.1B > C > A.3C > A > B.1C > B > A.2

    Research

    Market CombinatoricsPermutationsD > A > B > C.01D > A > C > B.02D > B > A > C.01A > D > B > C.01A > D > C > B.02B > D > A > C.05A > B > D > C.01A > C > D > B.2B > A > D > C.01A > B > C > D.01A > C > B > D.02B > A > C > D.01

    D > B > C > A.05D > C > A > B.1D > C > B > A.2B > D > C > A.03C > D > A > B.1C > D > B > A.02B > C > D > A.03C > A > D > B.01C > B > D > A.02B > C > D > A.03C > A > D > B.01C > B > D > A.02

    Researc

Recommended

View more >