a wisdom of the crowd approach to forecasting - funded by the

A Wisdom of the Crowd Approach to ForecastingFunded by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center

contract number D11PC20059

Brandon Turnerand

Mark Steyvers

UC, Irvine

December 17th, 2011

Turner & Steyvers (UC, Irvine) WoC Approach to Forecasting December 17th, 2011 1 / 18

The Research

UCI is one member of Team ARA, along with six other universities:


The Research

We work together toInvestigate good elicitation methodsBuild models that use this information to predict the future


The Research

Everyday people log on to a websiteThey make predictions about items (IFPs) they are interested inWe record lots of data and analyze it


The Research

The goal is to beat MITRE, a data collection company, at makingpredictions

MITRE uses the unweighted linear average on their own dataTeam ARA competes against four other teams to beat MITRE’sULinOP


The Research

The data comes in a variety of formsBinary IFPsMulti-Choice IFPsContinuous IFPs


The Research

We currently have over 50 modelsTo evaluate them, we compare them to our own ULinOpWe are now past the burn-in period


Outline

1 Wisdom of the Crowd

2 Data

3 Two Aggregation Models

4 Results

5 Conclusions/Future Directions


Wisdom of the Crowd

Outline


2 Data


4 Results



Wisdom of the Crowd

Motivation

The Wisdom of the Crowd EffectGroups of people make an estimate about a quantityThe “correctness” of these participants will varyThe mean of the estimates is better than the majority of the group


Wisdom of the Crowd

Motivation

WoC effects have been found in a variety of interesting problemsStatic judgmentsRank-ordering tasksEvent recallScene reconstructionCombinatorial problems


Wisdom of the Crowd

Motivation

Can the WoC effect be harnessed to predict the future?Build on previous “shared truth” modelsBuild on classic JDM confidence literature


Data

Outline


2 Data


4 Results



Data

Data

817 participants (general public)Provided estimates of the probability of theoccurrence of future events51 (binary) questionsJudgments made over a one-month period


Data

Complications

At first, there are no known answersQuestions are designed to eventually resolve

“Who will win the January 2012 Taiwan Presidential election?”“By 1 January 2012 will the Iraqi government sign a securityagreement that allows US troops to remain in Iraq?”

18 questions resolved during the one-month periodFocused on binary items only


Two Aggregation Models

Outline


2 Data


4 Results




Modeling Approach

Assume some latent shared truth (CCT)Model the aggregate of the judgments (WoC)Assume the shared truth is systematically inaccurateAssume a distortion occurs, prohibiting accurate forecasting

By QuestionBy Subject



Modeling Approach



New Modeling Attempts


Results

Outline


2 Data


4 Results



Results

Results

Distortion by QuestionPerformed 4.7% better than unweighted averageMean predictive error was 0.337

Distortion by SubjectPerformed 9.6% better than unweighted averageMean predictive error was 0.320


Results

Posterior Predictive Distributions


Conclusions/Future Directions

Outline


2 Data


4 Results




Conclusions

An accurate shared truth does not perform wellA distorted version of the shared truth does wellDistortion by subject is better than by question



Future (Current) Directions

Exploit non-stationarityJudgments might change over time and recent judgment might bemore accurate; track opinions over time

Recalibrate judgmentsRecalibrate individual judgments before aggregatingRecalibrate the aggregate

Exploit individual differencesEstimate expertise from resolved IFPs and user profilesMatch between user profile and IFP profile



Future (Current) Directions

Model missingnessIncorporate information about the specific IFPs a user chooses toforecast, along with information about the number of IFPs that auser forecasts

Supervised learning algorithmsEnter a large number of features in various supervised learningalgorithms, determine which are related to individuals Brier scores

Bayesian nonparametricsIsolate subgroups of users with different forecasts/opinions,aggregate based on these subgroups


a wisdom of the crowd approach to forecasting - funded by the

Documents