TOPICS & CONDITIONS RESTAURANT INSPECTION DATABASE HEALTH CARE KIDS' HEALTH DIETING FITNESS HOSPITAL GUIDE TOOLS/DATA EVENTS SCIENCE

2 COMMENTS

Philip Tetlock, a Penn psychology professor.

GALLERY: Fortune telling:Crowds surpass pundits

Fortune telling: Crowds surpasspundits

Melissa Dribben, Inquirer Staff Writer

POSTED: Wednesday, February 4, 2015, 1:08 AM

Experts and pundits tend to be terriblefortune tellers.

Often wrong but rarely in doubt, theybecome invested in their own theories,rejecting new information that challengestheir beliefs.

The evidence is overwhelming, from AlbertEinstein's prediction that "there is not theslightest indication that nuclear energy willever be obtainable," to George Will's thatMitt Romney would win the 2012presidential election by a landslide.

Just as solidly proven, but far less known, is that in mostcases, a group of average citizens venturing good guesses ismore likely to make accurate forecasts than a typical authorityon a subject, especially a smugly confident one.

This counterintuitive truth has fascinated social scientists,psychologists, and statisticians for more than a century. But itwas not until four years ago that the nation's intelligencecommunity decided to focus its attention ­ and largesse ­ onfiguring out how to take advantage of what has come to beknown as "the wisdom of the crowd."

Hoping to improve its accuracy forecasting critical worldevents, the federal Intelligence Advanced Research ProjectsActivity (IARPA) organized a tournament.

The agency, which is the research and development branchof the Office of the Director of National Intelligence, invitedfive academic and industry groups to enter, including onebased at the University of Pennsylvania.

Latest Science Video

Sign In | RegisterMonday, February 9, 2015

Reprints & Permissions »

More videos:

Scientists given rare glimpse of 350­kilocolossal squidAFP

The Expert ArcherfishThe New York Times

30 Philadelphia, PABolaris’ Forecast

°

| News | Sports | Entertainment | Business | Opinion | Food | Lifestyle | Health | More

Travel Deals

$309 & up ­­ 5­NightCaribbean Cruise incl.Cayman Islands

See all travel deals »

Each team was asked to predict the outcome of hundreds ofinternational political, military, and economic scenarios. Theobjective was to beat the accuracy of a control group.

The contest was supposed to last four years. But at the end ofthe second year, the Penn­led team, dubbed the GoodJudgment Project, was so far ahead of the others that all theother teams were fired, said Barbara Mellers.

She and her husband, Philip Tetlock, both psychologyprofessors at Penn, and their colleague Don Moore at theUniversity of California, Berkeley, are the force behind thatwinning team.

In last month's issue of the Journal of ExperimentalPsychology: Applied, they described how they have beenbreaking ground in the quest to make better guesses aboutthe future.

"Our study is the first to keep score and track categories ofvariables that predict performance in the politically sensitivedomain of intelligence analysis," they wrote.

The Good Judgment Project recruited thousands ofvolunteers through professional and scholarly listservs andword of mouth.

To qualify, forecasters had to be at least 18 years old. Nearlyall had finished college and many held advanced degrees.

They had to complete a battery of psychological and politicaltests that assessed what kind of forecasters they were andhow well they behaved in an open­minded way toward theevidence, Mellers said. The participants tended to be "newsjunkies," she said ­ people who "enjoy reading about details ofstories, researching them, and finding out what's going onbehind the scenes."

To optimize the team's success, Tetlock, Meller, and theircolleagues are using complex algorithms, subtle adjustments,and sophisticated evaluations.

But the main principle behind the experiment is startlinglysimple. Its origins can be traced to a Victorian country fair.

In 1906, at the West of England Fat Stock and PoultryExhibition, 800 people bought tickets for the chance to guessthe weight of a butchered ox. The scientist Francis Galton,interested in the "trustworthiness and peculiarities of popularjudgments," collected the answers, tossed out a dozen thatwere illegible or defective, and calculated the median from therest.

The result ­ 1,207 pounds ­ turned out to be a few dinnerplates' difference from the ox's true weight of 1,198 pounds.The average of the guesses was even more dead on: 1,197pounds.

The accuracy of collective thinking has been proven in study

HOVER FORCIRCULAR

HOVER FORCIRCULAR

Most Viewed Health Stories:

DNA test endorsed asPap smear alternative

C­sections could putmother and child atrisk

Jefferson's planentices employees toget fit

Hospitals strive toput spin on women'shealth

3 factors affect highblood pressureprognosis

Also on Philly.com:

NEWS:Bolaris: Freezing rain,snow this morning

BUSINESS:On Passyunk Ave., a children's­wearstar was born in 1989

after study since Galton's day.

In 2004, the journalist James Surowiecki came out with thebook The Wisdom of Crowds, reviewing many of thosestudies and exploring the phenomena in sports, stockmarkets, science, and the game show Who Wants to Be aMillionaire.

"Under the right circumstances," Surowiecki wrote, "groupsare remarkably intelligent, and are often smarter than thesmartest people in them."

From the outset, the tournament's goal was to define thosecircumstances and see if there was a way to beat the crowd'sreliably pretty good odds, said Steven Rieber, the programmanager at IARPA.

This does not mean that the CIA is planning to rely on randomgroups of citizens to predict the effect of new sanctions onIran.

"Forecasting is just one part of intelligence analysis," Riebersaid. Analysts will still need to cultivate sources, study history,and understand context.

But based on what the experiment has already found, it isclear that better forecasts may be less dependent on anindividual's deep knowledge of a particular subject than theability to think nimbly.

"Ignorance is not a virtue," Rieber said. "But subject matterexpertise doesn't matter as much as you might think."

The tournament will end this summer, he said, when theGood Judgment Project's predictions are compared withthose of a control group selected by the intelligencecommunity.

Thus far, Rieber said, the Good Judgment forecasts havebeen impressive, with nearly perfect predictions on questionssuch as "Will a significant foreign or multinational militaryforce invade or enter Iran between 17 December 2012 and 31March 2013?" (the answer was no) and "Will a Zimbabweanreferendum vote approve a new constitution before 1 April2013?" (the answer was yes).

Each of the original five teams employed brilliant thinkers in abroad range of disciplines from some of the nation's mostdistinguished universities.

Still, Rieber said, it was not terribly surprising when the GoodJudgment Project pulled ahead.

Tetlock, the author of Expert Political Judgment: How Good IsIt? How Can We Know?, has devoted most of his academiccareer to the study of expert opinion. Mellers is an authorityon decision­making and Moore's research has focused onoverconfidence.

One of the features that set their team apart from the others

SPORTS:Early look at Eagles'2015 opponents

LIFESTYLE:Keeping a sister inher place

ENTERTAINMENT:Sex on TV: It finally isfor adults

JOBS:Video: Moreapplicants lying onresumes

Stay Connected

Get the latest Healthy Week newsletter delivered toyour email. Sign up now!

Enter email address to sign up

Already a philly.com member? Yes No

was the training model they developed to teach forecastershow to choose relevant information and avoid bias. They alsodiscovered that individuals make better guesses when theyperiodically receive feedback about how successful they havebeen, and are allowed some interaction with others who areworking on the same questions.

Two years ago, the project began skimming the top 2 percentof forecasters ­ the 250 volunteers who most consistentlymade accurate predictions ­ and organized them into asubcategory of "superforecasters."

"I'm just a guinea pig in this study," said Karen Ruth Adams,an associate professor of political science at the University ofMontana, who is one of only 20 women who has earnedsuperforecaster status.

Adams was motivated to sign up, she said, by a philosophicalbelief that knowledge should be applied and theories testedbeyond the walls of academia.

During the four years she has participated in the GoodJudgment Project, Adams has worked alone as well as part ofa group.

She was surprised to learn of the gender imbalance in thepool of volunteers, she said. About 30 percent of politicalscientists, security scholars, and policy analysts in U.S. thinktanks are women, Adams said, while they constitute only 17percent of all forecasters in the Good Judgment Project.

At this point, there does not seem to be a good answer forwhy that happened, she said.

"The main problem may have been recruiting," she said. Theproject was not set up to cull through applicants to create anyparticular distribution in age, gender, expertise, or field ofinterest.

In her current challenge, Adams and her colleagues on thesuperforecaster prediction market have been given a bankrollof fake money called "inkles" to bet on the probability ofevents such as "Will Kim Jong Un meet with a major head ofstate by June?"

In September, Adams believed there was a 50 percentchance that he would, so back in August, she wagered 1,500inkles at 29 inkles per share.

One of the intellectual qualities that distinguishes asuperforecaster is the willingness to keep an open mind, toseek and integrate new information and revise predictionsbased on changing events.

Accordingly, Adams closely follows news about North Koreaand periodically adjusts her wager. When Vladimir Putininvited Kim to Russia's World War II commemoration, sheraised her odds to 75 percent and bought additional shares.In late January, when the inkle market price topped 80, she

Melissa DribbenInquirer Staff Writer

2 COMMENTS

MORE FROM THE WEB

Male Nurses in Demand in 2015 ­ Factsabout Male Nurses (Kaplan University)

11 Things You Should Never Do AgainAfter 50 (AARP)

4 Surgeries to Avoid (AARP)

Your 401(k) Isn't Growing As Fast As itShould ­ Here's Why (Mint.com)

The 7 Deadliest Soldiers In HumanHistory (TheRichest)

People With Crohn's Disease ShouldPass On These Foods (HealthCentral)

MORE FROM PHILLY.COM

Pope urges bishops to cooperate withsex abuse commission

Polaneczky: Innocent frequent flierdetained after run­in with TSA

Man with bodies in yard discussedslayings with family

Shots fired in Society Hill; teen incustody

Comcast worker on ladder leaningagainst utility pole dies

Police: Pa. dance instructor sent student,13, sexual texts

cashed out on a hunch that circumstances could get in theway of Kim's trip.

"We're just at the tip of the iceberg learning what makes for agood analyst and forecaster," she said. "You really have to bewilling to say, I was wrong."

mdribben@phillynews.com 215­854­2590

2 Comments Philly.com Login1

Sort by Best Share ⤤

Join the discussion…

Ken Rainey • 5 days ago

This article raisesmore questions than it answers for me. I will not abandon my longheld view that Einstein will do a better job on physics than 800country people at a county fair or that 800 country people at acounty fair will do better at ox weight estimating than Einstein. Ido not in any way want to belittle the study reported or the field ofresearch. The article just does not represent very well what thestudy and does not tell us.

It is not surprisingthat Einstein or George Will could sometimes be wrong. A crowd orNathan Silver might regularly do better than George Will. But allexperts and forecasters, even the very best, are sometimes wrong andfrom what the article tells us about this research is that sometimescrowds do better. Again, no surprise.

There is a bigdifference between estimating measurable things (ox weight) andpredicting future events. The Galton study for example. Eight

Favorite

Reprints & Permissions »

Share this story with friends who would like it.We recommend which of your friends would enjoy this story.

?

Fattah 'campaigning' ... for his lawyers?47 comments

1 dead in West Philadelphia triple shooting11 comments

Ferrick: Brace for Super PAC free­for­all inmayor's race 6 comments

DN Editorial: The gift that keeps on taking8 comments

AROUND THE WEB

ALSO ON PHILLY.COM

• Reply •

hundred county people at a country fair in 1908 are, in fact, a groupof experts (or pundits if you wish) in estimating ox weight. Theyshould be far better than the general population at estimating (butless good than a scale) ox weight. If those same people were askedwhen WW II would start or the distance to the moon they would nothave been nearly as close or even have provided any usefulinformation on the topics.

I would beinterested in what Nathan Silver (The Signal and the Noise) or,closer to home, Allen Palos (A Mathematician Reads the Newspaper)would have to say about this research.

• Reply •

Brandi Travis • 5 days ago

There are several prediction markets out there. PredictIt.org is a legal politicalprediction market in the US where you can play with real money. Politico called it "thestock market for politics".

WHAT'S THIS?

An Extremely Brilliant Way toPay Off Mortgage

Lifestyle Journal

11 Foods That No One ShouldEat

Diet.st

Forget 'Googling' Their Name,Use This New Site!

Instant Checkmate What Makes Your Skim MilkWhite? You Don't Want toKnow

Coupon Connections

Subscribe Add Disqus to your sited Privacy

Share ›

Share ›

comments powered by Disqus

Commenting policy | Comments FAQ

News | Sports | Entertainment | Business | Food

| Lifestyle | Health

The Inquirer Digital Edition

Subscriber Services

Daily News Digital Edition

Subscriber Services

Partners:

Philly DealYo

Parade Magazine

Classifieds:

Jobs | Cars | Real Estate | Rentals |

Marketplace | Celebrations | Print Offers

Online | Weekly Circulars

Site Services:

Advertise with us | Mobile Site | Apps | Reprints

& Permissions

Subscribe

Newspapers in Education

Internships

Subscribe

About Philly.com | Contact Us | Terms of Use & Privacy Statement | Copyright 2015

© Copyright 2015 Interstate General Media, LLC