harnessing the wisdom of crowds - … harnessing the wisdom of crowds zhi da, university of notre...

1

HARNESSING THE WISDOM OF

CROWDS

Z h i D a , U n i v e r s i t y o f N o t r e D a m e

X i n g H u a n g , M i c h i g a n S t a t e U n i v e r s i t y

S e c o n d A n n u a l N e w s & F i n a n c e C o n f e r e n c e

M a r c h 8 , 2 0 1 7

2

Many important decisions in l i fe are

made in a group

F O M C m e e t i n gB o a r d m e e t i n g

J u r y

“Who wants to be a millionaire”

A s k t h e a u d i e n c e

3

Wisdom of crowdsJ e l l y b e a n s i n t h e j a r e x p e r i m e n t

Ask a group of people to guess:

How many jelly beans in the jar?

A large group’s average answer to a question involving

quantity estimation is generally as good as, and often

better than, the answer provided by any individual in

that group

Law of large number requires independence

94

13599

77

67

97

112

102

59161

68

78

47

126

80

106

95 jelly beans

4

CONS: “MEN THINK IN HERDS”

Herding can reduce the accuracy of a group’s average answer

“The more influence we exert on each other, the more likely it is that we will believe the same things and make the same mistakes. That means it’s possible that we could become individually smarter but collectively dumber.”

---James Surowiecki, The Wisdom of Crowds.

Sequential sett ing: Pros vs. Cons

95

5“None of us is as dumb as al l of us.”

6

CONS: “MEN THINK IN HERDS”

Herding can reduce the accuracy of a group’s average answer

“The more influence we exert on each other, the more likely it is that we will believe the same things and make the same mistakes. That means it’s possible that we could become individually smarter but collectively dumber.”

---James Surowiecki, The Wisdom of Crowds.

Additional information production may improve the accuracy of private signals.

We become both individually smarter and collectively smarter.

PROS:GENERATE ADDITIONAL INFORMATION PRODUCTION

Sequential sett ing: Pros vs. Cons

95

7

WE STUDY THE QUESTION:

WE QUANTIFY THE IMPACT OF HERDING ON ECONOMIC OUTCOMES

Do individuals herd in a sequential setting and reduce the usefulness of information aggregated across individuals (i.e., the wisdom of crowds)?

The empirical challenge is that individual’s information set is usually unobservable

We overcome the empirical challenge by directly measuring and randomizing on individual’s information set

In this paper

8

WHY IT’S A GOOD SETTING?

• Forecasts and realizations are clearly measured and easily observable• The forecasters do not have direct influence on realizations• Corporate earnings are of crucial importance

Our sett ingA c r o w d - b a s e d e a r n i n g s f o r e c a s t p l a t f o r m

( E s t i m i z e . c o m )

9

M A I N A N A LY S I S

• Estimize.com related• Sample statistics

1D AT A A N D S T AT I S T I C S

2

• Herding behavior• The influence of herding on

forecast accuracyo Individual forecasto Consensus forecast

I N F L U E N T I A L U S E R A N D H E R D I N G



3R A N D O M I Z E DE X P E R I M E N T

4

• Are users herding more with “influential” users?

• Does herding lead to return predictability

Roadmap

11

Open web-based platform founded in 2011, where users can make earnings forecasts

Diverse user group: buy-side | sell-side | independent analysts | other working professionals | students

Estimize consensus is more accurate than WS | IBES consensus, also complementary

• Jame et. al. (2016) | Adebambo and Bliss (2015)• Now available on Bloomberg

Various incentives for making forecasts

• Competition, monetary and professional prizes• Reputation, useful track record• Altruism, social preference

Estimize.com

Scoring system [-25, 25]:- Positive if more accurate than WS

consensus- Awarded on an exponential scale,

which encourages independent opinion

12

Sample period for :

• Main analysis: 2012/03 - 2015/03• Randomized experiment: Q2 and Q3 in 2015

Ticker-quarter-forecast panel:

• 2147 quarterly earnings • 2516 users covering • 730 stocks: mostly large-growth

An average release (a sequence of forecasts for a ticker-quarter):

• 20 forecasts from 16 users

User activity on Estimize.com is tracked by Mixpanel

• Events: release page, estimate page, submit estimates, etc.• Timestamp, location, device, etc.

Sample statistics

13Release page view

Viewing activity = 1: If a user spent more than 5 seconds on the release page before making her own forecast

14




2







4



Roadmap

15

Note: Fixed effects subsume the need to control for stock or user characteristicsClustered standard errors account for autocorrelations in forecast errors

Herding behavior

Information weighting regression, Chen and Jiang (2006, RFS)

FE 0Dev 0 0 : overweight on public information0 0 : overweight on private information

I n d i v i d u a l s h e r d m o r e w h e n t h e y v i e w o t h e r s ’ f o r e c a s t s

More weight on the consensus forecast after viewing the release page

16

Influence of herding on forecast accuracy• Herding makes individual forecast more accurate • but reduces the accuracy of the consensus forecast

17

Note: Nonzero Views = 1: if a user spent more than 5 seconds on the release page before making her own forecast

CTA dummy = 1: if the forecast is submitted during the last three days before announcements

Individual’s absolute forecast error decreases after viewing release page

Influence of herding on forecast accuracyH e r d i n g m a k e s i n d i v i d u a l f o r e c a s t m o r e a c c u r a t e

18

Note: LnNumView = ln (1+ the percentage of forecasts with release views)Std Dev(FE)/Abs(Median(FE)) controls for uncertainty

Consensus’s absolute forecast error increases as there are more viewing activities within a release

Magnitude: 0.0551 X ln(1+1) = 3.82 centsThis is more than the distance between the perfect forecast and the forecast with median Abs(FE) (3 cents)

The consensus of group without viewing activity wins more than 50% with statistical significance

Influence of herding on forecast accuracyH e r d i n g m a k e s c o n s e n s u s f o r e c a s t l e s s a c c u r a t e

19

Note: Consistent bias indicator = 1: if the bias in both early period and CTA period are in the same direction.

LnNumView = ln (1+ the percentage of forecasts with release views)

More viewing activity in the close-to-announcement period, the biases in these two periods are more likely to be consistent.

Influence of herding on bias persistence

Early period Close-to-announcement period

20

VIEWING ACTIVITY MAY BE ENDOGENOUS

• Less informed users are more likely to view others’ forecasts.• Consistent with less accurate consensus.• But inconsistent with more accurate individual forecast.

ADDRESS THIS CONCERN:

• randomizing users’ information sets

Endogeneity concern

21




2







4



Roadmap

22

RANDOMIZED EXPERIMENTS

• Pilot round (Q2 in 2015): 13 stocks are randomly selected• Second round (Q3 in 2015): 90 stocks are randomly selected

WHAT WE DO

• Randomly select users and disable the release page • Ask them to make earnings forecasts (blind forecasts)• Afterwards, the release page is restored • They can immediately revise their forecasts (revised forecasts)• Others not selected still view the original release page and make forecasts (default forecasts)

Bl ind experiments

23Blind view

24

Note: Estimate view, Avg #releases, #tickers, and abs(FE) are based on users’ forecasts before the experiment

Users in blind and default group are similar in observable user characteristics.

Blind vs. DefaultU s e r c h a r a c t e r i s t i c s

25

HERDING BEHAVIOR

Default forecasts put more weight on Estimizeconsensus relative to blind forecasts

Blind vs. DefaultH e r d i n g a n d c o n s e n s u s a c c u r a c y

CONSENSUS ACCURACY (HORSE RACE)

• Blind consensus significantly beats default consensus 56.3% of the time• Default consensus significantly beats blind consensus 36.9% of the time

Placebo test with concurrent WS consensus

26

Note: This test only includes releases in the pilot round, because only in the pilot round, most users revise their forecasts immediately after the release page is restored.

The blind consensus is much more accurate than the revised consensus

Blind vs. RevisedC o n s e n s u s f o r e c a s t a c c u r a c y

Information weighting regression

27




2







4



Roadmap

28

Note: page rank is also how Google ranks websites.

Measuring user influenceP a g e r a n k a l g o r i t h m b a s e d o n v i e w i n g a c t i v i t i e s

D views A’s estimate

B and C are influential users in this network

• Viewed by many users in general• Viewed by another influential user

Factors behind user influence:

29

Note: Influenced = 1 if the number of influential users ahead of the observed user is above the 80th percentile across all observations

Individuals put more weight on the consensus if it contains the forecasts of influential users

Influential user and herdingI n f o r m a t i o n w e i g h t i n g r e g r e s s i o n

Robustness check using data from the experiment

30

Predict ing forecast errors of inf luent ial users

SENTIMENT-BASED MEASURE

• Influential users’ revision: upward | downward • Contemporaneous firm returns: positive | negative

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

31

Market does not seem to undo the optimism bias completely and is negatively surprised on average at the earnings announcement

Predicting earnings announcement returns

32

Herding improves the accuracy of individual forecast but reduces the accuracy of consensus forecast

Wisdom of crowds can be better harnessed by encouraging independent voice

Conclusions

Aftermath:

Our experiment results convinced Estimize.com to

switch to a “blind” platform in November 2015

harnessing the wisdom of crowds - … harnessing the wisdom of crowds zhi da, university of notre...

Documents