Download - The Wisdom of Crowds - WordPress.com...Wisdom of crowds is not a very well de ned term or topic. It describes a bunch of phenomena for which groups: perform better than each of the

The Wisdom of Crowds

Timo Freiesleben

GSN/MCMP

[email protected]

CTPS

Timo Freiesleben The Wisdom of Crowds CTPS 1 / 36

Overview

1 Experiments

2 What is the wisdom of crowds?

3 What do we require for a wise crowd?

4 Explanations for wisdom of crowdsThe Diversity Prediction TheoremCondorcet Jury TheoremsInformational Cascades

5 Opinion Leaders, Polarizers, and InfluencersThe Russian-CaseThe US-CaseThe European-Case

6 Other Interesting Networks


Experiments

How many red M&M’s are in the glass?


Experiments

For this experiment, please close your eyes while voting.

Which of the following countries has the highest population?

1 Turkey

2 Pakistan

3 Japan

4 Mexico


Experiments

For this experiment, you can keep your eyes open.

Which of the following Norwegian cities has the most inhabitants?

1 Stavanger

2 Narvik

3 Trondheim

4 Bergen


What is the wisdom of crowds?

Wisdom of crowds is not a very well defined term or topic. Itdescribes a bunch of phenomena for which groups:

perform better than each of the individuals in the group.can do tasks that individuals could not.with mediocre agents outperform groups of experts.


Examples

Francis Galton and the weight of an ox


Examples

Who wants to be a millionaire


Examples

Wikipedia


What do we require for a wise crowd?

Diversity of opinion

Independence

Decentralization/Specialization

Aggregation

Trust

See Surowiecki (2004)


Surowiecki’s ideas summarized?

It is possible to describe how people in a group think as a whole.

In some cases, groups are remarkably intelligent and are often smarterthan the smartest people in them.

The three conditions for a group to be intelligent are diversity,independence, and decentralization.

The best decisions are a product of disagreement and contest.

Too much communication can make the group as a whole lessintelligent.

There is no need to chase the expert


Explanations for wisdom of crowds

The two pictures

Truth is out there and accessible but it is blurred. Everyone draws hisor her guess from this blurred distribution.

People observe different relevant features and build a model based onthese.


The Diversity Prediction Theorem

In one informal sentence the Diversity Prediction Theorem says thefollowing:

For a group to perform well in an estimation task the individuals’performance needs to be inversely proportional to the group’s diversity.



Assume we model the agents’ guesses as as random variables Xi with

i = 1, . . . , n distributed around a value T := E(Xi ) and let C :=∑n

i=1 Xin

be the crowds average guess. Then, the following holds:

E[(T − C )2]︸︷︷︸Group

Accuracy

=1

n

n∑i=1

E[(Xi − T )2]︸︷︷︸Mean Squared

Error

− 1n

n∑i=1

E[(Xi − C )2]︸︷︷︸Diversityof Crowd



E[(T − C )2]︸︷︷︸Group

Accuracy

=1

n

n∑i=1

E[(Xi − T )2]︸︷︷︸Mean Squared

Error

− 1n

n∑i=1

E[(Xi − C )2]︸︷︷︸Diversityof Crowd

Group Accuracy≥ 0 ⇒ (MSE≥ Diversity of Crowd≥ 0)

Thus, Group Accuracy is good if

MSE and Diversity are both low or

MSE and Diversity are both high.


Example, Remark, and Critique

Remarks and Critiques:Interestingly, the theorem is not restricted to the Brier score as our distancemetric but holds true for any Bregman divergence. (Pettigrew (2007))The Theorem does not make any assumptions about the variables nor any ofthe numbers, thus, one should be very skeptical about the power of thestatement.


Condorcet Jury Theorems

Assume you have a binary decision problem in which one option iscorrect and one option is wrong. Assume moreover you have a group ofagents who vote independently and each of them has a probability p > 12to pick the right option (competence). Also, assume that the groupvotes via majority vote. Then,

group competence is monotonically increasing with group size and

group competence converges to 1 for n→∞.


Formal Version

Let (Ji )i∈N be a sequence of binary independent random variables withJi ∼ Ber(p) for all i ∈ N and p > 12 . Assume Ji : Ω→ {0, 1} wherewithout loss of generality 1 denotes the better/correct option and 0 theworse/wrong option. We define

MV2n+1 :=

1 iff2n+1∑i=1

Ji > n

0 else

Then,

for all odd numbers j holds P(MVj = 1) < P(MVj+2 = 1) and

limn→∞

P(MV2n+1 = 1) = 1.


Critiques

Attacking the premises/setting

Usually, we are not only facing binary decision problems where oneoption is correct/better than the other and also the options are oftennot given.Voters very rarely vote independent from each other.Competence might be violated.Why majority vote?Independence and Competence counteract each other. (See Dietrich(2008))

Attacking the conclusions

Even very large groups fail in picking good options empirically(democracies)Larger groups can perform worse than small groups empirically(responsibility problem)


Reactions and Generalizations

Defending the premises/settingThe Theorem can be generalized to n-ary decision problems. (See Listand Goodin (2001))We can allow for a bunch of dependencies. (See Peleg and Zamir(2011) or Pivato (2016))The theorem can be generalized to agents having differentcompetences as long as the average of competences is greater one half.(See Niztan and Paroush (1982))Other voting methods might be better in accuracy but are perceived asless fair. (See Nurmi (2002))Independence can be made plausible by conditional independence onstates. (See Dietrich & Spieckermann (2013))

Defending the conclusionsDemocracies seem to work better than dictatorships.In many contexts crowds do perform better than individuals. (SeeSurowiecki (2004))

Interestingly, the reasoning of the theorem is used in modern AItechnologies, if you are interested then have a look intoboosting/ensemble learning.Timo Freiesleben The Wisdom of Crowds CTPS 20 / 36

Informational Cascades: How groups fail

The following gives a very simple scenario of how groups can fail even in avery simple task. Assume there are two urns in a tent, a black and a white

urn.

Will this group certainly find the correct urn if the queue is infinitely longand all of the participants are Bayesians?


Informational Cascades: How groups fail

Informational cascades are one example of how groups can fail terribly.They are used as an explanation in the social sciences in a variety of

contexts as for instance for analyzing the financial crisis, the spread of fakenews, etc.


Opinion Leaders, Polarizers, and Influencers

In very many groups there is one agent present who is particularlyinfluential. Examples are democratic leaders, managers, group leaders,

class representatives, etc. Even though they usually have only one vote,they influence others due to their power, charisma, media presence, etc.

Does Condorcet’s Theorem still hold? How do groups with such influentialagents perform epistemically?


Opinion Leaders, Polarizers, and Influencers

We can model this scenario by the following Bayesian Network:

IA

MV

J1 J2 · · · J2m

Our assumption here is:

Ji |= J1, . . . , Ji−1, Ji+1, . . . , J2m|IA.


Scenario 1: Opinion Leaders (The Russian Case)

Assume our group is very homogeneous and the network is full specified bythe following assignments:

P(OL=1)=p, P(Ji |OL)=j and P(Ji |¬OL)=l with p,j ,l∈[0,1].

Then,

P(MV2m+1=1)=

(2m∑i=m

(2mi )ji (1−j)2m−i

)p+

(2m∑

i=m+1(2mi )l

i (1−l)2m−i)(1−p).

And for z:= limm→∞

P(MVm=1), we get

z l > 12 l =12 l <

12

j > 12 11+p2 p

j = 12 1−p2

12

p2

j < 12 1− p1−p2 0


Scenario 1: Opinion Leaders (The Russian Case)

In this scenario neither part (i) nor part (ii) of Condorcet’s Theoremhold in general.We can even have very pathological cases:

All agents are competent while the group is incompetent.All agents are incompetent while the group as a whole is competent.

Groups can have an optimal number of group members.

0 5 10 15 200.55

0.6

0.65

0.7

0.75

0.8

m (Number of agents is computed by 2m+1)

P(M

V_{

2m+

1}=

1)

Probability of the Majority Vote being coorect with j=0.45, l=0.9 and p=0.6.

0 20 40 60 80 1000.7

0.72

0.74

0.76

0.78

0.8

0.82

m (Number of agents is computed by 2m+1)

P(M

V_{

2m+

1}=

1)

Probability of the Majority Vote being coorect with j=0.6, l=0.45 and p=0.8.


Scenario 2: Polarizers (The US-Case)

Assume we have one polarizer and two groups influenced in different ways:

PONegatively Correlated Positively Correlated

MVs

D1 · · · Dns Rms· · ·R1



Assume our group is split and the network is fully specified by thefollowing assignments:

P(PO=1)=p, P(Ri |PO)=j1, P(Ri |¬PO)=l1, P(Di |PO)=j2 and P(Di |¬PO)=l2

with p,j1,j2,l1,l2∈[0;1].Then,

P(MVm,n=1)=

m+n∑k=m+n2

k∑i=0

( mk−i)(ni)j

k−i1 (1−j1)m−k+i j i2(1−j2)n−i

p+ m+n∑

k=m+n2 +1

k∑i=0

( mk−i)(ni)l

k−i1 (1−l1)m−k+i l i2(1−l2)n−i

(1−p).For z:= lim

s→∞P(MVs=1) we get dependent on m,n∈N and l1,l2,j1,j2∈[0;1] the

following values:z l1

mm+n + l2

nm+n >

12 l1

mm+n + l2

nm+n =

12 l1

mm+n + l2

nm+n <

12

j1m

m+n + j2n

m+n >12 1

1+p2 p

j1m

m+n + j2n

m+n =12 1−

p2

12

p2

j1m

m+n + j2n

m+n <12 1− p

1−p2 0



Given that we have similar group sizes and strong negative correlationbetween the groups, part (ii) of Condorcet’s Theorem is retained,however, it is not necessarily very stable.Also, more pathological cases occur:

Most agents in the group are competent while the group is insane.Most agents in the group are incompetent while the group reachesexpert level.

0 5 10 15 200.2

0.3

0.4

0.5

0.6

0.7

0.8

s

P(M

V_{

s}=

1)

Probability of the Majority Vote being coorect with j_1=0.95j_2=0.4, l_1=0.35, l_2=0.8, m=3, n=5 and p=0.3.

0 5 10 15 200.25

0.3

0.35

0.4

0.45

0.5

0.55

s

P(M

V_{

s}=

1)

Probability of the Majority Vote being coorect with j_1=0.95j_2=0.4, l_1=0.3, l_2=0.8, m=3, n=5 and p=0.3.


Scenario 3: Influencers (The European-Case)

Assume we have one influencer and several groups influenced in differentways:

IN

MVn

G1

G2

. ..

Gn

Variety of dependencies



Assume our group is split into n different subgroups and the network isfully specified by the following assignments:

P(IN=1)=p, P(Jg,i |IN)=jg , P(Jg,i |¬IN)=lg

with p,jg ,lg∈[0;1] for all 1≤g≤n.Then,

P(MVn=1)=

w1+···+wn∑k=

w1+···+wn2

∑A∈Bk

∏i∈A

j∗i∏

u∈AC(1−j∗u )

p+ w1+···+wn∑

k=w1+···+wn

2 +1

∑A∈Bk

∏i∈A

l∗i∏

u∈AC(1−l∗u )

(1−p) (1)For z:= lim

n→∞P(MVn=1) we get dependent on wg∈ and lg ,jg∈[0;1] for all g∈N:

z∞∑i=1

si li >12

∞∑i=1

si li =12

∞∑i=1

si li <12

∞∑i=1

si ji >12 1

1+p2 p

∞∑i=1

si ji =12 1−

p2

12

p2

∞∑i=1

si ji <12 1− p

1−p2 0



Condorcet Diversity Theorem

Assume p ∈ [0, 1] and every subgroup has the same size wi = wj for alli , j ∈ N. Let α > 12 denote the competence of each individual. If for everysubgroup g ∈ holds that

jg ∼ Unif([max(0,α + p − 1

p),min(1,

α

p)])

with lg :=α−jgp1−p we obtain

1 limn→∞

P(MVn = 1) = 1 and

2 E(MVn) ≤ E(MVn+1)

Hence, infallibility remains valid as long as dependencies vary andindividuals are competent. Even for finite groups we have an epistemicgain in expectation by adding another group with above properties.


A lot of highly un(der)explored networks are awaiting. Ideas?

Cambridge Analytica

Facebook

Election

U1

U2

. ..

Un


Rumor Spread

P1

P5P2

P3

P4

P6


Insider Trading

E1

EZBE2

T3

T4

BlackRock


Thank you for your attention!


References


Berend D. and Paroush J. (1998), When Is Condorcet’s Jury TheoremValid?, Social Choice and Welfare 15: pp 481-488.

Berg, S. (1993), Condorcet’s jury theorem, dependency among jurors,Soc Choice Welf 17: pp 87-96.

Berg, S. (1993), Condorcet’s jury theorem revisited, Eur J PoliticalEcon 9: pp 437-446.

Boland P. J., Proschan F., and Tong Y. L. (1989), ModellingDependence in Simple and Indirect Majority Systems, Journal ofApplied Probability 26: pp 81-88.

Boland P. J. (1989), Majority Systems and the Condorcet JuryTheorem, Statistician 38: pp 181-189.

Jean-Antoine-Nicolas de Caritat Condorcet, marquis de: Essai surl’application de l’analyse à la probabilité des décisions rendues à lapluralité des voix. Imprimerie royale, Paris 1785

Dietrich, F. and List, C. (2004), A model of jury decision where all thejurors have the same evidence, Synthese 142: pp 175-202.


Dietrich, F. (2008), The premises of Condorcet’s theorem are notsimultaneously justified, Episteme Volume 5 Issue 1: pp 56-73.

Dietrich, F. Spiekermann, K. (2013) “ EPISTEMIC DEMOCRACYWITH DEFENSIBLE PREMISES”, Economics and Philosophy,29(1),pp 87-120.

Estlund, D.M. (1994), Opinion Leaders, Independence, andCondorcet’s Jury Theorem, Theory and Decision 36(2): pp 131-162.

Galton, F. (1907). ”Vox populi”. Nature. 75 (1949): 450–451.Bibcode:1907Natur..75..450G. doi:10.1038/075450a0.

Grofman, B., Guillermo O. and Scott L. F. (1983), Thirteen Theoremsin Search of the Truth, Theory and Decision 15: pp 261-278.

May, K. (1952) “A set of independent necessary and sufficientconditions for simple majority decisions”, Econometrica, Vol 20 (4):pp. 680-684.

Niztan, S. and Paroush, J. (1982), Optimal decision rules indichotomous choice situations. Int Econ Rev 23: pp 289-297.


Nurmi, Hannu (2002), Voting Procedures under Uncertainty,Springer-Verlag.

Page, S. E. (2007). Making the difference: Applying a logic ofdiversity, Academy of Management Perspectives, 21(4), 6-20.

Peleg, B and Zamir, S. (2011), Extending the Condorcet JuryTheorem to a general dependent jury, Soc Choice Welf 39: pp 91-125.

Shapley, LS and Grofman, B. (1984), Optimizing group judgementalaccuracy in the presence of interdependencies, Public Choice 43: pp329-343.

Shirari, S.N. (1982), Reliability analysis of majority vote systems,Inform. Sci. 26, pp. 243-256.

Surowiecki, James. The Wisdom of Crowds. Doubleday, 2004. ISBN978-0-385-50386-0.


ExperimentsWhat is the wisdom of crowds?What do we require for a wise crowd?Explanations for wisdom of crowdsThe Diversity Prediction TheoremCondorcet Jury TheoremsInformational Cascades

Opinion Leaders, Polarizers, and InfluencersThe Russian-CaseThe US-CaseThe European-Case

Other Interesting NetworksAppendix

Download - The Wisdom of Crowds - WordPress.com...Wisdom of crowds is not a very well de ned term or topic. It describes a bunch of phenomena for which groups: perform better than each of the

Top Related