# the wisdom of crowds - ... wisdom of crowds is not a very well de ned term or topic. it describes a

Post on 27-Jan-2021

0 views

Embed Size (px)

TRANSCRIPT

The Wisdom of Crowds

Timo Freiesleben

GSN/MCMP

timo.freiesleben@campus.lmu.de

CTPS

Timo Freiesleben The Wisdom of Crowds CTPS 1 / 36

Overview

1 Experiments

2 What is the wisdom of crowds?

3 What do we require for a wise crowd?

4 Explanations for wisdom of crowds The Diversity Prediction Theorem Condorcet Jury Theorems Informational Cascades

5 Opinion Leaders, Polarizers, and Influencers The Russian-Case The US-Case The European-Case

6 Other Interesting Networks

Timo Freiesleben The Wisdom of Crowds CTPS 2 / 36

Experiments

How many red M&M’s are in the glass?

Timo Freiesleben The Wisdom of Crowds CTPS 3 / 36

Experiments

For this experiment, please close your eyes while voting.

Which of the following countries has the highest population?

1 Turkey

2 Pakistan

3 Japan

4 Mexico

Timo Freiesleben The Wisdom of Crowds CTPS 4 / 36

Experiments

For this experiment, you can keep your eyes open.

Which of the following Norwegian cities has the most inhabitants?

1 Stavanger

2 Narvik

3 Trondheim

4 Bergen

Timo Freiesleben The Wisdom of Crowds CTPS 5 / 36

What is the wisdom of crowds?

Wisdom of crowds is not a very well defined term or topic. It describes a bunch of phenomena for which groups:

perform better than each of the individuals in the group. can do tasks that individuals could not. with mediocre agents outperform groups of experts.

Timo Freiesleben The Wisdom of Crowds CTPS 6 / 36

Examples

Francis Galton and the weight of an ox

Timo Freiesleben The Wisdom of Crowds CTPS 7 / 36

Examples

Who wants to be a millionaire

Timo Freiesleben The Wisdom of Crowds CTPS 8 / 36

Examples

Wikipedia

Timo Freiesleben The Wisdom of Crowds CTPS 9 / 36

What do we require for a wise crowd?

Diversity of opinion

Independence

Decentralization/Specialization

Aggregation

Trust

See Surowiecki (2004)

Timo Freiesleben The Wisdom of Crowds CTPS 10 / 36

Surowiecki’s ideas summarized?

It is possible to describe how people in a group think as a whole.

In some cases, groups are remarkably intelligent and are often smarter than the smartest people in them.

The three conditions for a group to be intelligent are diversity, independence, and decentralization.

The best decisions are a product of disagreement and contest.

Too much communication can make the group as a whole less intelligent.

There is no need to chase the expert

Timo Freiesleben The Wisdom of Crowds CTPS 11 / 36

Explanations for wisdom of crowds

The two pictures

Truth is out there and accessible but it is blurred. Everyone draws his or her guess from this blurred distribution.

People observe different relevant features and build a model based on these.

Timo Freiesleben The Wisdom of Crowds CTPS 12 / 36

The Diversity Prediction Theorem

In one informal sentence the Diversity Prediction Theorem says the following:

For a group to perform well in an estimation task the individuals’ performance needs to be inversely proportional to the group’s diversity.

Timo Freiesleben The Wisdom of Crowds CTPS 13 / 36

The Diversity Prediction Theorem

Assume we model the agents’ guesses as as random variables Xi with

i = 1, . . . , n distributed around a value T := E(Xi ) and let C := ∑n

i=1 Xi n

be the crowds average guess. Then, the following holds:

E[(T − C )2]︸ ︷︷ ︸ Group

Accuracy

= 1

n

n∑ i=1

E[(Xi − T )2]︸ ︷︷ ︸ Mean Squared

Error

− 1 n

n∑ i=1

E[(Xi − C )2]︸ ︷︷ ︸ Diversity of Crowd

Timo Freiesleben The Wisdom of Crowds CTPS 14 / 36

The Diversity Prediction Theorem

E[(T − C )2]︸ ︷︷ ︸ Group

Accuracy

= 1

n

n∑ i=1

E[(Xi − T )2]︸ ︷︷ ︸ Mean Squared

Error

− 1 n

n∑ i=1

E[(Xi − C )2]︸ ︷︷ ︸ Diversity of Crowd

Group Accuracy≥ 0 ⇒ (MSE≥ Diversity of Crowd≥ 0)

Thus, Group Accuracy is good if

MSE and Diversity are both low or

MSE and Diversity are both high.

Timo Freiesleben The Wisdom of Crowds CTPS 15 / 36

Example, Remark, and Critique

Remarks and Critiques: Interestingly, the theorem is not restricted to the Brier score as our distance metric but holds true for any Bregman divergence. (Pettigrew (2007)) The Theorem does not make any assumptions about the variables nor any of the numbers, thus, one should be very skeptical about the power of the statement.

Timo Freiesleben The Wisdom of Crowds CTPS 16 / 36

Condorcet Jury Theorems

Assume you have a binary decision problem in which one option is correct and one option is wrong. Assume moreover you have a group of agents who vote independently and each of them has a probability p > 12 to pick the right option (competence). Also, assume that the group votes via majority vote. Then,

group competence is monotonically increasing with group size and

group competence converges to 1 for n→∞.

Timo Freiesleben The Wisdom of Crowds CTPS 17 / 36

Formal Version

Let (Ji )i∈N be a sequence of binary independent random variables with Ji ∼ Ber(p) for all i ∈ N and p > 12 . Assume Ji : Ω→ {0, 1} where without loss of generality 1 denotes the better/correct option and 0 the worse/wrong option. We define

MV2n+1 :=

1 iff 2n+1∑ i=1

Ji > n

0 else

Then,

for all odd numbers j holds P(MVj = 1) < P(MVj+2 = 1) and

lim n→∞

P(MV2n+1 = 1) = 1.

Timo Freiesleben The Wisdom of Crowds CTPS 18 / 36

Critiques

Attacking the premises/setting

Usually, we are not only facing binary decision problems where one option is correct/better than the other and also the options are often not given. Voters very rarely vote independent from each other. Competence might be violated. Why majority vote? Independence and Competence counteract each other. (See Dietrich (2008))

Attacking the conclusions

Even very large groups fail in picking good options empirically (democracies) Larger groups can perform worse than small groups empirically (responsibility problem)

Timo Freiesleben The Wisdom of Crowds CTPS 19 / 36

Reactions and Generalizations

Defending the premises/setting The Theorem can be generalized to n-ary decision problems. (See List and Goodin (2001)) We can allow for a bunch of dependencies. (See Peleg and Zamir (2011) or Pivato (2016)) The theorem can be generalized to agents having different competences as long as the average of competences is greater one half. (See Niztan and Paroush (1982)) Other voting methods might be better in accuracy but are perceived as less fair. (See Nurmi (2002)) Independence can be made plausible by conditional independence on states. (See Dietrich & Spieckermann (2013))

Defending the conclusions Democracies seem to work better than dictatorships. In many contexts crowds do perform better than individuals. (See Surowiecki (2004))

Interestingly, the reasoning of the theorem is used in modern AI technologies, if you are interested then have a look into boosting/ensemble learning. Timo Freiesleben The Wisdom of Crowds CTPS 20 / 36

Informational Cascades: How groups fail

The following gives a very simple scenario of how groups can fail even in a very simple task. Assume there are two urns in a tent, a black and a white

urn.

Will this group certainly find the correct urn if the queue is infinitely long and all of the participants are Bayesians?

Timo Freiesleben The Wisdom of Crowds CTPS 21 / 36

Informational Cascades: How groups fail

Informational cascades are one example of how groups can fail terribly. They are used as an explanation in the social sciences in a variety of

contexts as for instance for analyzing the financial crisis, the spread of fake news, etc.

Timo Freiesleben The Wisdom of Crowds CTPS 22 / 36

Opinion Leaders, Polarizers, and Influencers

In very many groups there is one agent present who is particularly influential. Examples are democratic leaders, managers, group leaders,

class representatives, etc. Even though they usually have only one vote, they influence others due to their power, charisma, media presence, etc.

Does Condorcet’s Theorem still hold? How do groups with such influential agents perform epistemically?

Timo Freiesleben The Wisdom of Crowds CTPS 23 / 36

Opinion Leaders, Polarizers, and Influencers

We can model this scenario by the following Bayesian Network:

IA

MV

J1 J2 · · · J2m

Our assumption here is:

Ji |= J1, . . . , Ji−1, Ji+1, . . . , J2m|IA.

Timo Freiesleben The Wisdom of Crowds CTPS 24 / 36

Scenario 1: Opinion Leaders (The Russian Case)

Assume our gr