the wisdom of crowds - ... wisdom of crowds is not a very well de ned term or topic. it describes a

Download The Wisdom of Crowds - ... Wisdom of crowds is not a very well de ned term or topic. It describes a

Post on 27-Jan-2021

0 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • The Wisdom of Crowds

    Timo Freiesleben

    GSN/MCMP

    timo.freiesleben@campus.lmu.de

    CTPS

    Timo Freiesleben The Wisdom of Crowds CTPS 1 / 36

  • Overview

    1 Experiments

    2 What is the wisdom of crowds?

    3 What do we require for a wise crowd?

    4 Explanations for wisdom of crowds The Diversity Prediction Theorem Condorcet Jury Theorems Informational Cascades

    5 Opinion Leaders, Polarizers, and Influencers The Russian-Case The US-Case The European-Case

    6 Other Interesting Networks

    Timo Freiesleben The Wisdom of Crowds CTPS 2 / 36

  • Experiments

    How many red M&M’s are in the glass?

    Timo Freiesleben The Wisdom of Crowds CTPS 3 / 36

  • Experiments

    For this experiment, please close your eyes while voting.

    Which of the following countries has the highest population?

    1 Turkey

    2 Pakistan

    3 Japan

    4 Mexico

    Timo Freiesleben The Wisdom of Crowds CTPS 4 / 36

  • Experiments

    For this experiment, you can keep your eyes open.

    Which of the following Norwegian cities has the most inhabitants?

    1 Stavanger

    2 Narvik

    3 Trondheim

    4 Bergen

    Timo Freiesleben The Wisdom of Crowds CTPS 5 / 36

  • What is the wisdom of crowds?

    Wisdom of crowds is not a very well defined term or topic. It describes a bunch of phenomena for which groups:

    perform better than each of the individuals in the group. can do tasks that individuals could not. with mediocre agents outperform groups of experts.

    Timo Freiesleben The Wisdom of Crowds CTPS 6 / 36

  • Examples

    Francis Galton and the weight of an ox

    Timo Freiesleben The Wisdom of Crowds CTPS 7 / 36

  • Examples

    Who wants to be a millionaire

    Timo Freiesleben The Wisdom of Crowds CTPS 8 / 36

  • Examples

    Wikipedia

    Timo Freiesleben The Wisdom of Crowds CTPS 9 / 36

  • What do we require for a wise crowd?

    Diversity of opinion

    Independence

    Decentralization/Specialization

    Aggregation

    Trust

    See Surowiecki (2004)

    Timo Freiesleben The Wisdom of Crowds CTPS 10 / 36

  • Surowiecki’s ideas summarized?

    It is possible to describe how people in a group think as a whole.

    In some cases, groups are remarkably intelligent and are often smarter than the smartest people in them.

    The three conditions for a group to be intelligent are diversity, independence, and decentralization.

    The best decisions are a product of disagreement and contest.

    Too much communication can make the group as a whole less intelligent.

    There is no need to chase the expert

    Timo Freiesleben The Wisdom of Crowds CTPS 11 / 36

  • Explanations for wisdom of crowds

    The two pictures

    Truth is out there and accessible but it is blurred. Everyone draws his or her guess from this blurred distribution.

    People observe different relevant features and build a model based on these.

    Timo Freiesleben The Wisdom of Crowds CTPS 12 / 36

  • The Diversity Prediction Theorem

    In one informal sentence the Diversity Prediction Theorem says the following:

    For a group to perform well in an estimation task the individuals’ performance needs to be inversely proportional to the group’s diversity.

    Timo Freiesleben The Wisdom of Crowds CTPS 13 / 36

  • The Diversity Prediction Theorem

    Assume we model the agents’ guesses as as random variables Xi with

    i = 1, . . . , n distributed around a value T := E(Xi ) and let C := ∑n

    i=1 Xi n

    be the crowds average guess. Then, the following holds:

    E[(T − C )2]︸ ︷︷ ︸ Group

    Accuracy

    = 1

    n

    n∑ i=1

    E[(Xi − T )2]︸ ︷︷ ︸ Mean Squared

    Error

    − 1 n

    n∑ i=1

    E[(Xi − C )2]︸ ︷︷ ︸ Diversity of Crowd

    Timo Freiesleben The Wisdom of Crowds CTPS 14 / 36

  • The Diversity Prediction Theorem

    E[(T − C )2]︸ ︷︷ ︸ Group

    Accuracy

    = 1

    n

    n∑ i=1

    E[(Xi − T )2]︸ ︷︷ ︸ Mean Squared

    Error

    − 1 n

    n∑ i=1

    E[(Xi − C )2]︸ ︷︷ ︸ Diversity of Crowd

    Group Accuracy≥ 0 ⇒ (MSE≥ Diversity of Crowd≥ 0)

    Thus, Group Accuracy is good if

    MSE and Diversity are both low or

    MSE and Diversity are both high.

    Timo Freiesleben The Wisdom of Crowds CTPS 15 / 36

  • Example, Remark, and Critique

    Remarks and Critiques: Interestingly, the theorem is not restricted to the Brier score as our distance metric but holds true for any Bregman divergence. (Pettigrew (2007)) The Theorem does not make any assumptions about the variables nor any of the numbers, thus, one should be very skeptical about the power of the statement.

    Timo Freiesleben The Wisdom of Crowds CTPS 16 / 36

  • Condorcet Jury Theorems

    Assume you have a binary decision problem in which one option is correct and one option is wrong. Assume moreover you have a group of agents who vote independently and each of them has a probability p > 12 to pick the right option (competence). Also, assume that the group votes via majority vote. Then,

    group competence is monotonically increasing with group size and

    group competence converges to 1 for n→∞.

    Timo Freiesleben The Wisdom of Crowds CTPS 17 / 36

  • Formal Version

    Let (Ji )i∈N be a sequence of binary independent random variables with Ji ∼ Ber(p) for all i ∈ N and p > 12 . Assume Ji : Ω→ {0, 1} where without loss of generality 1 denotes the better/correct option and 0 the worse/wrong option. We define

    MV2n+1 :=

    1 iff 2n+1∑ i=1

    Ji > n

    0 else

    Then,

    for all odd numbers j holds P(MVj = 1) < P(MVj+2 = 1) and

    lim n→∞

    P(MV2n+1 = 1) = 1.

    Timo Freiesleben The Wisdom of Crowds CTPS 18 / 36

  • Critiques

    Attacking the premises/setting

    Usually, we are not only facing binary decision problems where one option is correct/better than the other and also the options are often not given. Voters very rarely vote independent from each other. Competence might be violated. Why majority vote? Independence and Competence counteract each other. (See Dietrich (2008))

    Attacking the conclusions

    Even very large groups fail in picking good options empirically (democracies) Larger groups can perform worse than small groups empirically (responsibility problem)

    Timo Freiesleben The Wisdom of Crowds CTPS 19 / 36

  • Reactions and Generalizations

    Defending the premises/setting The Theorem can be generalized to n-ary decision problems. (See List and Goodin (2001)) We can allow for a bunch of dependencies. (See Peleg and Zamir (2011) or Pivato (2016)) The theorem can be generalized to agents having different competences as long as the average of competences is greater one half. (See Niztan and Paroush (1982)) Other voting methods might be better in accuracy but are perceived as less fair. (See Nurmi (2002)) Independence can be made plausible by conditional independence on states. (See Dietrich & Spieckermann (2013))

    Defending the conclusions Democracies seem to work better than dictatorships. In many contexts crowds do perform better than individuals. (See Surowiecki (2004))

    Interestingly, the reasoning of the theorem is used in modern AI technologies, if you are interested then have a look into boosting/ensemble learning. Timo Freiesleben The Wisdom of Crowds CTPS 20 / 36

  • Informational Cascades: How groups fail

    The following gives a very simple scenario of how groups can fail even in a very simple task. Assume there are two urns in a tent, a black and a white

    urn.

    Will this group certainly find the correct urn if the queue is infinitely long and all of the participants are Bayesians?

    Timo Freiesleben The Wisdom of Crowds CTPS 21 / 36

  • Informational Cascades: How groups fail

    Informational cascades are one example of how groups can fail terribly. They are used as an explanation in the social sciences in a variety of

    contexts as for instance for analyzing the financial crisis, the spread of fake news, etc.

    Timo Freiesleben The Wisdom of Crowds CTPS 22 / 36

  • Opinion Leaders, Polarizers, and Influencers

    In very many groups there is one agent present who is particularly influential. Examples are democratic leaders, managers, group leaders,

    class representatives, etc. Even though they usually have only one vote, they influence others due to their power, charisma, media presence, etc.

    Does Condorcet’s Theorem still hold? How do groups with such influential agents perform epistemically?

    Timo Freiesleben The Wisdom of Crowds CTPS 23 / 36

  • Opinion Leaders, Polarizers, and Influencers

    We can model this scenario by the following Bayesian Network:

    IA

    MV

    J1 J2 · · · J2m

    Our assumption here is:

    Ji |= J1, . . . , Ji−1, Ji+1, . . . , J2m|IA.

    Timo Freiesleben The Wisdom of Crowds CTPS 24 / 36

  • Scenario 1: Opinion Leaders (The Russian Case)

    Assume our gr

Recommended

View more >