james carroll inmm taos technical meeting, may 31, 2012 la-ur-12-21784

Operated by Los Alamos National Security, LLC for the U.S. Department of Energy’s NNSA

U N C L A S S I F I E D

Amplifying the Wisdom of the Crowd, Building and Measuring Expert Consensus for

Nonproliferation Issues

James CarrollINMM Taos Technical Meeting,

May 31, 2012LA-UR-12-21784



Abstract

Scientific and cultural progress mostly takes place in the marketplace of ideas. The internet has largely changed how this marketplace functions. Although it has brought many improvements, it has also created a new set of problems. We propose a web service that can be used to crowdsource the process of knowledge acquisition and of knowledge summary, mediate disagreements, and improve discourse on controversial issues, thereby improving the efficiency of the marketplace of ideas. Our approach involves a specific technique for integrating: a forum; a wiki organized into separate camps, with camps organized into a hierarchical structure; a survey system based upon camp support; a mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements; and a customizable mechanism for determining how votes are weighted based on credentials or expert assignment. A version of this approach has been implemented, released as open source, and a beta test has gone live. Initial use seems to validate the merits of this approach. (U)

Slide 2



Outline

1. Epistemology and the Marketplace of Ideas2. Inefficiencies in the Marketplace of Ideas

(In the internet age)3. A Potential Partial Solution

Slide 3



1. Epistemology and the Marketplace of Ideas

Slide 4



Logic, the quest for truth:

Socrates Plato Aristotle

Slide 5



Logic, the quest for truth:

Socrates Plato Aristotle

Slide 6



The Destruction of Objectivity

David Hume (1711-1776) The Problem of Induction a question of how things behave

when they go “beyond the present testimony of the senses, and the records of our memory” [Hume1748].

The problem of Generalization (no-free-lunch).

Kurt Gödel (1906-1978) for any self-

consistent recursive axiomatic system powerful enough to describe the arithmetic of the natural numbers (for example Peano arithmetic), there are true propositions about the naturals that cannot be proved from the axioms.

Slide 7



No Free Lunch Thomas Mitchell (The Futility of Bias Free Learning):

• There are as many hypotheses consistent with past data that predict one outcome as that predict another off training set (when generalizing).

David Wolpert (The Supervised Learning No Free Lunch Theorem):• When all hypotheses are equally likely, there is no “best” supervised learning

algorithm off training set, for the 0-1 loss function.

James Carroll (No-Free-Lunch and Bayesian Optimality):• When all hypotheses are equally likely, then the “best” learning algorithm will report

that all outputs are equally likely off training set.

Slide 8



Empirical Falsifiable You can’t prove which theory is true, so stop

trying. Theories should be used that are falsifiable, but

which have not yet been falsified after concerted effort.

Such theories will be useful.

Slide 9

Karl Popper (1902-1994)



The Marketplace of Ideas

“Ideas that can get themselves accepted in a competitive market of ideas will tend to be of better quality than other ideas. The marketplace of ideas therefore improves the quality of our ideas and our thinking. If different

Slide 10

ideas are presented in an “ideas-market”, and if that market is populated by a maximum number of free agents expressing themselves freely, then those competing ideas will be exposed to a maximum number of supporting and dissenting arguments, and the balance of arguments in favor of or against an idea will be compared to the same balance for counter-ideas. The idea with the best balance will “survive”, because alternative ideas will be seen as comparatively defective, given the fact that the arguments in favor of them are weaker or the arguments against them are stronger.”



The Marketplace of Ideas This doesn’t just lead to ideas that are true, but to hypotheses that are

useful, in the sense of making the most general predictions about the most things, with the most accuracy.

These survive in the marketplace.

Slide 11



Another Rout to the Marketplace: Rout 2… Induction…. Occam’s razor solves the problem of inference.

We compute p(h|data)…. But we didn’t personally collect all the data, ourselves Nor can we all do the complex inference for every problem There are more problems you could be an expert on than there are

seconds in your lifetime. SO… we have to specialize and trade. But some people lie, so… We have to determine who to trust, and what data to trust, and

ultimately what to believe. Result: The marketplace of ideas.

Slide 12



I, Pencil, My Family Tree as told to Leonard E. Read

Slide 13



Collective Intelligence

Slide 14



Inefficiencies in the Marketplace of Ideas:

1) the limited access to information, 2) the un-manageable volume of information, 3) the reliability of information, 4) the need to determine who to consider an expert, 5) the need to determine what the experts believe, 6) the difficulty in changing popular notions that are resistant to change, 7) the incivility, repetition, and lack of focus in discourse, and 8) the difficulty in overcoming confirmation bias.

Slide 15



Current Cultural and Internet Mechanisms to Improve Marketplace Functionality:

1. Search Engines2. Peer Review3. Wikis4. Forums, Bulletin Boards, and Email Lists5. Survey Systems

Slide 16



The Canonizer Solution A wiki to leverage the wisdom of the crowd for knowledge collection and

summary• Wiki pages are organized into camps with differing opinions, instead of single pages

for each topic• Camps are organized into a hierarchical structure to encourage agreement

A mechanism for dynamically reorganizing the structure of the camp hierarchy while mediating user disagreements to encourage agreement and highlight areas of continuing disagreement

A survey system based upon camp support for determining consensus, and A mechanism for customizing how votes are weighted based on credentials or

expert assignment to allow users to explore different demographic effects, or to allow users to give extra weight to experts in different fields if they so choose.

A forum which also functions as an email list for discussion

Slide 17


U N C L A S S I F I E D Slide 18


U N C L A S S I F I E D Slide 19



Weighting Votes (hierarchy of theories of consciousness):

Theories of Mind and

Consciousness

Approachable Via Science

Representational Qualia Theory

Property Dualism

Representational Functionalism

MisIdeal Monism

MisMis

Slide 20



Weighting Votes (1 person 1 vote)


Consciousness (39)

Approachable Via Science (36)


(26.25)

Property Dualism (20)


(4.75)

Mis(1.5)

Ideal Monism (2.25)

Mis (7.5)Mis (3)

Slide 21



Weighting Votes (PhDs)


Consciousness (3)

Approachable Via Science (3)

Representational Qualia Theory (2)



(1)

Mis(0)

Ideal Monism (0)

Mis (1)Mis (3)

Slide 22



Weighting Votes (“expert” philosophers of mind)


Consciousness (10.41)

Approachable Via Science (10.41)


(10.41)

Property Dualism (9.46)


(0.94)

Mis(0.01)

Ideal Monism (0)

Mis (0)Mis (0)

Slide 23



Weighting Votes (“expert” computer scientists)


Consciousness (1.65)

Approachable Via Science (1.65)


(1.65)



(1.65)

Mis(0.00)

Ideal Monism (0)

Mis (0)Mis (0)

Slide 24



Deliverables/Results

1. A history of the debate and conversation2. A hierarchy of opinions, driven towards consensus and points of

mutual agreement3. A single page for each “camp” summarizing the arguments for and

against each position. 4. A survey of participants, showing which view has the most support5. A mechanism for varying how the votes are weighted to determine who

believes what

Slide 25



References Hume, D., 1747, “An Enquiry Concerning Human Understanding,” p. 108 Thomas Mitchell, 1980, “The Need for Biases in Learning Generalizations”,

Rutgers Computer Science. David Wolpert, 2001, “The Supervised Learning No Free Lunch Theorem”

NASA Ames Research Center. James Carroll, 2007, “No-Free-Lunch and Bayesian Optimality” in Meta-

Learning IJCNN Workshop 2007.

Slide 26

james carroll inmm taos technical meeting, may 31, 2012 la-ur-12-21784

Documents

s s i f i e doperated

s s i f i e dlogic

s s i f i e d1

los alamos national

marketplace of ideasslide

marketplace functions

expert consensus

socrates plato aristotleslide