3 july 2011 copyright 2011 haym hirsh 1hirsh/icml-2011-tutorial... · types of collective...
TRANSCRIPT
3 July 2011 Copyright 2011 Haym Hirsh 1
3 July 2011 Copyright 2011 Haym Hirsh 2
3 July 2011 Copyright 2011 Haym Hirsh 3
3 July 2011 Copyright 2011 Haym Hirsh 4
3 July 2011 Copyright 2011 Haym Hirsh 5
3 July 2011 Copyright 2011 Haym Hirsh 6
3 July 2011 Copyright 2011 Haym Hirsh 7
3 July 2011 Copyright 2011 Haym Hirsh 11
3 July 2011 Copyright 2011 Haym Hirsh 12
3 July 2011 Copyright 2011 Haym Hirsh 13
3 July 2011 Copyright 2011 Haym Hirsh 14
3 July 2011 Copyright 2011 Haym Hirsh 15
3 July 2011 Copyright 2011 Haym Hirsh 16
3 July 2011 Copyright 2011 Haym Hirsh 17
3 July 2011 Copyright 2011 Haym Hirsh 18
3 July 2011 Copyright 2011 Haym Hirsh 19
3 July 2011 Copyright 2011 Haym Hirsh 20
3 July 2011 Copyright 2011 Haym Hirsh 21
Collective Intelligence
3 July 2011 Copyright 2011 Haym Hirsh 22
Collective Intelligence
Swarm Intelligence
3 July 2011 Copyright 2011 Haym Hirsh 23
Collective Intelligence
Swarm Intelligence
Hive Intelligence
3 July 2011 Copyright 2011 Haym Hirsh 24
Collective Intelligence
Swarm Intelligence
Hive Intelligence
Natural Computation
3 July 2011 Copyright 2011 Haym Hirsh 25
3 July 2011 Copyright 2011 Haym Hirsh 26
3 July 2011 Copyright 2011 Haym Hirsh 27
3 July 2011 Copyright 2011 Haym Hirsh 28
3 July 2011 Copyright 2011 Haym Hirsh 29
3 July 2011 Copyright 2011 Haym Hirsh 30
3 July 2011 Copyright 2011 Haym Hirsh 31
3 July 2011 Copyright 2011 Haym Hirsh 32
3 July 2011 Copyright 2011 Haym Hirsh 33
3 July 2011 Copyright 2011 Haym Hirsh 34
Exercise 1
3 July 2011 Copyright 2011 Haym Hirsh 36
3 July 2011 Copyright 2011 Haym Hirsh 37
Exercise 2
3 July 2011 Copyright 2011 Haym Hirsh 39
3 July 2011 Copyright 2011 Haym Hirsh 40
3 July 2011 Copyright 2011 Haym Hirsh 41
3 July 2011 Copyright 2011 Haym Hirsh 42
3 July 2011 Copyright 2011 Haym Hirsh 43
3 July 2011 Copyright 2011 Haym Hirsh 44
3 July 2011 Copyright 2011 Haym Hirsh 45
3 July 2011 Copyright 2011 Haym Hirsh 46
3 July 2011 Copyright 2011 Haym Hirsh 47
3 July 2011 Copyright 2011 Haym Hirsh 48
3 July 2011 Copyright 2011 Haym Hirsh 49
3 July 2011 Copyright 2011 Haym Hirsh 50
3 July 2011 Copyright 2011 Haym Hirsh 51
3 July 2011 Copyright 2011 Haym Hirsh 52
3 July 2011 Copyright 2011 Haym Hirsh 53
3 July 2011 Copyright 2011 Haym Hirsh 54
3 July 2011 Copyright 2011 Haym Hirsh 55
3 July 2011 Copyright 2011 Haym Hirsh 56
3 July 2011 Copyright 2011 Haym Hirsh 57
3 July 2011 Copyright 2011 Haym Hirsh 58
3 July 2011 Copyright 2011 Haym Hirsh 59
3 July 2011 Copyright 2011 Haym Hirsh 60
3 July 2011 Copyright 2011 Haym Hirsh 61
3 July 2011 Copyright 2011 Haym Hirsh 62
• Introduction: Broad overview of collective intelligence, a framework for understanding it, and its various connections to machine learning – Animals to humans images/videos
– CI mosaic (icons, then list of keywords)
– CI, crowdsourcing, human computation
– Overt vs covert • Collaborative creation
• Collaborative decision making
• Smartest in the crowd / contests
• HC and micro-labor markets
• Crowd mining
– Roles of Machine Learning 3 July 2011 Copyright 2011 Haym Hirsh 63
3 July 2011 Copyright 2011 Haym Hirsh 64
Clickworkers Collaborative Filtering
Collaborative Intelligence Collaborative Systems Collective Intelligence Community Systems
Computer Assisted Proof Crowdsourcing Crowd Wisdom
Genius in the Crowd Human Computation Mass Collaboration
Participatory Journalism Peer Production
Persuasive Computing
Prediction Markets Pro-Am Collaboration
Recommender Systems Reputation Systems
Smart Mobs Social Commerce Social Computing
Social Systems Social Technology
Swarm Intelligence User Generated Content User-Powered Systems
Wikinomics Wisdom of the Crowds
3 July 2011 Copyright 2011 Haym Hirsh 65
Collective Intelligence Crowdsourcing
Human Computation Social Computing
3 July 2011 Copyright 2011 Haym Hirsh 66
“Human Computation: A Survey and Taxonomy of a Growing Field”, Alexander J. Quinn and Benjamin B. Bederson, CHI 2011,
Vancouver, BC, Canada
3 July 2011 Copyright 2011 Haym Hirsh 67
Collective Intelligence
• “groups of individuals doing things collectively that seem intelligent”
“The Collective Intelligence Genome”, Malone, Laubacher and Dellarocas, MIT Management Review, April 1, 2010
• “How can people and computers be connected so that—collectively—they act more intelligently than any individuals, groups, or computers have ever done before?”
http://cci.mit.edu
3 July 2011 Copyright 2011 Haym Hirsh 68
Crowdsourcing
• “Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.”
“The Rise of Crowdsourcing,” Jeff Howe. Wired. Jun 2006
3 July 2011 Copyright 2011 Haym Hirsh 69
Crowdsourcing
• “We say that a system is a [crowdsourcing] system if it enlists a crowd of humans to help solve a problem defined by the system owners, and if in doing so, it addresses the following four fundamental challenges: – How to recruit and retain users? – What contributions can users make? – How to combine user contributions to solve the target
problem? – How to evaluate users and their contributions?”
“Crowdsourcing systems on the World-Wide Web,” Anhai Doan, Raghu Ramakrishnan, Alon Y. Halevy. Communications of the ACM,
Vol. 54 No. 4, Pages 86-96, April 2011
3 July 2011 Copyright 2011 Haym Hirsh 70
• “The idea behind digital computers may be explained by saying that these machines are intended to carry out any operations which could be done by a human computer.”
“Computing Machinery and Intelligence.” A.M. Turing, Mind,
59:236, (1950), 433-460.
Human Computation
3 July 2011 Copyright 2011 Haym Hirsh 71
3 July 2011 Copyright 2011 Haym Hirsh 72
Human Computation
• Human computation is “a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.”
Human Computation, Luis van Ahn. Doctoral Dissertation, Department
of Computer Science, Carnegie Mellon University (2005).
3 July 2011 Copyright 2011 Haym Hirsh 73
Human Computation
• Human computation:
– The problems fit the general paradigm of computation, and as such might someday be solvable by computers.
– The human participation is directed by the computational system or process
Quinn and Bederson CHI 2011
3 July 2011 Copyright 2011 Haym Hirsh 74
Crowdsourcing vs Human Computation
• Whereas human computation replaces computers with humans, crowdsourcing replaces traditional human workers with members of the public.
Quinn and Bederson CHI 2011
3 July 2011 Copyright 2011 Haym Hirsh 75
Social Computing
• “applications and services that facilitate collective action and social interaction online with rich exchange of multimedia information and evolution of aggregate knowledge”
Parameswaran, M. & Whinston, A.B. Social Computing: An
Overview. CAIS 19:37, (2007), 762-780
3 July 2011 Copyright 2011 Haym Hirsh 76
1
2
4
8
16
32
64
128
256
512
10241
99
5
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
Collective Intelligence
Human Computation
Social Computing
Crowdsourcing
Amazon Mechanical Turk
Games with a Purpose
3 July 2011 Copyright 2011 Haym Hirsh 77
1
2
4
8
16
32
64
128
256
512
10241
99
5
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
Collective Intelligence
"Collective Intelligence"+ Human"Collective Intelligence"- Human
3 July 2011 Copyright 2011 Haym Hirsh 78
1
2
4
8
16
32
64
128
256
512
10241
99
5
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
Collective Intelligence
Human Computation
Social Computing
Crowdsourcing
Amazon Mechanical Turk
Games with a Purpose
3 July 2011 Copyright 2011 Haym Hirsh 79
1
2
4
8
16
32
64
128
256
512
1024
2006 2007 2008 2009 2010
Collective Intelligence
Human Computation
Social Computing
Crowdsourcing
Amazon Mechanical Turk
Games with a Purpose
3 July 2011 Copyright 2011 Haym Hirsh 80
Types of Collective Intelligence
• “Overt” vs “Covert”: Are human participants explicitly participating to achieve the collective outcomes, or is some form of mining of human activity achieving the collective outcomes
– Overt: Amazon reviews, Wikipedia, AMT
– Covert (“Crowd Mining”): Amazon recommendations, Google, Spam detection
3 July 2011 Copyright 2011 Haym Hirsh 81
Types of Collective Intelligence
• Overt
– Collecting
– Collaborative Creation
– Smartest in the Crowd
– Collaborative Decisions
– Human Computation and Micro-Crowdsourcing
• Covert
– Crowd Mining
3 July 2011 Copyright 2011 Haym Hirsh 82
Types of Collective Intelligence
• Collecting
3 July 2011 Copyright 2011 Haym Hirsh 83
3 July 2011 Copyright 2011 Haym Hirsh 84
Types of Collective Intelligence
• Collecting
– Amazon reviews
3 July 2011 Copyright 2011 Haym Hirsh 85
3 July 2011 Copyright 2011 Haym Hirsh 86
Types of Collective Intelligence
• Collecting
– Amazon reviews
– YouTube
3 July 2011 Copyright 2011 Haym Hirsh 87
3 July 2011 Copyright 2011 Haym Hirsh 88
Types of Collective Intelligence
• Collecting
– Amazon reviews
– YouTube
– Flickr
3 July 2011 Copyright 2011 Haym Hirsh 89
3 July 2011 Copyright 2011 Haym Hirsh 90
3 July 2011 Copyright 2011 Haym Hirsh 91
Types of Collective Intelligence
• Collecting: Assembling large collections of on-line materials
– Amazon reviews
– YouTube
– Flickr
– TripAdvisor
3 July 2011 Copyright 2011 Haym Hirsh 92
3 July 2011 Copyright 2011 Haym Hirsh 93
Types of Collective Intelligence
• Collecting: Assembling large collections of on-line materials
– Amazon reviews
– YouTube
– Flickr
– TripAdvisor
– Ten Thousand Cents
3 July 2011 Copyright 2011 Haym Hirsh 94
Types of Collective Intelligence
• Collaborative Creation: Jointly crafting on-line materials that are more holistic
3 July 2011 Copyright 2011 Haym Hirsh 95
Types of Collective Intelligence
• Collaborative Creation: Jointly crafting on-line materials that are more holistic
– Wikipedia
3 July 2011 Copyright 2011 Haym Hirsh 96
Types of Collective Intelligence
• Collaborative Creation: Jointly crafting on-line materials that are more holistic
– Wikipedia
– Open Source Software
3 July 2011 Copyright 2011 Haym Hirsh 97
3 July 2011 Copyright 2011 Haym Hirsh 98
Types of Collective Intelligence
• Collaborative Creation: Jointly crafting on-line materials that are more holistic
– Wikipedia
– Open Source Software
– Gowers’ Polymath Project
3 July 2011 Copyright 2011 Haym Hirsh 99
Types of Collective Intelligence
• Smartest in the Crowd: People compete for some prize or recognition
– Innocentive
– Goldcorp Challenge
– Threadless
– Netflix Challenge
3 July 2011 Copyright 2011 Haym Hirsh 100
Types of Collective Intelligence
• Collaborative Decisions: People’s efforts serve to make selections out of some list of options
– Iowa Electronic Markets
– Ebbsfleet United
– Kasparov against the World
3 July 2011 Copyright 2011 Haym Hirsh 101
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
3 July 2011 Copyright 2011 Haym Hirsh 102
103 3 July 2011 Copyright 2011 Haym Hirsh
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
3 July 2011 Copyright 2011 Haym Hirsh 104
3 July 2011 Copyright 2011 Haym Hirsh 105
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
– ReCAPTCHA
3 July 2011 Copyright 2011 Haym Hirsh 106
3 July 2011 Copyright 2011 Haym Hirsh 107
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
– ReCAPTCHA
– Search for Jim Gray
3 July 2011 Copyright 2011 Haym Hirsh 108
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
– ReCAPTCHA
– Search for Jim Gray
– “Games with a Purpose“ • ESP Game
3 July 2011 Copyright 2011 Haym Hirsh 109
3 July 2011 Copyright 2011 Haym Hirsh 110
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
– ReCAPTCHA
– Search for Jim Gray
– “Games with a Purpose“ • ESP Game
• fold.it
3 July 2011 Copyright 2011 Haym Hirsh 111
Types of Collective Intelligence • Human Computation and Micro-Crowdsourcing:
Individuals perform numerous small tasks that are hard for computers and that collectively solve larger, more difficult problems
– GalaxyZoo
– ReCAPTCHA
– Search for Jim Gray
– “Games with a Purpose“ • ESP Game
• fold.it
– Amazon Mechanical Turk
3 July 2011 Copyright 2011 Haym Hirsh 112
3 July 2011 Copyright 2011 Haym Hirsh 113
3 July 2011 Copyright 2011 Haym Hirsh 114
3 July 2011 Copyright 2011 Haym Hirsh 115
3 July 2011 Copyright 2011 Haym Hirsh 116
3 July 2011 Copyright 2011 Haym Hirsh 117
3 July 2011 Copyright 2011 Haym Hirsh 118
3 July 2011 Copyright 2011 Haym Hirsh 119
3 July 2011 Copyright 2011 Haym Hirsh 120
3 July 2011 Copyright 2011 Haym Hirsh 121
3 July 2011 Copyright 2011 Haym Hirsh 122
3 July 2011 Copyright 2011 Haym Hirsh 123
3 July 2011 Copyright 2011 Haym Hirsh 124
3 July 2011 Copyright 2011 Haym Hirsh 125
3 July 2011 Copyright 2011 Haym Hirsh 126
3 July 2011 Copyright 2011 Haym Hirsh 127
3 July 2011 Copyright 2011 Haym Hirsh 128
3 July 2011 Copyright 2011 Haym Hirsh 129
3 July 2011 Copyright 2011 Haym Hirsh 130
3 July 2011 Copyright 2011 Haym Hirsh 131
3 July 2011 Copyright 2011 Haym Hirsh 132
3 July 2011 Copyright 2011 Haym Hirsh 133
3 July 2011 Copyright 2011 Haym Hirsh 134
3 July 2011 Copyright 2011 Haym Hirsh 135
Types of Collective Intelligence
• Crowd Mining: Achieve desired outcomes by mining the observed behaviors of people
– Amazon recommendations
– Spam detection
3 July 2011 Copyright 2011 Haym Hirsh 136
3 July 2011 Copyright 2011 Haym Hirsh 137
3 July 2011 Copyright 2011 Haym Hirsh 138
Types of Collective Intelligence
• Crowd Mining: Achieve desired outcomes by mining the observed behaviors of people
– Amazon recommendations
– Spam Detection
– Google Flu Trends
3 July 2011 Copyright 2011 Haym Hirsh 139
3 July 2011 Copyright 2011 Haym Hirsh 140
Types of Collective Intelligence
• Crowd Mining: Achieve desired outcomes by mining the observed behaviors of people
– Amazon recommendations
– Spam Detection
– Google Flu Trends
– Google Translate
3 July 2011 Copyright 2011 Haym Hirsh 141
3 July 2011 Copyright 2011 Haym Hirsh 142
Types of Collective Intelligence
• Crowd Mining: Achieve desired outcomes by mining the observed behaviors of people
– Amazon recommendations
– Spam Detection
– Google Flu Trends
– Google Translate
– Spelling correction
3 July 2011 Copyright 2011 Haym Hirsh 143
Perspectives on Collective Intelligence: Framework Papers
3 July 2011 Copyright 2011 Haym Hirsh 144
3 July 2011 Copyright 2011 Haym Hirsh 145
Perspectives on Collective Intelligence: Leveraging Cyberspace – Kalil 1996
• Can dependencies between parts of the task be eliminated or managed?
• What will motivate people to participate?
• Is there part of the task that must be centrally administered?
• Does the initiative demonstrate increasing returns?
3 July 2011 Copyright 2011 Haym Hirsh 146
Perspectives on Collective Intelligence: Wisdom of Crowds – Surowiecki 2004
• Requires participant:
–Diversity
– Independence
–Decentralization
–Aggregation
3 July 2011 Copyright 2011 Haym Hirsh 147
Perspectives on Collective Intelligence: Malone, Laubacher and Dellarocas
• Who is doing it?
• Why are they doing it?
• How is it being done?
• What is being done?
3 July 2011 Copyright 2011 Haym Hirsh 148
Perspectives on Collective Intelligence: Malone, Laubacher and Dellarocas
• Who is doing it?
– Organization / Authority selects – Crowd: Individuals select
• Why are they doing it? – Money – Love (e.g., enjoyment, socializing) – Glory
• What is being done? – Create – Decide
3 July 2011 Copyright 2011 Haym Hirsh 149
Perspectives on Collective Intelligence: Malone, Laubacher and Dellarocas
• How is it being done?
– Create
• Collection (incl. Contest)
• Collaboration
– Decide
• Group – Voting, Consensus, Averaging, Markets
• Individual –Markets
– Social Network 3 July 2011 Copyright 2011 Haym Hirsh 150
Perspectives on Collective Intelligence: Doan, Ramakrishnan, and Halevy
• How to recruit and retain users?
• What contributions can users make?
• How to combine user contributions to solve the target problem?
• How to evaluate users and their contributions?
3 July 2011 Copyright 2011 Haym Hirsh 151
Perspectives on Collective Intelligence: Quinn and Bederson
• Motivation:
– Pay (AMT)
– Altruism (search for Jim Gray)
– Fun (ESP Game)
– Reputation (Threadless)
– Implicit Work [Coercion] (ReCAPTCHA)
– [Economical models (e.g., game theory)]
3 July 2011 Copyright 2011 Haym Hirsh 152
3 July 2011 Copyright 2011 Haym Hirsh 153
Perspectives on Collective Intelligence: Quinn and Bederson
• Reliability – Output agreement (ESP Game) – Input agreement (Tag-a-tune) – Economical models (e.g., game theory) – Defensive task design (AMT) – Reputation system (AMT, Wikipedia) – Redundancy/voting (AMT) – Ground truth seeding (AMT) – Statistical filtering/aggregating – Multilevel review (AMT) – Expert review – Automatic checking (fold.it)
3 July 2011 Copyright 2011 Haym Hirsh 154
Perspectives on Collective Intelligence: Quinn and Bederson
• Aggregation
– Collection (Wikipedia, reCaptcha)
– Statistical processing (e.g., averaging) (IEM, Wisdom of the Crowds, Kasparov against the World)
– Iterative improvement (TurkIt, Soylent)
– Active learning
– Search
– [Genetic algorithm]
– None
3 July 2011 Copyright 2011 Haym Hirsh 155
156 3 July 2011 Copyright 2011 Haym Hirsh
3 July 2011 Copyright 2011 Haym Hirsh 157
3 July 2011 Copyright 2011 Haym Hirsh 158
Perspectives on Collective Intelligence: Quinn and Bederson
• Human Skill
• Process Order
• Task Request Cardinality
3 July 2011 Copyright 2011 Haym Hirsh 159
Looking Ahead: Multi-Layered Collective Intelligence
3 July 2011 Copyright 2011 Haym Hirsh 160
3 July 2011 Copyright 2011 Haym Hirsh 161
3 July 2011 Copyright 2011 Haym Hirsh 162
3 July 2011 Copyright 2011 Haym Hirsh 163
3 July 2011 Copyright 2011 Haym Hirsh 164
3 July 2011 Copyright 2011 Haym Hirsh 165
3 July 2011 Copyright 2011 Haym Hirsh 166
Looking Ahead: Programming the Social Computer
3 July 2011 Copyright 2011 Haym Hirsh 167
168 3 July 2011 Copyright 2011 Haym Hirsh
169 3 July 2011 Copyright 2011 Haym Hirsh
170 3 July 2011 Copyright 2011 Haym Hirsh
3 July 2011 Copyright 2011 Haym Hirsh 171
3 July 2011 Copyright 2011 Haym Hirsh 172
Looking Ahead: Understanding People
3 July 2011 Copyright 2011 Haym Hirsh 173
3 July 2011 Copyright 2011 Haym Hirsh 174
3 July 2011 Copyright 2011 Haym Hirsh 175
3 July 2011 Copyright 2011 Haym Hirsh 176
3 July 2011 Copyright 2011 Haym Hirsh 177
3 July 2011 Copyright 2011 Haym Hirsh 178
3 July 2011 Copyright 2011 Haym Hirsh 179
Looking Ahead: Seeking Reliability
3 July 2011 Copyright 2011 Haym Hirsh 180
3 July 2011 Copyright 2011 Haym Hirsh 181
3 July 2011 Copyright 2011 Haym Hirsh 182
1
2
4
8
16
32
64
128
256
512
1024
2006 2007 2008 2009 2010
Crowdsourcing
Crowdsourcing reliab
Crowdsourcing quality
Crowdsourcing reputation
3 July 2011 Copyright 2011 Haym Hirsh 183
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
2006 2007 2008 2009 2010
Crowdsourcing reliab
Crowdsourcing quality
Crowdsourcingreputation
3 July 2011 Copyright 2011 Haym Hirsh 184
3 July 2011 Copyright 2011 Haym Hirsh 185
3 July 2011 Copyright 2011 Haym Hirsh 186
Looking Ahead: The Dark Side
3 July 2011 Copyright 2011 Haym Hirsh 187
3 July 2011 Copyright 2011 Haym Hirsh 188
Looking Ahead: The Dark Side
Examples:
• eBay fraud
• Fake Amazon reviews
• Gold diggers
• Griefers, trolls, sockpuppets
• Human flesh search engine
• Kasparov against the World
• Link spam
• Obama open dialog poll
• Porn monkeys
• Predicting sexual orientation from social media
• ReCAPTCHA
• Reverse engineering SSNs
• Subvert and Profit
• Trapster
• Wikipedia vandalism
3 July 2011 Copyright 2011 Haym Hirsh 189
1
2
4
8
16
32
64
128
256
512
10241
99
5
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
Collective Intelligence
Human Computation
Social Computing
Crowdsourcing
Amazon Mechanical Turk
Games with a Purpose
3 July 2011 Copyright 2011 Haym Hirsh 190
3 July 2011 Copyright 2011 Haym Hirsh 191
3 July 2011 Copyright 2011 Haym Hirsh 192