Wisdom (and Folly) of the CrowdsJessica Keup

OutlineDefinitionsUsesPros/ConsCrowdsourcing MarketplacesCrowdsourcing Research

History“Crowdsourcing is a method of solving problems with, and outsourcing work, to the collective online intelligence”

James Surowiecki - The Wisdom of Crowds, 2005

History“Remember outsourcing? Sending jobs to India and China is so 2003. The new pool of cheap labor: everyday people using their spare cycles to create content, solve problems, even do corporate R & D.”

Jeff Howe - The Rise of Crowdsourcing, 2006

Crowdsourcing vs . . .Outsourcing

◦work given to pre-decided person or group

Open Source◦initiated by public, not a client

RequestSubmission

s/Work

Reward

Examples?Digg/Reddit iStockPhoto Netflix prizeThreadlessReCAPTCHAGoogle image labelerEquitySplash / Piqqem

Best UsesImage taggingSearch relevance evaluationContent generationProduct categorizationCollecting feedback and reviewsTranscriptions

ProsLabor at reduced costAround-the-clockHuge potential workforceVariety of opinions, ideas,

experienceConsensus and better answers

ConsCheating/manipulation

Inexperience/ignorance

Confidentiality

FixesCheating/manipulation

◦corroboration, ratingsInexperience/ignorance

◦qualification testsConfidentiality

◦?

Crowdsourcing MarketplacesCrowdFlowerMechanical TurkCloud Crowd BootbcrowdSPRING Innocentive TopCoder Kluster

mTurk “Artificial Artificial Intelligence” Human Intelligence Tasks (HITs) $0.00 and up, mostly a few cents SDKs in Java, Ruby, Perl, .NET Or, create through website UI*

Training AI“Computer Music Composition

using Crowdsourcing and Genetic Algorithms”

DarwinTunes

Research Examples collecting creative drawings rating audio and video sample

quality tagging location-sensitive queries

and points of interest with mobile devices

remote massage task delegation in Wikipedia collecting volunteer expertise to

help the homeless annotating political campaign ads

More Research examples describing and organizing

geometric shapes* tracking popular music rating search result relevance collecting user feedback extracting product and brand

information tagging specific objects in an

image training speech recognition

software

Questions?

References Alonso, O., Rose, D.E., and Stewart, B. (2008). Crowdsourcing for Relevance

Evaluation. SIGIR Forum 42(2) 9-15. Amazon.com, Inc. (2010). Amazon Mechanical Turk.

https://www.mturk.com/mturk/welcome. CastingWords (2010). Audio transcription services: plus video, CDs, podcasts and

more. http://castingwords.com/ Chung, K., Chiu, C., Xiao, X., & Chi, P. (2009). Stress outsourced: A haptic social

network via crowdsourcing. Paper presented at the CHI EA '09: Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems, Boston, MA, USA. 2439-2448. 

DarwinTunes (2010). Welcome to DarwinTunes. http://darwintunes.org/ Ganjisaffar, Y., Javanmardi, S., & Lopes, C. (2009). Leveraging crowdsourcing

heuristics to improve search in wikipedia. Paper presented at the WikiSym '09: Proceedings of the 5th International Symposium on Wikis and Open Collaboration, Orlando, Florida. 1-2.

Harper, F. M., Raban, D., Rafaeli, S. & Konstan, J. A. (2008). Predictors of answer quality in online Q & A sites. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 865-874.

Hintikka, K. A. (2008). Web 2.0 and the collective intelligence. Paper presented at the MindTrek '08: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era, Tampere, Finland. 163-166.

Howe, J. (2006). The Rise of Crowdsourcing. Wired, 14(6). http://www.wired.com/wired/archive/14.06/crowds.html. 

More References Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A

study of annotation selection criteria. Paper presented at the HLT '09: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, Boulder, Colorado. 27-35.

Jagadeesan, A. P., Lynn, A., Corney, J. R., Yan, X. T., Wenzel, J., Sherlock, A., et al. (2009). Geometric reasoning via internet CrowdSourcing. Paper presented at the SPM '09: 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling, San Francisco, California. 313-318.

Kittur, A., Chi, ,E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 453-456.

Koblin, A. M. (2009). The sheep market. Paper presented at the C&C '09: Proceeding of the Seventh ACM Conference on Creativity and Cognition, Berkeley, California, USA. 451-452.

Krieger, M., Stark, E. M., & Klemmer, S. R. (2009). Coordinating tasks on the commons: Designing for personal goals, expertise and serendipity. Paper presented at the CHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems, Boston, MA, USA. 1485-1494.

Ledlie, J., Odero, B., Minkov, E., Kiss, I., & Polifroni, J. (2009). Crowd translator: On building localized speech recognizers through micropayments. SIGOPS Operating.Systems Review, 43(4), 84-89.

Li, C., Buyuktur, A. G., Hutchful, D. K., Sant, N. B., & Nainwal, S. K. (2008). Portalis: Using competitive online interactions to support aid initiatives for the homeless. Paper presented at the CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy. 3873-3878.

Even More References Mannes, A. E. (2009). Are we wise about the wisdom of crowds? the use of group

judgments in belief revision. Management Science, 55(8), 1267-1279.  Mason, W., & Watts, D. J. (2009). Financial incentives and the performance of

crowds. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 77-85. 

Milne, D., & Witten, I. H. (2008). Learning to link with wikipedia. Paper presented at the CIKM '08: Proceedings of the 17th ACM Conference on Information and Knowledge Management, Napa Valley, California, USA. 509-518.

Roman, D. (2009). Crowdsourcing and the question of expertise. Commnuications of the.ACM, 52(12), 12-12. 

Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. Paper presented at the HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France. 50-53.

Su, Q., Pavlov, D., Chow, J., & Baker, W. C. (2007). Internet-scale collection of human-reviewed data. Paper presented at the WWW '07: Proceedings of the 16th International Conference on World Wide Web, Banff, Alberta, Canada. 231-240.

Surowiecki, J. (2005). The wisdom of crowds. Anchor. Tanaka, A., Tokui, N., & Momeni, A. (2005). Facilitating collective musical

creativity. Paper presented at the MULTIMEDIA '05: Proceedings of the 13th Annual ACM International Conference on Multimedia, Hilton, Singapore. 191-198.

Von Ahn L., & Dabbish, L. (2004). Labeling images with a computer game. Paper presented at the CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria. 319-326.