wisdom and folly of the crowds

Download Wisdom and Folly of the Crowds

Post on 20-Jan-2015




2 download

Embed Size (px)




  • 1. Wisdom (and Folly) of the Crowds
    Jessica Keup

2. Outline
Crowdsourcing Marketplaces
Crowdsourcing Research
3. History
Crowdsourcing is a method of solving problems with, and outsourcing work, to the collective online intelligence
James Surowiecki - The Wisdom of Crowds, 2005
4. History
Remember outsourcing? Sending jobs to India and China is so 2003. The new pool of cheap labor: everyday people using their spare cycles to create content, solve problems, even do corporate R & D.
Jeff Howe - The Rise of Crowdsourcing, 2006
5. Crowdsourcing vs . . .
work given to pre-decided person or group
Open Source
initiated by public, not a client
6. Examples?
Netflix prize
Google image labeler
EquitySplash / Piqqem
7. Best Uses
Image tagging
Search relevance evaluation
Content generation
Product categorization
Collecting feedback and reviews
8. Pros
Labor at reduced cost
Huge potential workforce
Variety of opinions, ideas, experience
Consensus and better answers
9. Cons
10. Fixes
corroboration, ratings
qualification tests
11. Crowdsourcing Marketplaces
Mechanical Turk
Cloud Crowd
12. mTurk
Artificial Artificial Intelligence
Human Intelligence Tasks (HITs)
$0.00 and up, mostly a few cents
SDKs in Java, Ruby, Perl, .NET
Or, create through website UI*
13. Training AI
Computer Music Composition using Crowdsourcing and Genetic Algorithms
14. 15. 16. Research Examples
collecting creative drawings
rating audio and video sample quality
tagging location-sensitive queries and points of interest with mobile devices
remote massage
task delegation in Wikipedia
collecting volunteer expertise to help the homeless
annotating political campaign ads
17. More Research examples
describing and organizing geometric shapes*
tracking popular music
rating search result relevance
collecting user feedback
extracting product and brand information
tagging specific objects in an image
training speech recognition software
18. Questions?
19. References
Alonso, O., Rose, D.E., and Stewart, B. (2008). Crowdsourcing for Relevance Evaluation. SIGIR Forum 42(2) 9-15.
Amazon.com, Inc. (2010). Amazon Mechanical Turk. https://www.mturk.com/mturk/welcome.
CastingWords (2010). Audio transcription services: plus video, CDs, podcasts and more. http://castingwords.com/
Chung, K., Chiu, C., Xiao, X., & Chi, P. (2009). Stress outsourced: A haptic social network via crowdsourcing. Paper presented at the CHI EA '09: Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems, Boston, MA, USA. 2439-2448.
DarwinTunes (2010).Welcome to DarwinTunes. http://darwintunes.org/
Ganjisaffar, Y., Javanmardi, S., & Lopes, C. (2009). Leveraging crowdsourcing heuristics to improve search in wikipedia. Paper presented at theWikiSym '09: Proceedings of the 5th International Symposium on Wikis and Open Collaboration,Orlando, Florida. 1-2.
Harper, F. M., Raban, D., Rafaeli, S. & Konstan, J. A. (2008). Predictors of answer quality in online Q & A sites. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 865-874.
Hintikka, K. A. (2008). Web 2.0 and the collective intelligence. Paper presented at the MindTrek '08: Proceedings of the 12th International Conference on Entertainment and Media in the Ubiquitous Era, Tampere, Finland. 163-166.
Howe, J. (2006). The Rise of Crowdsourcing. Wired, 14(6). http://www.wired.com/wired/archive/14.06/crowds.html.
20. More References
Hsueh, P., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: Astudy of annotation selection criteria. Paper presented at theHLT '09: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing,Boulder, Colorado. 27-35.
Jagadeesan, A. P., Lynn, A., Corney, J. R., Yan, X. T., Wenzel, J., Sherlock, A., et al. (2009). Geometric reasoning via internet CrowdSourcing. Paper presented at theSPM '09: 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling,San Francisco, California. 313-318.
Kittur, A., Chi, ,E. H., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. Paper presented at the CHI '08: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Florence, Italy. 453-456.
Koblin, A. M. (2009). The sheep market. Paper presented at theC&C '09: Proceeding of the Seventh ACM Conference on Creativity and Cognition,Berkeley, California, USA. 451-452.
Krieger, M., Stark, E. M., & Klemmer, S. R. (2009). Coordinating tasks on the commons: Designing for personal goals, expertise and serendipity. Paper presented at theCHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems,Boston, MA, USA. 1485-1494.
Ledlie, J., Odero, B., Minkov, E., Kiss, I., & Polifroni, J. (2009). Crowd translator: On building localized speech recognizers through micropayments.SIGOPS Operating.Systems Review,43(4), 84-89.
Li, C., Buyuktur, A. G., Hutchful, D. K., Sant, N. B., & Nainwal, S. K. (2008). Portalis: Using competitive online interactions to support aid initiatives for the homeless. Paper presented at theCHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems,Florence, Italy. 3873-3878.
21. Even More References
Mannes, A. E. (2009). Are we wise about the wisdom of crowds? the use of group judgments in belief revision.Management Science,55(8), 1267-1279.
Mason, W., & Watts, D. J. (2009). Financial incentives and the performance of crowds. Paper presented at theHCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation,Paris, France. 77-85.
Milne, D., & Witten, I. H. (2008). Learning to link with wikipedia. Paper presented at theCIKM '08: Proceedings of the 17th ACM Conference on Information and Knowledge Management,Napa Valley, California, USA. 509-518.
Roman, D. (2009). Crowdsourcing and the question of expertise.Commnuications of the.ACM,52(12), 12-12.
Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. Paper presented at theHCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human Computation,Paris, France. 50-53.
Su, Q., Pavlov, D., Chow, J., & Baker, W. C. (2007). Internet-scale collection of human-reviewed data. Paper presented at the WWW '07: Proceedings of the 16th International Conference on World Wide Web, Banff, Alberta, Canada. 231-240.
Surowiecki, J. (2005). The wisdom of crowds. Anchor.
Tanaka, A., Tokui, N., & Momeni, A. (2005). Facilitating collective musical creativity. Paper presented at theMULTIMEDIA '05: Proceedings of the 13th Annual ACM International Conference on Multimedia, Hilton, Singapore. 191-198.
Von Ahn L., & Dabbish, L. (2004). Labeling images with a computer game. Paper presented at the CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria. 319-326.


View more >