crowdtruth games @nlesc ehumanities day 2015
TRANSCRIPT
The Wisdom of the Crowd through Gamification
Lora Aroyo, Merel van Empel & Carlos Martinez Ortiz
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
crowdsourcing – interpreta5on at scale …
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
paid crowdsourcing solves only part …
there are massive niches of experts online …
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
• entertaining • compe--ve • quick • s-cky • targeted campaigns • effec-ve engagement
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
games with a purpose (GWAP)
Winner EuroITV CompetitionBest Archives on the Web Award
h7p://waisda.nl
Video Tagging – time-stamped tags
diversity of opinions from the crowd mul-tude of contexts
collected in a decentralized way independent interpreta5ons
aggregated view – the big picture
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
understanding perspec5ves
a new approach to understanding seman5cs harnessing the power & diversity of the crowd
harnessing disagreement
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
informs a vector space model of truth instead of a boolean, fuzzy or sta-s-cal model
disagreement is signal
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
http://game.CrowdTruth.org
h7p://game.crowdtruth.org
• Games engage niche crowds - effective in filling the gap of expert annotations in traditional crowdsourcing
• Annotator disagreement is signal, not noise:
• indicative of the variation in human semantic interpretation of signs
• can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality
• Data analysis challenges
Take Home Message
CrowdTruth Team: Lora Aroyo, Chris Welty Robert-‐Jan Sips, Carlos Mar5nez Or5z, Anca Dumitrache, Oana Inel, Benjamin Timmermans, Susanna van de Ven, Merel van Empel, Jelle v.d. Ploeg, Ta-ana Cristea, Khalid Khamkham, Harrië7e Smook, Rens van Honschooten, Arne Rutjes
CrowdTruth.org github.com/CrowdTruth
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
http://game.CrowdTruth.org