do mechanical turks dream of square pie charts?
DESCRIPTION
Presenter: Robert Kosara, Caroline ZiemkiewiczBELIV 2010 Workshophttp://www.beliv.org/beliv2010/TRANSCRIPT
![Page 1: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/1.jpg)
Do Mechanical Turks Dream of Square Pie Charts?
Robert Kosara and Caroline Ziemkiewicz
![Page 2: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/2.jpg)
University vs. Online Studies
University Online
Controlled environment
Easy to recruit Standard incentives
Homogenous users Slow, work-intensive Small samples
Broader demographics Fast, low effort Larger numbers
Environmental control Vote flooding Recruitment and
advertising
2 Caroline Ziemkiewicz
![Page 3: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/3.jpg)
Mechanical Turk Amazon’s micro-payment job market Addresses many issues with online studies!
Prevents vote flooding Manages payment automatically Handles recruitment
3 Caroline Ziemkiewicz
![Page 4: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/4.jpg)
Our experiences Turking studies We’ve done a lot of these by now!
2 years 7 studies Over 550 participants
What have we learned?
4 Caroline Ziemkiewicz
![Page 5: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/5.jpg)
Participants Who are the Turkers?
57.4% female, 42.6% male (in our studies) Contrast with student populations
Mean age of 32, and a broad distribution…
5 Caroline Ziemkiewicz
![Page 6: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/6.jpg)
Participants Generally faster at
tasks than student populations
Some performance differences
Personality
6 Caroline Ziemkiewicz
![Page 7: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/7.jpg)
Designing studies Preventing cheaters
Bonus system allows for accuracy/quality incentives
Building checks into data collection Pre-testing/calibration
Other issues Informed consent Good instructions Feedback Audit trails
7 Caroline Ziemkiewicz
![Page 8: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/8.jpg)
Running studies Participation varies over time of day…
Highest rate during the afternoon + evening EST
8 Caroline Ziemkiewicz
![Page 9: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/9.jpg)
Running Studies Timing affects speed of completion Rate of participation starts high and
decreases over time Placement on front page is crucial!
Longer tasks get fewer participants Turkers like doing studies!
9 Caroline Ziemkiewicz
![Page 10: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/10.jpg)
To Turk or not to Turk Studies that lend themselves well to Turking
Simple interactions and data collection Short performance time Responses that are hard to fake
Studies that don’t Accuracy of self-reported data is crucial Complex tasks Very long task duration
10 Caroline Ziemkiewicz
![Page 11: Do Mechanical Turks Dream of Square Pie Charts?](https://reader033.vdocuments.site/reader033/viewer/2022061217/54b4853e4a7959bb068b475d/html5/thumbnails/11.jpg)
Thanks!
Let’s discuss!