delivering web experiments using amazon mechanical...

DELIVERING WEB EXPERIMENTS USINGAMAZON MECHANICAL TURK

CHUN CHAN

18 OCT 2017

INTRODUCTION

¡ Software developer with the Linguistics Department and CSD

¡ Developing MTurk based experiments for about 3 years

MTURK URLS

¡ Sandbox

¡ https://requestersandbox.mturk.com/begin_signin

¡ https://workersandbox.mturk.com/mturk/beginsignin

¡ Production

¡ https://requester.mturk.com/mturk/beginsignin

¡ https://www.mturk.com/mturk/beginsignin

https://acceptability-judgements-demo.firebaseapp.com/

1. External Link 2. Internal Single Page Survey 3. Embedded

Easy setupDifficult close out

Complex setupEasier close out

EXTERNAL LINK – WORKER’S VIEWS

¡ Live demo

EXTERNAL LINK – REQUESTER’S VIEW

¡ Tutorial

¡ https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-qualtrics-f5366f0bd880

1. External Link 2. Internal Single Page Survey 3. Embedded

Easy setupDifficult close out

Complex setupEasier close out

SINGLE PAGE SURVEY

¡ Worker doesn’t have to visit an external URL

¡ No confirmation codes to validate

¡ Involves writing some HTML

¡ https://bradylab.ucsd.edu/ttt/

¡ Data gets saved in Mturk

SINGLE PAGE SURVEY – WORKER’S VIEWS

¡ Live demo

1. External Link 2. Internal Single Page Survey 3. Embedded

Easy setupDifficult close out

Complex setupEasier close out

EMBEDDED

¡ Worker doesn’t have to visit external URL

¡ No confirmation codes to validate

¡ Data is saved to Firebase or Qualtrics

¡ Better functionality from worker’s perspective

EMBEDDED SURVEY – WORKER’S VIEWS

¡ Live demo

EMBEDDED

¡ Mturk Command Line Tools (CLI)

¡ https://requester.mturk.com/developer/tools/clt

¡ http://web.mit.edu/cathywu/Public/AMT/aws-mturk-clt-1.3.0/UserGuide.html#externalhit

¡ AMT Manager

¡ https://nusubdb.ling.northwestern.edu/AMTManager/

¡ Invite only for now

EMBEDDED

¡ Firebase or Qualtrics compatible

¡ Some special code “hooks” required

¡ Templates available if interested

¡ Speech In Noise frameworks

https://acceptability-judgements-demo.firebaseapp.com/

workerId = ABC1234567

https://www.mturk.com/mturk/externalSubmit?workerId = ABC1234567

IN SUMMARY

External URL¡ Worker must visit an external URL

¡ Can’t close original window

¡ Worker enters confirmation code (possible typos)

¡ Data is saved to Firebase or Qualtrics

¡ Requester matches confirmation codes to workers

Single Page Survey¡ Worker doesn’t have to visit an external URL

¡ Long scrolling page, difficult for worker

¡ No confirmation codes to validate

¡ Involves writing some HTML

¡ Data gets saved in Mturk

IN SUMMARY

Embedded Survey¡ Worker doesn’t have to visit external URL

¡ No confirmation codes to validate

¡ Data is saved to Firebase or Qualtrics

¡ Better functionality from worker’s perspective

¡ Little more work to get set up initially (Firebase/Qualtrics templates)

¡ Can embed games, other types of interesting tasks

¡ Speech in Noise frameworks

¡ https://northwestern.box.com/s/9g2rigz1iqh4ymfkgunq6t31u3iycbpr

THE LIFE OF AN MTURK WORKER

https://turkopticon.ucsd.edu/

WHY USE MTURK

¡ Easy set up

¡ Diverse population

¡ More data over a short period of time

¡ Everyone else is doing it

REASONS NOT TO USE MTURK

¡ You have to manage everything

¡ Payments

¡ Technical issues

¡ Angry workers

¡ Preventing the same workers from repeating your HIT

¡ Can get expensive

¡ Not all workers are naive

¡ https://doi.org/10.1016/j.tics.2017.06.007)

¡ Not all languages and countries are evenly represented

¡ https://www.semanticscholar.org/paper/The-Language-Demographics-of-Amazon-Mechanical-Tur-Pavlick-Post/034bd3a6f63caf2bb2dc2762e43401cd204b86b5)

ALTERNATIVES

¡ Turkprime

¡ https://www.turkprime.com

¡ Prolific

¡ https://www.prolific.ac/

QUESTIONS?