How to run experiments on Mechanical Turk
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
THE TECHNICAL DETAILS, TUTORIALS, WALK-THROUGHS
A few posts back, we showed how classic decision making experiments are being replicated on Amazon’s insta-subject-pool otherwise know as Mechanical Turk (aka MT).
After that, Steven Pinker, at the SJDM keynote, presented Mechanical-Turk-collected data on perceptions of whether the past or present is perceived as more violent.
This week, Decision News News, currently stationed at Yahoo! Research in New York, points to a useful guide to running experiments on MT, written by fellow Yahoo researcher and all-around wizard of computational psychology, Winter Mason.
If you want to do some very basic experiments using MT, you can probably get started using their templates. However, if you want to have participants engage in more complex interactive tasks, you probably want to use their command line tools or API. Here’s a guide to help you decide. If you go the command-line route, Winter’s instructions will save you some of the pain of figuring it out for yourself.
Here at Yahoo, we’ve been able to do some amazing MT experiments, including group decision making tasks, in which the groups are assembled on the fly. Think of it, no more inviting people to the lab and having to cancel when too few show up.
- Winter Mason’s Guide to doing experiments with Mechanical Turk
- Amazon’s Mechanical Turk Resource Center
- Amazon’s Mechanical Turk How To Guides
- Amazon’s Mechanical Turk Technical Documentation
- The resources page over at experimentalturk.wordpress.com
Photo credit: http://en.wikipedia.org/wiki/File:Tuerkischer_schachspieler_windisch4.jpg
[…] Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available. […]
May 1, 2010 @ 4:14 pm
Is there a way to make sure the workers taking the survey are from a specific state? Therefore, I wouldnt have to pay for users that take the survey unless they quality that specification?
December 17, 2012 @ 7:27 pm