[ View menu ]

How to run experiments on Mechanical Turk

Filed in Encyclopedia ,Ideas ,Research News ,Tools
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)



A few posts back, we showed how classic decision making experiments are being replicated on Amazon’s insta-subject-pool otherwise know as Mechanical Turk (aka MT).

After that, Steven Pinker, at the SJDM keynote, presented Mechanical-Turk-collected data on perceptions of whether the past or present is perceived as more violent.

This week, Decision News News, currently stationed at Yahoo! Research in New York, points to a useful guide to running experiments on MT, written by fellow Yahoo researcher and all-around wizard of computational psychology, Winter Mason.

If you want to do some very basic experiments using MT, you can probably get started using their templates. However, if you want to have participants engage in more complex interactive tasks, you probably want to use their command line tools or API. Here’s a guide to help you decide. If you go the command-line route, Winter’s instructions will save you some of the pain of figuring it out for yourself.

Here at Yahoo, we’ve been able to do some amazing MT experiments, including group decision making tasks, in which the groups are assembled on the fly. Think of it, no more inviting people to the lab and having to cancel when too few show up.

Photo credit: http://en.wikipedia.org/wiki/File:Tuerkischer_schachspieler_windisch4.jpg


  1. CrowdFlower’s Blog » The Case for Online Experimentation says:

    […] Online labor markets dramatically lower the cost and hassle of conducting experiments. On Amazon’s Mechanical Turk, it is easy to run multiple experiments per week. Figuring out how to run experiments isn’t that hard, as there are already some nice tutorials available. […]

    May 1, 2010 @ 4:14 pm

  2. Tatiana says:

    Is there a way to make sure the workers taking the survey are from a specific state? Therefore, I wouldnt have to pay for users that take the survey unless they quality that specification?

    December 17, 2012 @ 7:27 pm

RSS feed Comments

Write Comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>