[ View menu ]

Amazon’s Mechanical Turk meets classic JDM experiments

Filed in Ideas ,Research News ,SJDM ,Tools
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)



Gabriele Paolacci sends along the following announcement. Decision Science News is also a fan of Amazon’s Mechanical Turk (or mturk as we insiders call it), and it and its colleagues at Yahoo! Research are actively using with the evolving methodology.

You are probably aware of the growing interest for Amazon’s Mechanical Turk (AMT) as a tool for conducting experimental research in the web. By allowing to allocate tasks to paid subjects, Mechanical turk is an online labor market where subjects can be recruited and paid on a large scale and in short time. Being an online labor market, it also is a potentially interesting “ecologically valid” context for experiments: subjects are looking for “real” jobs when they decide to participate to an experiment.

While the potential for using AMT in experiments is huge, there are obvious issues of validating its reliability. In this spirit, hoping to provide a service to the experimental community, Massimo Warglien and me started a blog. In this blog we are sharing our first attempts to replicate classical lab experiments in AMT and compare the outcomes with the original ones. As replications cumulate, we hope to be able to provide in the future more sophisticated meta-analyses of the results.

Please give a look at the blog if you are curious: http://experimentalturk.wordpress.com/

We invite you to share this information with anyone that might be interested. In particular, we invite to send us other replication exercises that might be performed to enrich the data set of replications and broaden the empirical basis of the validation effort. Other kind of experimental or survey experiences or useful tools to improve experimentation in the web via crowd-sourcing resources such as AMT are welcome as well; useful links are also gratefully received.

Among other things, AMT might provide a great tool for having students implement simple and cheap experimental projects during courses on experimental methods and results. Replication studies could be great projects for students and could help to quickly implement a large data set of comparative nature with other experimental settings such as the traditional lab.

Hope you like the blog. If anyone is interested, I will be available to talk more about this at the SJDM conference.

If you want to learn about the original mechanical turk, then it’s Wikipedia to the rescue.


  1. Amazon Blog says:

    This is really a good article..Mechanical Turk enables developers and businesses to achieve their goals more quickly and at a lower cost than was previously possible.

    July 21, 2010 @ 10:35 am

  2. Conducting experiments on Amazon's Mechanical Turk | Decision Science News says:

    […] Amazon’s Mechanical Turk meets classic JDM experiments […]

    October 19, 2010 @ 12:54 am

RSS feed Comments

Write Comment

XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>