Amazon Mechanical Turk (MTurk) and TurkPrime 101

What is MTurk?

Amazon Mechanical Turk (Mturk) is a crowdsourcing internet marketplace that enables individuals and businesses (Requesters) to coordinate the use of human workers (Workers/Turkers) to perform tasks that computers are currently unable to do.

Who are MTurk Workers?

MTurk workers are predominantly located in the US (roughly 75%) or India (roughly 20%). There are roughly equal numbers of male and female MTurk workers and roughly 75% of workers were born between 1980-2000. You can find more details about MTurk demographics here.

How might you use MTurk?

1. Social science experiments such as using MTurk to pilot test a survey, have a general sample for your study, etc.
2. Panel studies such as pre/post designs, screener studies, and longitudinal studies.
3. Processing images and text such as coding images or text, choosing the best image from a group, or auditing images for inappropriate content. Jon Schwabish recently had MTurk workers categorize and rate images. We used a variety of logic models to compare the efficacy of a variety of conditions.
4. Collect information such as have MTurk workers collect email addresses for executives.
5. Transcribe and/or translate audio/text.

What are some of the benefits of using MTurk?

1. More diverse sample than the typical social science study and less of a WEIRD
2. Cheap way to collect data. For instance, at federal wage of $7.25/hr, a 10-minute study with 300 participants would cost $1.20/person or $360 total.
3. Quick way to collect data. Especially if you use batching techniques, you can nearly 300 responses within an hour.

What are some of the limitations of using MTurk?

I list some of the limitations I routinely hear about. However, as you read the limitations, note that many of the limitations can be offset with high quality research practice.

1. Though MTurk workers are a diverse population, they are not representative of the typical US population.
2. Some argue that MTurk workers have low attention spans.
3. Some argue that MTurk workers do not like doing tasks with a heavy cognitive load (e.g., reading lengthy text, watching videos, or anything that will take time or too much work).
4. Some argue that using MTurk requires larger sample sizes because they give smaller effect sizes and/or larger standard deviations. However, others disagree with this.
5. Newer, lower-reputation workers may not provide as high quality of data. However, you can limit your sample to just workers who have performed a minimum number of hits and/or have a high reputation (e.g., “Masters”).
6. There may be messy data. For instance, on an open-ended response, one participant copy-pasted “Does anyone even read these???” multiple times, possibly thinking there was a minimum count that he had to bypass. Regardless, I do not thing the data is messier than what you typically see in the real world.
7. Participants may lie in order to get the hit. It is important that you screen out participants who do not fit your sample demographics without telling them what you are looking for, otherwise they may claim to be a 20-year-old woman when they are actually a 40-year-old man. Recently, there was a claim of an increase in the use of bots on MTurk but TurkPrime suggests this may not be true (here and here).

What is TurkPrime and why should I use it?

TurkPrime is a separate interface for working with MTurk. It essentially supercharges MTurk and makes it easier to collect data! Some of the many features it has includes:

• Microbatching and hyperbatching studies to reduce costs and, in the case of hyperbatching, increase speed of data collection.
• Easily allows you to prevent duplicate workers, both within and across studies.
• Automatic payment for MTurk workers (which they appreciate).
• Easily contact MTurk workers.
• Easily include/exclude certain workers (e.g., highly active workers, master workers).
• Easily pause, resume, restart a HIT and grant bonuses to workers.
• Easily anonymize worker IDs since worker IDs are not anonymized (i.e., you can find a person based on their ID).

I personally ONLY use TurkPrime to collect my data. It’s really simple to use and paying a little extra for TurkPrime actually save money instead of using MTurk.

How can I learn more about MTurk or TurkPrime?

First, I highly recommend you review these guidelines for academic researchers. They include statements about fair payment, how to be a good requester, and more. If you need some good citations on using MTurk in your studies, here is a list. Finally, the TurkPrime blog and help guides are extremely useful if you have any questions about using TurkPrime.

p.s. Check your requester ratings!

MTurk workers rate requesters! There are two main sites that do this.

1. Turkopticon: I don’t think this one is used as often anymore, but here is my requester page. I only have data from 2016 for my study on aesthetic experiences (study will be published soon). I didn’t know that I should pay workers more fairly at this time.
2. TurkerView: As an example, here is my requester page. I collected data a few days ago and one person rated me as a fair pay ($7.86/hour).