Amazon’s Mechanical Turk; MTurk; Online Data Collection; Psychometrics; Score Reliability
Previous research has supported the use of Amazon’s Mechanical Turk (MTurk) for online data collection in individual differences research. Although MTurk Masters have reached an elite status because of strong approval ratings on previous tasks (and therefore gain higher payment for their work) no research has empirically examined whether researchers actually obtain higher quality data when they require that their MTurk Workers have Master status. In two different online survey studies (one using a personality test and one using a cognitive abilities test), the psychometric reliability of MTurk data was compared between a sample that required a Master qualification type and a sample that placed no status-level qualification requirement. In both studies, the Master samples failed to outperform the standard samples.
Journal of Individual Differences
Rouse, Steven V., "Reliability of MTurk Data from Masters and Workers for Personality and Cognitive Abilities" (2019). Pepperdine University, Faculty Open Access Publications. Paper 188.