Presentation Type
Poster
Keywords
statistics, crowdsourcing, methodology
Department
Psychology
Major
Psychology
Abstract
Mechanical Turk, an online crowdsourcing platform, has recently received increased attention in the social sciences as studies continue to suggest its viability as a source for reliable experimental data. Given the ease with which large samples can be quickly and inexpensively gathered, it is worth examining whether Mechanical Turk can provide accurate experimental data for methodologies requiring such large samples. One such methodology is Item Response Theory, a psychometric paradigm that defines test items by a mathematical relationship between a respondent’s ability and the probability of item endorsement. To test whether Mechanical Turk can serve as a reliable source of data for Item Response Theory modeling (also known as latent trait modeling), researchers administered a verbal reasoning scale to 500 Mechanical Turk workers and compared the resulting Item Response Theory model to that of an existing normative sample. While Item Characteristic Curves did significantly differ, both models had high agreement on the fit of participants’ response patterns and on participant ability estimation. Future research should attempt to extend such findings to other variations of Item Response Theory models.
Faculty Mentor
Steve Rouse
Funding Source or Research Program
Not Identified
Location
Waves Cafeteria
Start Date
1-4-2016 2:00 PM
End Date
1-4-2016 3:00 PM
The Reliability of Crowdsourcing: Latent Trait Modeling with Mechanical Turk
Waves Cafeteria
Mechanical Turk, an online crowdsourcing platform, has recently received increased attention in the social sciences as studies continue to suggest its viability as a source for reliable experimental data. Given the ease with which large samples can be quickly and inexpensively gathered, it is worth examining whether Mechanical Turk can provide accurate experimental data for methodologies requiring such large samples. One such methodology is Item Response Theory, a psychometric paradigm that defines test items by a mathematical relationship between a respondent’s ability and the probability of item endorsement. To test whether Mechanical Turk can serve as a reliable source of data for Item Response Theory modeling (also known as latent trait modeling), researchers administered a verbal reasoning scale to 500 Mechanical Turk workers and compared the resulting Item Response Theory model to that of an existing normative sample. While Item Characteristic Curves did significantly differ, both models had high agreement on the fit of participants’ response patterns and on participant ability estimation. Future research should attempt to extend such findings to other variations of Item Response Theory models.