To the top

Page Manager: Webmaster
Last update: 9/11/2012 3:13 PM

Tell a friend about this page
Print version

Experts, Coders, and Crow… - University of Gothenburg, Sweden Till startsida
To content Read more about how we use cookies on

Experts, Coders, and Crowds: An analysis of substitutability

Working paper
Authors Kyle L. Marquardt
Daniel Pemstein
Constanza Sanhueza Petrarca
Brigitte Seim
Steven Lloyd Wilson
Michael Bernhard
Michael Coppedge
Staffan I Lindberg
Publisher University of Gothenburg, Varieties of Democracy Institute: Working Paper No. 53. October 2017
Place of publication Göteborg
Publication year 2017
Published at Quality of Government Institute (QoG)
Department of Political Science
V-Dem Institute
Language en
Subject categories Political Science


Recent work suggests that crowd workers can replace experts and trained coders in common coding tasks. However, while many political science applications require coders to both and relevant information and provide judgment, current studies focus on a limited domain in which experts provide text for crowd workers to code. To address potential over-generalization, we introduce a typology of data producing actors - experts, coders, and crowds - and hypothesize factors which affect crowd-expert substitutability. We use this typology to guide a comparison of data from crowdsourced and expert surveys. Our results provide sharp scope conditions for the substitutability of crowd workers: when coding tasks require contextual and conceptual knowledge, crowds produce substantively dierent data from coders and experts. We also find that crowd workers can cost more than experts in the context of cross-national panels, and that one purported advantage of crowdsourcing - replicability - is undercut by an insucient number of crowd workers.

Page Manager: Webmaster|Last update: 9/11/2012

The University of Gothenburg uses cookies to provide you with the best possible user experience. By continuing on this website, you approve of our use of cookies.  What are cookies?