Crowd-sourced peer review: wisdom or tyranny of the crowd?

Abstract

Recent work has deployed the logic of jury theorems to assert that crowd-sourced peer review is superior to traditional peer review. That argument is shown to fail by counterexample: in a model of scientific communities with incumbents that favour work consistent with their own, the probability of genuinely revolutionary ideas being published tends to zero as the size of the “jury” increases. A related problem is that the fundamental purpose of peer review in the context of intellectual inquiry and progress is given inadequate consideration. Remedying that calls into question the appropriateness of the jury theorem logic. The broader argument made for crowd-sourced peer review suffers from problems on three additional dimensions. First, it does not clearly establish what the problems are with traditional peer review that need to be resolved. Second, its appeals to diversity are suspect, contradicted by the counterexample, and arguably inadmissible because they find no expression in the formal model used. Finally, the assertion that these matters can be resolved by empirical evidence from implementation of alternative systems is premised on questionable epistemic assumptions that are neither stated nor defended. The conclusion drawn is that no case has been established for the implementation of crowd-sourced peer review, and claims made in that regard evince a concerning epistemic gap between the soundness of their arguments and the strength of their proposals.