Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial

Randomised trial of open review

Website
prev
next

In a nutshell

In 1998 we conducted a randomised trial at The BMJ, a large general medical journal, to examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. Outcome measures included review quality (using the validated Review Quality Instrument), the recommendation regarding publication, the time taken to review.

Overview

Type of project

Trial
Operated by

BMJ
Listing status

Verified
Website

https://www.bmj.com/content/318/7175/23

Goals and intentions

Previous RCTs at The BMJ failed to confirm that blinding of reviewers to authors' identities improved the quality of reviews so we decided to experiment with open identities peer review. To us it seemed unjust that authors should be “judged” by reviewers hiding behind anonymity: either both should be unknown or both known, and it is impossible to blind reviewers to the identity of authors all of the time. We therefore conducted a randomised trial to confirm that open review did not lead to poorer quality opinions than traditional review and that reviewers would not refuse to review openly (because open review would then be unworkable).

Areas of innovation

What is reviewed

Review of data or code

No

Format of review

Free-form commenting

Process

Transparency

Open identities

Review features

Eligible reviewers/editors

Consecutive manuscripts received by the BMJ and sent by editors for peer review during the first seven weeks of 1998 were eligible for inclusion. Four potential clinical reviewers were selected by one of the BMJ's 13 editors. Two of these four reviewers were chosen to review the manuscript. The other two were kept in reserve in case a selected reviewer declined. The selected reviewers were randomised either to be asked to have their identity revealed to authors (intervention group) or to remain anonymous (control group), forming a paired sample.

Results

Number of scholarly outputs commented on

100-1,000
Metrics

A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
Results summary

Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Most authors were in favour of open peer review. (van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 1999; 318: 23 – 27.)
Results URL

https://www.bmj.com/content/318/7175/23

No comments yet.