Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial

Randomised trial of open review

In a nutshell

In 1998 we conducted a randomised trial at The BMJ, a large general medical journal, to examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper. Outcome measures included review quality (using the validated Review Quality Instrument), the recommendation regarding publication, the time taken to review.

Goals and intentions

Previous RCTs at The BMJ failed to confirm that blinding of reviewers to authors' identities improved the quality of reviews so we decided to experiment with open identities peer review. To us it seemed unjust that authors should be “judged” by reviewers hiding behind anonymity: either both should be unknown or both known, and it is impossible to blind reviewers to the identity of authors all of the time. We therefore conducted a randomised trial to confirm that open review did not lead to poorer quality opinions than traditional review and that reviewers would not refuse to review openly (because open review would then be unworkable).

Review process

Review requested by

Authors
Reviewer selected by

Editor, service, or community
Public interaction

Not included
Opportunity for author response

Included
Recommendation

Binary decision

Review policy

Review coverage

Complete paper
Reviewer identity known to

Public
Competing interests

Checked

Review features

Review of code or data

No
Eligible reviewers/editors

Consecutive manuscripts received by the BMJ and sent by editors for peer review during the first seven weeks of 1998 were eligible for inclusion. Four potential clinical reviewers were selected by one of the BMJ's 13 editors. Two of these four reviewers were chosen to review the manuscript. The other two were kept in reserve in case a selected reviewer declined. The selected reviewers were randomised either to be asked to have their identity revealed to authors (intervention group) or to remain anonymous (control group), forming a paired sample.

Transparency

Open identities

Results

Number of scholarly outputs commented on

100-1,000
Metrics

A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.
Results summary

Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Most authors were in favour of open peer review. (van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 1999; 318: 23 – 27.)
Results URL

https://www.bmj.com/content/318/7175/23

No comments yet.