Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial

Randomised controlled trial on the quality of openly published peer reviews

In a nutshell

We conducted a randomised controlled trial to see whether telling peer reviewers that their signed reviews of original research papers might be posted on The BMJ's website would affect the quality of their reviews.

Goals and intentions

Traditional peer review for scientific and medical journals has major flaws. Although the peer reviewer knows the identity of the author of the manuscript being assessed, the author doesn’t know the identity of the peer reviewer. Over recent years this form of peer review has come under increasing criticism, largely for its lack of accountability. Unscrupulous reviewers can delay or prevent the publication of work with which they disagree, or promote inappropriately the work of likeminded researchers. Worse still, reports exist of anonymous reviewers appropriating ideas and words from the manuscripts they have been reviewing. Although the peer review process has changed and reviewers for some journals have become used to more openness, evidence about the effect of opening review further is still required. Trust in the processes of science is still relevant, and reluctance remains, at least in some areas of medical research, to fully open up peer review. We set out to investigate the effects of reviewers’ foreknowledge that their signed review might be shared not just with co-reviewers and authors, but also with any interested reader.

Project status

Completed

Types of outputs

Privately shared manuscripts

Review process

Review requested by

Authors
Reviewer selected by

Editor, service, or community
Public interaction

No
Author response

Yes
Decision

Binary decision

Review policy

Review coverage

Complete paper
Reviewer identity known to

Public
Competing interests

Checked

Review features

Notes

Consecutive eligible papers were randomised either to have the reviewer’s signed report made available on the BMJ’s website alongside the published paper (intervention group) or to have the report made available only to the author—the BMJ’s normal procedure (control group). The intervention was the act of revealing to reviewers—after they had agreed to review but before they undertook their review—that their signed report might appear on the website. The main outcome measure was the quality of the reviews, as independently rated on a scale of 1 to 5 using a validated instrument by two editors and the corresponding author.
Review of code or data

No
Eligible reviewers/editors

The first of the editors’ four potential reviewers was contacted with details of the study, including the title and author of the paper, and asked whether he or she was willing to review the paper. At this point, no information was given about the study arm into which the paper had been allocated. If the first reviewer declined, the next was contacted, continuing until a consenting reviewer was found. If none of the four potential reviewers consented, the paper was excluded.
Criteria for inclusion

Consecutive research manuscripts chosen by editors for external review between November 1999 and July 2000 were eligible for inclusion.

Transparency

Open identities

Results

Number of scholarly outputs commented on

100-1,000
Results summary

Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.
Results URL

https://www.bmj.com/content/341/bmj.c5729

No comments yet.