Effects of training on quality of peer review: randomised controlled trial

Training peer reviewers

In a nutshell

Single blind randomised controlled trial at The BMJ, a general medical journal, to test the effect of two types of training on the quality of review, number of errors detected, time taken to review and recommendation for publication. There were two intervention groups (workshop versus self-taught package) and a control group receiving no training.

Goals and intentions

Many studies have illustrated the inadequacies of peer review and its limitations in improving the quality of research papers. However, few studies have evaluated interventions that try to improve peer review. We conducted a randomised controlled trial to examine the effects of training. Training that would be feasible for reviewers to undergo and for a journal to provide would have to be short or provided at a distance. Although the effectiveness of short educational interventions is questionable, some brief interventions have been shown to be successful (depending on what is being taught and the methods used). We aimed to determine whether reviewers for The BMJ who underwent training would produce reviews of better quality than those who received no training; whether face to face training would be more beneficial than a self taught package; and whether any training effect would last at least six months. The training materials for the trial are freely available on bmj.com (https://www.bmj.com/about-bmj/resources-reviewers/training-materials).

Types of outputs

Privately shared manuscripts

Gallery

Review features

Eligible reviewers/editors

All UK reviewers who had recently completed a review at the time of the study were eligible and invited to take part in a research project.

Results

Number of scholarly outputs commented on

1-10
Results summary

Reviewers in the self taught group scored higher in review quality after training than did the control group, but the difference was not of editorial significance and was not maintained in the long term. Both intervention groups identified significantly more major errors after training than did the control group. The evidence for benefit of training was no longer apparent on further testing six months after the interventions. Training had no impact on time taken to review but was associated with an increased likelihood of recommending rejection. (1) Schroter S, Black N, Evans S, Smith R, Carpenter J, Godlee F. Effects of training on the quality of peer review: A randomised controlled trial. BMJ 2004;328:673-5. (2) Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008:101: 507-14.
Results URL

https://www.bmj.com/content/328/7441/673

No comments yet.