Bias in publishing is a serious issue, as even small advantages in acceptance rates can have large effects when propagated forward to grants, jobs, and awards. Since traditional journal peer review happens mostly in a black box, it’s hard to understand the extent of the problem, let alone devise solutions to address it.
As leaders of experimental peer review trials, platforms, and services work to reimagine the peer review workflow, they can also innovate in the reduction of bias. To start a conversation about these questions, we hosted our first ReimagineReview community call on August 16th. Here’s a summary of some of the highlights.
Our first speaker, Dr. Jennifer Raymond (Professor of Neurobiology at Stanford University and eLife Reviewing Editor) discussed key findings from her study of disparities in acceptance rates at eLife. This study showed that male corresponding authors had a 13% advantage in overall acceptance over female authors. The disparities in success rates were even greater between authors from different countries. Jennifer emphasized that in addition to implicit bias, factors such as domestic responsibilities, funding, mentoring and ‘opting out’ of competition could affect success rates in publishing.
To better isolate the effect of bias, Jennifer’s team asked if the gender of the reviewers correlated to the outcome of review. If reviewers are unbiased, they would provide consistent evaluations regardless of their own demographic profiles. In contrast, the results showed that reviewers have a preference for authors of the same gender and nationality. Due to the higher numbers of male reviewers and authors from US and Europe, the effects of homophily are more likely to benefit authors who match the better-represented demographics
Our second speaker, Dr. Misha Teplistkiy (Assistant Professor at the School of Information at the University of Michigan) discussed recent findings from his work based on data provided by PLOS ONE. Using co-authorship distance as a measure for intellectual closeness, his group found that closer that reviewer is to the author, the lower the likeliness of rejection. This correlation persists even for co-authorship distance greater than two (ie, when authors and reviewers have never been coauthors), which by conventional standards should have little to no conflict of interest. Thus intellectual closeness may result in favourable outcomes, potentially due to agreement in validity shared by the community, beyond nepotism.
Next Misha highlighted findings from his recent study on biases in committees, using data from a research proposal competition review at Harvard Medical School. Reviewers evaluated proposals as usual, then received anonymous artificial scores. The reviewers were informed that these were “scores from fellow reviewers,” then were given a chance to update their original score. In these committees, bad scores were much less likely to be changed than favorable scores – bad scores were “sticky.” Furthermore, highly cited scientists updated their scores less than others; men updated 13% less than women. The percentage that updated correlated with the gender composition of the subfield. Women were more likely to update their scores in a female under-represented subfield. Thus, opinions from underrepresented demographics are more likely to be discounted.
You can watch the full recordings of the talks here.
After the talks, we launched into an unrecorded discussion from which emerged a number of ideas for practical interventions that can reduce bias in the peer review process.
The studies discussed in this call were in many cases only possibly because journals were willing to disclose otherwise hidden information about their editorial processes. Our next community call—during Peer Review Week, on Friday, September 20—will further explore the theme of transparency as it relates to quality in peer review. Read more and register here!