Assessing the broader impacts of funding

Next, Jonah Cool (Science Program Officer, CZI) explained how preprints and other scientific outputs help inform the funder’s decision. All CZI funded projects must be deposited as preprints prior to peer review. Acknowledging that many research outcomes are currently unnecessarily shoehorned into the shape of a manuscript, other outputs such as developed technologies and methodologies are considered valuable and important for evaluation.  CZI also welcomes scientists to include the feedback on their preprints in grant applications and reports. Supporting open access and preprints is a part of CZI’s commitment to accelerating the pace of science, with priorities on increasing the utility of funded research outcome. 

Reviewer contribution should be better appreciated in evaluation

Our fourth speaker, Anna Hatch (Community Manager, DORA), spoke about how credit for peer review could be used to inform the evaluation of scientists. Crediting reviewers is challenging due to traditionally opaque review processes. We should consider looking at quality as well as quantity of peer review, but evaluating the former is challenging when the text of reviews is inaccessible. Currently, many universities and institutes do not have clear statements on how peer review activity will contribute to evaluation. Importantly, we need to remind ourselves that if peer review is considered in scientific evaluation, access to peer review opportunity is not equitable for all researchers. 

Useful feedback and increased visibility to discoveries for scientists

Last but not least, we heard Shawn Ferguson (Associate Professor of Cell Biology and Neuroscience, Yale University) describe his transition from being a preprint skeptic to committing to preprint all of his lab’s work.After posting three initial preprints, Shawn appreciated the benefits of sharing discoveries 6-8 months prior to journal publication. During this time he could see usage and readership of the preprints grow to hundreds of downloads. Furthermore, one of the preprints was highlighted and reviewed in preLights, which provided positive feedback and increased visibility. He was given the opportunity to present his preprinted work at a Gordon Research Conference and was included in his evaluation for tenure. 

Questions and answers

After hearing from each of the panelists, we moved into a question and answer period. Highlights are summarized below.

Jessica Polka: How is the peer review of preprint and journal submitted manuscript different?

Veronique shared that peer review of preprint can cover smaller parts of the preprint rather than the manuscript as a whole. Granular evaluation enables distinct evaluations of question, conclusion, and specific aspects of the methodology such as code and statistics. Boyana added that peer review of preprints can focus on improving the paper. This uncouples evaluating the quality of the work from its suitability for a journal, which reduces the role of reviewer as gatekeeper. In addition, Shawn noted that peer review of preprints can be an extended discussion of the work with the greater research community.

Richard Sever(bioRxiv, CSHL Press): Should everything be peer reviewed? What fraction between 0-100%?

While panelists declined to give a firm answer, it was clear that many felt that peer review is essential, but not for all manuscripts. 

Vivian Siegel(MIT): What is the most important information to evaluate in the peer review?

Veronique answered that peer review should evaluate whether the methodology is appropriate and the conclusions are sound and justified? Boyana remarked that it is important to identify which part of the work is strong, and which parts need further work, noting that these signals of quality can accumulate over time. Peer review should enable the broader community to build on top of the work, added Jonah. 

Looking ahead

What became clear from this session is that publishers and funders are actively working towards scientist-driven dissemination and peer review. Peer review of preprints can inform many existing publishing workflows and institutional/grant evaluations. Insights from these new peer review experiments can inform evidence-based modifications to more established review models that will help us improve how scientific discoveries are validated and curated. 

The next steps in this transition requires wide-spread awareness and adoption of preprint peer review and commenting in the scientific community. Some simple actions you can take as a researcher to make this a reality are: 

• Learn about ReimagineReview listed projects
• Take part in reviewing preprints and published manuscripts
• Submit your manuscript to peer review trials and experiments
• Read and consider peer review of preprints when evaluating an article

Help us continue the discussion with the hashtag #ReimagineReview.