Policies

Our policies

We welcome listings describing experimental peer review or scholarly evaluation projects or trials. Contributions may cover any scholarly field. Projects in the planning stages (ie, without a web presence) are also welcome to be listed, but for these we require a link to a publicly-distributed and freely available white paper archived in a commonly-used repository (for example, OSF, Zenodo, or Internet Archive).

ASAPbio reserves the right to moderate the content of ReimagineReview for scope and clarity.

By posting content to ReimagineReview, users agree to release submitted text under a Creative Commons Attribution 4.0 International license. Users grant ReimagineReview the right to display submitted logos, images, videos, and brands; for these, all rights reserved are reserved, except where otherwise noted.

Eligibility

ReimagineReview welcomes listings describing projects or trials involving experimental peer review or evaluation of scholarly outputs. Contributions may cover any scholarly field and can be operated by established journals, new start-ups, or groups of individuals. Projects can be completed trials with documented outcomes, be currently active, or be in the planning stages. For projects without an active web presence, we require a link to a publicly-distributed and freely available white paper, protocol, or publication archived in a commonly-used repository (for example, OSF, Zenodo, PubMed Central, or Internet Archive).

In order to be eligible for inclusion in the directory, the project must:

• feature new peer review/evaluation models, formats or technologies, for scientific publishing as a whole or for the disciplinary scope it serves
• address one or more major issues found in traditional peer review systems (transparency, speed, recognition for reviewing, quality, incentives, curation, bias)

The following examples would all be suitable as directory listings:

• A trial run by a journal or group of journals
• A post-publication peer review service
• An overlay journal
• A browser extension that allows readers to upvote papers

Since they do not represent interventions, analyses or meta-analyses should not be represented with listings. However, we welcome researchers to get in touch with us to author a guest post on the blog describing their findings. Please contact victoria.yan@asapbio.org.

Outcomes reporting

Stimulating rigorous evaluation of peer review interventions is a major goal of ReimagineReview. As such, we encourage projects to measure and share metrics and best practices for their collection and dissemination. In the future, we plan to introduce a “Reporting results” badge which will confer additional visibility to projects transparently reporting outcomes. The criteria for this badge will be determined by community consultation; but we are considering the following factors for discussion:

• For innovations or trials within journals, the following aggregate outcomes:
  • Author opt-in rate or submission rate
    • Rate of subsequent drop-out or withdrawal from peer review process
  • Reviewer invitation acceptance rate
  • Reviewer recommendation outcomes – ie accept, accept with revisions, reject, etc (if applicable)
  • Editorial decision outcomes
  • Total time to publication
  • All outcomes reported relative to metrics for a control group, eg submissions not part of the trial.
  • Ideally, all outcomes would be broken down by career stage, gender, geographic region, and ethnicity of authors and/or reviewers where privacy can be maintained.
• For non-journal venues, the following aggregate outcomes:
  • Distribution of number of comments per paper
  • Distribution of word length of comments/reviews
  • Survey of user satisfaction with the platform
  • Ideally, report career stage, gender, geographic region, and ethnicity of participating groups where privacy can be maintained.
• For all projects:
  • Preregistration of experiments.
  • Number of manuscripts and reviews/comments collected
  • Discuss ethical research practices employed, if applicable
  • A link to (preferrably open) data
  • A discussion of how the project affects core issues in scholarly communication (adapted from Cohen & Altman, 2018):
    • Rigor of the review process
    • Bias in review
    • Inclusion and equity of diverse researchers
    • Time to publication
    • Cost of the review process and/or publication
    • Accountability of decision-making (eg transparency in the editorial process)
    • Benefit to society (eg accessibility of information, etc)

We look forward to discussing best practices with the community.

Moderation

ASAPbio reserves the right to moderate the content of ReimagineReview for scope and clarity.

Registered users may report inaccurate or offensive listings by using the “Report” button that appears at the top right of the listings.

Intellectual property

By posting content to ReimagineReview, users agree to release submitted text under a Creative Commons Attribution 4.0 International license. Users grant ReimagineReview the right to display submitted logos, images, videos, and brands; for these, all rights reserved are reserved, except where otherwise noted.