You are here

  • Home
  • Online First
  • How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

Article Text

Article menu
Download PDFPDF
Special article
How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal
  1. Aidan S Weitzner1,
  2. Matthew Davis2,
  3. Andrew H Han3,
  4. Olivia O Liu1,
  5. Anuj B Patel4,
  6. Brian D Sites5 and
  7. Steven P Cohen6,7,8
  1. 1Johns Hopkins Medicine, Baltimore, Maryland, USA
  2. 2Departments of Learning Health Sciences and Nursing, University of Michigan, Ann Arbor, Michigan, USA
  3. 3Department of Anesthesiology & Critical Care Medicine, Johns Hopkins Medicine, Baltimore, MD, USA
  4. 4Department of Anesthesiology, Dartmouth Health, Lebanon, New Hampshire, USA
  5. 5Departments of Anesthesiology and Orthopedics, Dartmouth College Geisel School of Medicine, Hanover, New Hampshire, USA
  6. 6Departments of Anesthesiology, Neurology, Physical Medicine & Rehabilitation, Psychiatry and Neurological Surgery, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
  7. 7Departments of Anesthesiology & Critical Care Medicine, Neurology, Physical Medicine & Rehabilitation and Psychiatry & Behavioral Sciences, Johns Hopkins School of Medicine, Baltimore, MD, USA
  8. 8Departments of Physical Medicine & Rehabilitation and Anesthesiology, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
  1. Correspondence to Dr Steven P Cohen, Anesthesiology, Pain Medicine Division, Northwestern University Feinberg School of Medicine, Chicago, IL 60611, USA; steven.cohen{at}northwestern.edu

Abstract

Background Peer review represents a cornerstone of the scientific process, yet few studies have evaluated its association with scientific impact. The objective of this study is to assess the association of peer review scores with measures of impact for manuscripts submitted and ultimately published.

Methods 3173 manuscripts submitted to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021 were analyzed, with those containing an abstract included. Articles were categorized by topic, type, acceptance status, author demographics and open-access status. Articles were scored based on means for the initial peer review where each reviewer’s recommendation was assigned a number: 5 for ‘accept’, 3 for ‘minor revision’, 2 for ‘major revision’ and 0 for ‘reject’. Articles were further classified by whether any reviewers recommended ‘reject’. Rejected articles were analyzed to determine whether they were subsequently published in an indexed journal, and their citations were compared with those of accepted articles when the impact factor was <1.4 points lower than RAPM’s 5.1 impact factor. The main outcome measure was the number of Clarivate citations within 2 years from publication. Secondary outcome measures were Google Scholar citations within 2 years and Altmetric score.

Results 422 articles met inclusion criteria for analysis. There was no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.038, p=0.47), Google Scholar citations (r=0.053, p=0.31) or Altmetric score (p=0.38). There was no significant difference in 2-year Clarivate citations between accepted (median (IQR) 5 (2–10)) and rejected manuscripts published in journals with impact factors >3.7 (median 5 (2–7); p=0.39). Altmetric score was significantly higher for RAPM-published papers compared with RAPM-rejected ones (median 10 (5–17) vs 1 (0–2); p<0.001).

Conclusions Peer review rating scores were not associated with citations, though the impact of peer review on quality and association with other metrics remains unclear.

  • EDUCATION
  • OUTCOMES
  • CHRONIC PAIN

Data availability statement

Data are available upon reasonable request. Data available upon reasonable request.

https://doi.org/10.1136/rapm-2024-105490

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Data availability statement

    Data are available upon reasonable request. Data available upon reasonable request.

    View Full Text

    Footnotes

    • X @sites_brian

    • Contributors Concept and design—SPC. Acquisition, analysis, or interpretation of data—BDS and SPC. Drafting of the manuscript—SPC and ASW. Critical revision of the manuscript for important intellectual content—all authors. Guarantor—SPC.

    • Funding This study was supported in part by the US Department of Defense, Musculoskeletal Injury Rehabilitation Research for Operational Readiness (MIRROR) (HU00011920011).

    • Competing interests BDS, MD and SPC receive support for editorial work at Regional Anesthesia & Pain Medicine. SPC received research support from Avanos and Scilex (paid to institution) and serves as a consultant for Scilex, Avanos, SPR Therapeutics, Persica and Sword.

    • Provenance and peer review Not commissioned; internally peer reviewed.