Exploring ChatGPT's abilities in medical article writing and peer review.

AIM: To evaluate the quality of ChatGPT-generated case reports and assess the ability of ChatGPT to peer review medical articles. METHODS: This study was conducted from February to April 2023. First, ChatGPT 3.0 was used to generate 15 case reports, which were then peer-reviewed by expert human reviewers. Second, ChatGPT 4.0 was employed to peer review 15 published short articles. RESULTS: ChatGPT was capable of generating case reports, but these reports exhibited inaccuracies, particularly when it came to referencing. The case reports received mixed ratings from peer reviewers, with 33.3% of professionals recommending rejection. The reports' overall merit score was 4.9±1.8 out of 10. The review capabilities of ChatGPT were weaker than its text generation abilities. The AI as a peer reviewer did not recognize major inconsistencies in articles that had undergone significant content changes. CONCLUSION: While ChatGPT demonstrated proficiency in generating case reports, there were limitations in terms of consistency and accuracy, especially in referencing.

[The peer-review process: critical issues and challenges from an online survey]. / Il processo della revisione tra pari: criticità e sfide emerse da una survey online.

BACKGROUND: the peer-review process, which is the foundation of modern scientific production, represents one of its essential elements. However, despite numerous benefits, it presents several critical issues. OBJECTIVES: to collect the opinions of a group of researchers from the epidemiological scientific community on peer-review processes. DESIGN: cross-sectional study using a questionnaire evaluation. SETTING AND PARTICIPANTS: a 29-question survey was administered to 516 healthcare professionals through the SurveyMonkey platform. The questions focused on the individual characteristics of the respondents and their perceived satisfaction with some characteristics of the review process as well as their propensity of changing some aspects of it. In addition, three open-ended questions were included, allowing respondents to provide comments on the role that reviewers and the review process should play. Descriptive statistics were produced in terms of absolute frequencies and percentages for the information collected through the questionnaire. Secondly, a multiple logistic regression analysis was conducted to assess the willingness to change certain aspects of peer review, adjusting for covariates such as age, sex, being the author of at least one scientific work, being a reviewer of at least one scientific work, and belonging to a specific discipline. The results are expressed as odds ratios (ORs) and their 95% confidence intervals (95%CI). Text analysis and representation using word cloud were also used for an open-ended question. MAIN OUTCOMES MEASURES: level of satisfaction regarding some characteristics of the peer-review process. RESULTS: a total of 516 participants completed the questionnaire. Specifically, 87.2% (N. 450) of the participants were the authors of at least one scientific publication, 78.7% were first authors at least once (N. 406), and 71.5% acted as reviewers within the peer-review process (N. 369). The results obtained from the multiple logistic regression models did not highlight any significant differences in terms of propensity to change for age and sex categories, except for a lower propensity of the under 35 age group towards unmasking, defined as the presence of reviewers and editorial boards names on the publish article (OR <35 years vs 45-54 years: 0.51; 95%CI 0.29-0.89) and a higher propensity for post-formatting proposals, defined as the possibility of formatting the article following journal guidelines after the acceptance, among those under 45 (OR <35 years vs 45-54 years: 1.73; 95%CI 0.90-3.31; OR 35-44 years vs 45-54 years: 2.02; 95%CI 1.10-3.72). Finally, approximately 50% of respondents found it appropriate to receive credits for the revision work performed, while approximately 30% found it appropriate to receive a discount on publication fees for the same journal in which they acted as reviewers. CONCLUSIONS: the peer-review process is considered essential, but imperfect, by the professionals who participated in the questionnaire, thus providing a clear picture of the value that peer-review adds rigorously to each scientific work and the need to continue constructive dialogue on this topic within the scientific community.

Perspective on the Journal of Physician Assistant Education : From Association Newsletter to Academic Peer-Reviewed Journal Celebrating 25 Years.

ABSTRACT: For 25 years, the Journal of PA Education (JPAE) and its predecessor publications have been the pre-eminent venues for disseminating and promulgating information and research on the physician assistant (PA) profession. In this article, former and current editors in chief have compiled a detailed history of the journal, its development, and its trajectory into the future, outlining the journey taken by Association of PA Programs/PA Education Association to catalog faculty scholarship through a peer-reviewed journal. Allowing for the referencing of articles and thus adding to the body of knowledge on PAs and PA education, JPAE has not only endured but thrived. This article speaks to the collective effort and excellence of staff, and the many volunteer reviewers, feature editors, and editorial board members who have nurtured JPAE along the way through numerous changes, challenges, and triumphs.

Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration. Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making. Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision. Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions?

How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors had roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction was 70% for an approximately 25% acceptance rate. (2) Female authors exhibited a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers were similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agreed with their predicted acceptance probabilities (93% agreement), but there was a notable 7% responses where authors predicted a worse outcome for their better paper. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate-about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review.

Elements of successful NIH grant applications.

Is there a formula for a competitive NIH grant application? The Serenity Prayer may provide one: "Grant me the serenity to accept the things I cannot change, the ability to change the things I can, and the wisdom to know the difference." But how to tell the difference? In this Perspective, we provide an inclusive roadmap-elements of NIH funding. Collectively, we have over 30 y of peer review experience as NIH Scientific Review Officers in addition to over 30 y of program experience as NIH Program Officers. This article distills our NIH experience. We use Euclid's 13-book landmark, The Elements, as our template to humbly share what we learned. We have three specific aims: inform, guide, and motivate prospective applicants. We also address ways that support diversity and inclusion among applicants and young investigators in biomedical research. The elements we describe come from a wide range of sources. Some themes will be general. Some will be specific. All will be candid. The ultimate goal is a competitive application, serenity, and hopefully both.

Conversion Rate of Abstracts Presented at Plastic Surgery The Meeting From 2010 to 2019: A 10-Year Analysis of Factors for Success.

BACKGROUND: Presentations are an important means of knowledge generation. Publication of these studies is important for dissemination of findings beyond meeting attendees. We analyzed a 10-year sample of presented abstracts at Plastic Surgery The Meeting and describe factors that improve rate and speed of conversion to peer-reviewed publication. METHODS: Abstracts presented between 2010 and 2019 at Plastic Surgery The Meeting were sourced from the American Society of Plastic Surgery Abstract Archive. A random sample of 100 abstracts from each year was evaluated. Abstract information and demographics were recorded. The title or author and keywords of each abstract were searched using a standardized workflow to find a corresponding published paper on PubMed, Google Scholar, and Google. Data were analyzed for trends and factors affecting conversion rate. RESULTS: A total of 983 presented abstracts were included. The conversion rate was 54.1%. Residents and fellows constituted the largest proportion of presenters (38.4%). There was a significant increase in medical student and research fellow presenters during the study period (P < 0.001). Conversion rate was not affected by the research rank of a presenter's affiliated institution (ß = 1.001, P = 0.89), geographic location (P = 0.60), or subspecialty tract (P = 0.73). US academics had a higher conversion rate (61.8%) than US nonacademics (32.7%) or international presenters (47.1%) (P < 0.001). Medical students had the highest conversion rate (65.6%); attendings had the lowest (45.0%). Research fellows had the lowest average time to publication (11.6 months, P = 0.007). CONCLUSIONS: Lower levels of training, factors associated with increased institution-level support, and research quality affect rate and time to publication. These findings highlight the success of current models featuring medical student and research fellow-led projects with strong resident and faculty mentorship.

Peer Review and Medical Journals.

Journal editors depend on peer reviewers to make decisions about submitted manuscripts. These reviewers help evaluate the methods, the results, the discussion of the results, and the overall organization and presentation of the manuscript. In addition, reviewers can help identify important mistakes and possible misconduct. Editors frequently have difficulty obtaining enough peer reviews which are submitted in a timely manner. This increases the workload of editors and journal managers and potentially delays the publication of clinical and research studies. This commentary discusses of the importance of peer reviews and make suggestions which potentially can increase the participation of academic faculty and researchers in this important activity.

[Reliability of peer review-like dialogue in the German statutory quality assurance program]. / Reliabilität des Strukturierten Dialogs in der gesetzlich verpflichtenden Qualitätssicherung.

BACKGROUND: Quality measurement in the German statutory program for quality in health care follows a two-step process. For selected areas of health care, quality is measured via performance indicators (first step). Providers failing to achieve benchmarks in these indicators subsequently enter into a peer review process (second step) and are asked by the respective regional authority to provide a written statement regarding their indicator results. The statements are then evaluated by peers, with the goal to assess the provider's quality of care. In the past, similar peer review-based approaches to the measurement of health care quality in other countries have shown a tendency to lack reliability. So far, the reliability of this component of the German statutory program for quality in health care has not been investigated. METHOD: Using logistic regression models, the influence of the respective regional authority on the peer review component of health care quality measurement in Germany was investigated using three exemplary indicators and data from 2016. RESULTS: Both the probability that providers are asked to provide a statement as well as the results produced by the peer review process significantly depend on the regional authority in charge. This dependence cannot be fully explained by differences in the indicator results or by differences in case volume. CONCLUSIONS: The present results are in accordance with earlier findings, which show low reliability for peer review-based approaches to quality measurement. Thus, different results produced by the peer review component of the quality measurement process may in part be due to differences in the way the review process is conducted. This heterogeneity among the regional authorities limits the reliability of this process. In order to increase reliability, the peer review process should be standardized to a higher degree, with clear review criteria, and the peers should undergo comprehensive training for the review process. Alternatively, the future peer review component could be adapted to focus rather on identification of improvement strategies than on reliable provider comparisons.

Journals must expand access to peer review data.

Empirical studies on peer review bias are primarily conducted by people from privileged groups and with affiliations with the journals studied. Data access is one major barrier to conducting peer review research. Accordingly, we propose pathways to broaden access to peer review data to people from more diverse backgrounds.