Validation of the CoRE Questionnaire for a Medical Journal Peer Review.

If a peer review instrument asks concrete questions (defined as items that can only generate disagreement if reviewers have different degrees of expertise), then questionnaires could become more meaningful in terms of resolving subjectivity thus leading to more reviewer agreement. A concrete item questionnaire with well-chosen questions can also help resolve disagreement when reviewers have the same level of expertise. We have recently created the core-item reviewer evaluation (CoRE) questionnaire for which decision-threshold score levels have been created, but which have not been validated. This prospective validation of these thresholds for the CoRE questionnaire demonstrated strong agreement between reviewer recommendations and their reported score levels when tested prospectively at Clinical Medicine and Research. We conclude that using the CoRE questionnaire will help reduce peer reviewer disagreement. More importantly, when reviewer expertise varies, editors can more easily detect this and decide which opinion reflects the greater expertise.

A core-item reviewer evaluation (CoRE) system for manuscript peer review.

Manuscript peer review is essential for ensuring accountability to all involved in the publication process, including authors, journals, and readers. Lack of consensus regarding what constitutes an accountable manuscript peer review process has resulted in varying practices from one journal to the next. Currently, reviewers are asked to make global judgments about various aspects of a paper for review irrespective of whether guided by a review checklist or not, and several studies have documented gross disagreement between reviewers of the same manuscript. We have previously proposed that the solution may be to direct reviewers to concrete items that do not require global judgments but rather provide a specific choice, along with referee justification for such choices. This study evaluated use of such a system via an international survey of health care professionals who had recently reviewed a health care--related manuscript. Results suggest that use of such a peer review system by reviewers does indeed improve interreviewer agreement, and thus, has the potential to support more consistent and effective peer review, if introduced into journal processes for peer review.

Reliability of reviewer ratings in the manuscript peer review process: an opportunity for improvement.

Accountability to authors and readers cannot exist without proper peer review practices. Thus, the information a journal seeks from its peer reviewers and how it makes use of this information is paramount. Disagreement amongst peer reviewers can be considerable, resulting in very diverse comments to authors. Incorporating a clear scoring system for key concrete items and requiring referees to provide justification for scores may ensure that reviewers contribute in a consistently fair and effective manner. This article evaluates information collected from reviewers and proposes an example of a system that aims to improve accountability, while having the potential to make it easier for reviewers to perform a more objective review.