[Reliability of peer review-like dialogue in the German statutory quality assurance program]. / Reliabilität des Strukturierten Dialogs in der gesetzlich verpflichtenden Qualitätssicherung.

BACKGROUND: Quality measurement in the German statutory program for quality in health care follows a two-step process. For selected areas of health care, quality is measured via performance indicators (first step). Providers failing to achieve benchmarks in these indicators subsequently enter into a peer review process (second step) and are asked by the respective regional authority to provide a written statement regarding their indicator results. The statements are then evaluated by peers, with the goal to assess the provider's quality of care. In the past, similar peer review-based approaches to the measurement of health care quality in other countries have shown a tendency to lack reliability. So far, the reliability of this component of the German statutory program for quality in health care has not been investigated. METHOD: Using logistic regression models, the influence of the respective regional authority on the peer review component of health care quality measurement in Germany was investigated using three exemplary indicators and data from 2016. RESULTS: Both the probability that providers are asked to provide a statement as well as the results produced by the peer review process significantly depend on the regional authority in charge. This dependence cannot be fully explained by differences in the indicator results or by differences in case volume. CONCLUSIONS: The present results are in accordance with earlier findings, which show low reliability for peer review-based approaches to quality measurement. Thus, different results produced by the peer review component of the quality measurement process may in part be due to differences in the way the review process is conducted. This heterogeneity among the regional authorities limits the reliability of this process. In order to increase reliability, the peer review process should be standardized to a higher degree, with clear review criteria, and the peers should undergo comprehensive training for the review process. Alternatively, the future peer review component could be adapted to focus rather on identification of improvement strategies than on reliable provider comparisons.

Peer evaluation and feedback for invasive medical procedures: a systematic review.

BACKGROUND: There is significant variability in the performance and outcomes of invasive medical procedures such as percutaneous coronary intervention, endoscopy, and bronchoscopy. Peer evaluation is a common mechanism for assessment of clinician performance and care quality, and may be ideally suited for the evaluation of medical procedures. We therefore sought to perform a systematic review to identify and characterize peer evaluation tools for practicing clinicians, assess evidence supporting the validity of peer evaluation, and describe best practices of peer evaluation programs across multiple invasive medical procedures. METHODS: A systematic search of Medline and Embase (through September 7, 2021) was conducted to identify studies of peer evaluation and feedback relating to procedures in the field of internal medicine and related subspecialties. The methodological quality of the studies was assessed. Data were extracted on peer evaluation methods, feedback structures, and the validity and reproducibility of peer evaluations, including inter-observer agreement and associations with other quality measures when available. RESULTS: Of 2,135 retrieved references, 32 studies met inclusion criteria. Of these, 21 were from the field of gastroenterology, 5 from cardiology, 3 from pulmonology, and 3 from interventional radiology. Overall, 22 studies described the development or testing of peer scoring systems and 18 reported inter-observer agreement, which was good or excellent in all but 2 studies. Only 4 studies, all from gastroenterology, tested the association of scoring systems with other quality measures, and no studies tested the impact of peer evaluation on patient outcomes. Best practices included standardized scoring systems, prospective criteria for case selection, and collaborative and non-judgmental review. CONCLUSIONS: Peer evaluation of invasive medical procedures is feasible and generally demonstrates good or excellent inter-observer agreement when performed with structured tools. Our review identifies common elements of successful interventions across specialties. However, there is limited evidence that peer-evaluated performance is linked to other quality measures or that feedback to clinicians improves patient care or outcomes. Additional research is needed to develop and test peer evaluation and feedback interventions.

Credentialing, Certification, and Peer Review Essentials for the Neurosurgeon.

Credentialing and certification are essential processes during hiring to ensure that the physician is competent and possesses the qualifications and skill sets claimed. Peer review ensures the continuing evolution of these skills to meet a standard of care. We have provided an overview and discussion of these processes in the United States. Credentialing is the process by which a physician is determined to be competent and able to practice, used to ensure that medical staff meets specific standards, and to grant operative privileges at an institution. Certification is a standardized affirmation of a physician's competence on a nationwide basis. Although not legally required to practice in the United States, many institutions emphasize certification for full privileges on an ongoing basis at a hospital. In the United States, peer review of adverse events is a mandatory prerequisite for accreditation. The initial lack of standardization led to the development of the Health Care Quality Improvement Act, which protects those involved in the peer review process from litigation, and the National Provider Databank, which was established as a national database to track misconduct. A focus on quality improvement in the peer review process can lead to improved performance and patient outcomes. A thorough understanding of the processes of credentialing, certification, and peer review in the United States will benefit neurosurgeons by allowing them to know what institutions are looking for as well and their rights and responsibilities in any given situation. It could also be useful to compare these policies and practices in the United States to those in other countries.

Realizing One's Own Subjectivity: Assessors' Perceptions of the Influence of Training on Their Conduct of Workplace-Based Assessments.

PURPOSE: Assessor training is essential for defensible assessments of physician performance, yet research on the effectiveness of training programs for promoting assessor consistency has produced mixed results. This study explored assessors' perceptions of the influence of training and assessment tools on their conduct of workplace-based assessments of physicians. METHOD: In 2017, the authors used a constructivist grounded theory approach to interview 13 physician assessors about their perceptions of the effects of training and tool development on their conduct of assessments. RESULTS: Participants reported that training led them to realize that there is a potential for variability in assessors' judgments, prompting them to change their scoring and feedback behaviors to enhance consistency. However, many participants noted they had not substantially changed their numerical scoring. Nonetheless, most thought training would lead to increased standardization and consistency among assessors, highlighting a "standardization paradox" in which participants perceived a programmatic shift toward standardization but minimal changes in their own ratings. An "engagement effect" was also found in which participants involved in both tool development and training cited more substantial learnings than participants involved only in training. CONCLUSIONS: Findings suggest that training may help assessors recognize their own subjectivity when judging performance, which may prompt behaviors that support rigorous and consistent scoring but may not lead to perceptible changes in assessors' numeric ratings. Results also suggest that participating in tool development may help assessors align their judgments with the scoring criteria. Overall, results support the continued study of assessor training programs as a means of enhancing assessor consistency.

Quality Assurance Peer Review for Radiotherapy for Haematological Malignancies.

AIMS: To assess the impact of weekly scheduled peer review of radiotherapy planning contours for definitive treatment of haematological malignancies based on rates of recommended changes. MATERIALS AND METHODS: Analysis of a prospective database of contour-based peer review at weekly scheduled meetings for patients undergoing definitive radiotherapy for haematological malignancies at a single large cancer centre between January and December 2018. Recommended changes were prospectively classified as involving the gross tumour volume (GTV), clinical target volume (CTV), planning target volume (PTV), organs at risk or dose fractionation. A univariate analysis was carried out to explore the associations between recommended changes and disease, treatment characteristics and consultant experience. RESULTS: In total, 158/171 (92%) of all cases of haematological malignancy undergoing definitive radiotherapy were prospectively peer reviewed over a 12-month period. Overall, 26/158 (16.5%) changes were recommended within the peer review meetings. This included a total of 27 contour changes (GTV, CTV or PTV) in 25 patients. An increase in CTV was the most common change, occurring in 20/158 (12.7%) cases. One dose-fractionation change was recommended. Additional advice regarding planning technique/set-up was documented in 5/158 (3.2%) patients. There were no significant associations between rates of recommended change and disease type, stage, prior chemotherapy, first line versus refractory/relapse, anatomical site, radiotherapy technique or consultant experience. CONCLUSIONS: Weekly contour-based peer review meetings resulted in a high rate of recommended changes. Compliance was high. Peer review was potentially beneficial for all disease and treatment characteristics and for any degree of clinician experience.

Some assembly required: tracing the interpretative work of Clinical Competency Committees.

OBJECTIVES: This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. METHODS: Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi-structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. RESULTS: Members of the CCCs held an assumption that they would be presented with high-quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed 'problematic evidence' (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with 'problematic evidence', they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. CONCLUSIONS: Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement.

Toxicologic Pathology Forum*: Opinion on Considerations for the Use of Whole Slide Images in GLP Pathology Peer Review.

Whole slide imaging (WSI) technology has advanced to a point where it has replaced the glass slide as the primary means of pathology evaluation within many areas of medical pathology. The deployment of WSI in the field of toxicologic pathology has been delayed by a lack of clarity around the degree of validation required for its use on Good Laboratory Practice (GLP) studies. The current opinion piece attempts to provide a high-level overview of WSI technology to include basic methodology, advantages and disadvantages over a conventional microscope, validation status of WSI scanners, and perceived concerns over regulatory acceptance for the use of WSI for (GLP) peer review in the field of toxicologic pathology. Observations are based on the extensive use by AstraZeneca of WSI for the peer review of non-GLP studies conducted at Charles River facilities and represent the experiences of the authors. Note: This is an opinion article submitted to the Toxicologic Pathology Forum. It represents the views of the author(s). It does not constitute an official position of the Society of Toxicologic Pathology, British Society of Toxicological Pathology, or European Society of Toxicologic Pathology, and the views expressed might not reflect the best practices recommended by these Societies. This article should not be construed to represent the policies, positions, or opinions of their respective organizations, employers, or regulatory agencies.

Integrating Peer Review into Nursing Practice.

: Clinical nurses develop a program to strengthen professional relationships and practice.

Investigating Physicians' Views on Soft Signals in the Context of Their Peers' Performance.

BACKGROUND: Physicians are responsible for delivering high quality of care. In cases of underperformance, hindsight knowledge indicates forewarning being potentially available in terms of concerns, signs, or signals. It is not known how the physicians involved perceive such signals. PURPOSE: To openly explore how physicians perceive soft signals and react on them. METHODS: In-depth interviews with 12 hospital-based physicians from various specialties and institutions following the interpretative phenomenological analysis approach. RESULTS: Physicians perceive soft signals as an observable deviation from a colleague's normal behavior, appearance, or communication. Once observed, they evaluate the signal by reflecting on it personally and/or by consulting others, resulting in either an active (i.e., speaking up) or passive (i.e., avoidance) reaction. Observer sensitivity, closeness to the peer, and cohesion of the physician group influence this observation-evaluation-reaction process. CONCLUSIONS AND IMPLICATIONS: Physicians perceive soft signals as indicators of well-being and collegiality, not as concerns about performance or patient safety. They feel it is their responsibility to be sensitive to and deal with expressed signals. Creating a psychological safe culture could foster such an environment. Because a threat to physicians' well-being may indirectly affect their professional performance, soft signals require serious follow-up.

National Multispecialty Survey Results: Comparing Morbidity and Mortality Conference Practices within and outside Otolaryngology.

Objective The objective is to describe variations in the otolaryngology morbidity and mortality (M&M) conference and to compare with other specialties. Design Cross-sectional survey. Setting The setting included otolaryngology departments across the United States and nonotolaryngology medical and surgical departments at 4 academic medical centers. Subjects and Methods Participants were members of a national otolaryngology quality/safety network and nonotolaryngology quality leaders at 4 large academic hospitals. Surveys were administered January 2017. Respondents described M&M conference practices, goals, and educational role. Results Twenty-eight of 39 individuals representing 28 institutions completed the otolaryngology survey (72% response rate). Of 197 individuals, 60 (30% response rate) representing 11 surgical and 20 nonsurgical specialties completed the comparison survey. Twenty-seven of 28 otolaryngologists (46 of 60 nonotolaryngologists) worked in academic settings. All otolaryngology programs conducted an M&M conference: 54% discussed all adverse events and errors; 32% used standard case selection processes; 70% used structured discussion, usually root cause analysis (64%); and 32% classified harm level. In comparison with other specialties, otolaryngology programs were more likely to discuss all adverse events and errors ( P = .01). Most conferences led to quality projects and intrainstitutional communication: 22% communicated to patients and families; 73% of respondents thought that M&M conferences should be standardized or use "best practices." In both surveys, improving patient care was rated the conference's most important function, followed by trainee education and culture change. Patient care and practice-based learning were rated the most relevant Accreditation Council for Graduate Medical Education Core Competencies in both surveys. Conclusions Academic otolaryngology M&M practices generally align with other specialties, but specifics vary widely, making collaborative quality improvement challenging. Educational and administrative priorities cross specialties. Most respondents thought that standardization and best practices are worthwhile. Nonacademic practice data are needed.

Making the First Cut: An Analysis of Academic Medicine Editors' Reasons for Not Sending Manuscripts Out for External Peer Review.

PURPOSE: Manuscripts submitted to Academic Medicine (AM) undergo an internal editor review to determine whether they will be sent for external peer review. Increasingly, manuscripts are rejected at this early stage. This study seeks to inform scholars about common reasons for internal editor review rejections, increase transparency of the process, and provide suggestions for improving submissions. METHOD: A mixed-methods approach was used to retrospectively analyze editors' free-text comments. Descriptive content analysis was performed of editors' comments for 369 manuscripts submitted between December 2014 and December 2015, and rejected prior to external peer review from AM. Comments were analyzed, categorized, and counted for explicit reasons for rejection. RESULTS: Nine categories of rejection reasons were identified: ineffective study question and/or design (338; 92%); suboptimal data collection process (180; 49%); weak discussion and/or conclusions (139; 37%); unimportant or irrelevant topic to the journal's mission (137; 37%); weak data analysis and/or presentation of results (120; 33%); text difficult to follow, to understand (89; 24%); inadequate or incomplete introduction (67; 18%); other publishing considerations (42; 11%); and issues with scientific conduct (20; 5%). Manuscripts had, on average, three or more reasons for rejection. CONCLUSIONS: Findings suggest that clear identification of a research question that is addressed by a well-designed study methodology on a topic aligned with the mission of the journal would address many of the problems that lead to rejection through the internal review process. The findings also align with research on external peer review.

Morbidity and Mortality Conference for Percutaneous Coronary Intervention.

BACKGROUND: Morbidity and mortality conference is a common educational and quality improvement activity performed in cardiac catheterization laboratories, but best practices for case selection and for maximizing the effectiveness of peer review have not been determined. METHODS AND RESULTS: We reviewed the 10-year percutaneous coronary intervention morbidity and mortality conference experience of an academic medical center. Cases were triggered for review by the occurrence of prespecified procedural events. Summary reports from morbidity and mortality conference discussions were linked to clinical data from the Duke Databank for Cardiovascular Disease to compare baseline and procedural characteristics and to assess postdischarge outcomes. Of 11 786 procedures, from 2004 to 2013, 157 (1.3%) were triggered for review. The most frequent triggering events were cardioversion/defibrillation (72, 0.6%), unplanned use of mechanical circulatory support (64, 0.5%), and major dissection (41, 0.3%). Selected procedures were more likely to include high-risk features, such as ST-segment-elevation myocardial infarction, cardiogenic shock, and multivessel disease, and were associated with higher mortality at 30 days. Only a minority of triggering events were caused by controversial or unacceptable physician behavior. CONCLUSIONS: This 10-year experience outlines the processes for conduct of an effective percutaneous coronary intervention morbidity and mortality conference, including a novel approach to case selection and structured peer review leading to actionable quality interventions. The prespecified clinical triggers, captured in the natural workflow by laboratory staff, identified complex cases that were associated with poor patient outcomes.

Does Peer Review of Radiation Plans Affect Clinical Care? A Systematic Review of the Literature.

PURPOSE: Peer review is a recommended component of quality assurance in radiation oncology; however, it is resource-intensive and its effect on patient care is not well understood. We conducted a systematic review of the published data to assess the reported clinical impact of peer review on radiation treatment plans. METHODS AND MATERIALS: A systematic review of published English studies was performed in accordance with the PRISMA guidelines using the MEDLINE and EMBASE databases and abstracts published from major radiation oncology scientific meeting proceedings. For inclusion, the studies were required to report the effect of peer review on ≥1 element of treatment planning (eg, target volume or organ-at-risk delineation, dose prescription or dosimetry). RESULTS: The initial search strategy identified 882 potentially eligible studies, with 11 meeting the inclusion criteria for full-text review and final analysis. Across a total of 11,491 patient cases, peer review programs led to modifications in a weighted mean of 10.8% of radiation treatment plans. Five studies differentiated between major and minor changes and reported weighted mean rates of change of 1.8% and 7.3%, respectively. The most common changes were related to target volume delineation (45.2% of changed plans), dose prescription or written directives (24.4%), and non-target volume delineation or normal tissue sparing (7.5%). CONCLUSIONS: Our findings suggest that peer review leads to changes in clinical care in approximately 1 of every 9 cases overall. This is similar to the reported rates of change in peer review studies from other oncology-related specialties, such as radiology and pathology.

Examining Quality Management Audits in Nuclear Medicine Practice as a lifelong learning process: opportunities and challenges to the nuclear medicine professional and beyond.

This essay will explore the critical issues and challenges surrounding lifelong learning for professionals, initially exploring within the profession and organizational context of nuclear medicine practice. It will critically examine how the peer-review process called Quality Management Audits in Nuclear Medicine Practice (QUANUM) of the International Atomic Energy Agency (IAEA) can be considered a lifelong learning opportunity to instill a culture of quality to improve patient care and elevate the status of the nuclear medicine profession and practice within the demands of social changes, policy, and globalization. This will be explored initially by providing contextual background to the identity of the IAEA as an organization responsible for nuclear medicine professionals, followed by the benefits that QUANUM can offer. Further key debates surrounding lifelong learning, such as compulsification of lifelong learning and impact on professional change, will then be weaved through the discussion using theoretical grounding through a qualitative review of the literature. Keeping in mind that there is very limited literature focusing on the implications of QUANUM as a lifelong learning process for nuclear medicine professionals, this essay uses select narratives and observations of QUANUM as a lifelong learning process from an auditor's perspective and will further provide a comparative perspective of QUANUM on the basis of other lifelong learning opportunities such as continuing professional development activities and observe parallelisms on its benefits and challenges that it will offer to other professionals in other medical speciality fields and in the teaching profession.

Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy.

UNLABELLED: Because surgical skill may be a key determinant of patient outcomes, there is growing interest in skill assessment. In the Michigan Urological Surgery Improvement Collaborative (MUSIC), we assessed whether peer and crowd-sourced (ie, layperson) video review of robot-assisted radical prostatectomy (RARP) could distinguish technical skill among practicing surgeons. A total of 76 video clips from 12 MUSIC surgeons consisted of one of four parts of RARP and underwent blinded review by MUSIC peer surgeons and prequalified crowd-sourced reviewers. Videos were rated for global skill (Global Evaluation Assessment of Robotic Skills) and procedure-specific skill (Robotic Anastomosis and Competency Evaluation). We fit linear mixed-effects models to estimate mean peer and crowd ratings for each video. Individual video ratings were aggregated to calculate surgeon skill scores. Peers (n=25) completed 351 video ratings over 15 d, whereas crowd-sourced reviewers (n=680) completed 2990 video ratings in 38 h. Surgeon global skill scores ranged from 15.8 to 21.7 (peer) and from 19.2 to 20.9 (crowd). Peer and crowd ratings demonstrated strong correlation for both global (r=0.78) and anastomosis (r=0.74) skills. The two groups consistently agreed on the rank order of lower scoring surgeons, suggesting a potential role for crowd-sourced methodology in the assessment of surgical performance. Lack of patient outcomes is a limitation and forms the basis of future study. PATIENT SUMMARY: We demonstrated the large-scale feasibility of assessing the technical skill of robotic surgeons and found that online crowd-sourced reviewers agreed with experts on the rank order of surgeons with the lowest technical skill scores.

[Quality Assurance in Sociomedical Evaluation by Peer Review: A Pilot Project of the German Statutory Pension Insurance]. / Qualitätssicherung in der Sozialmedizinischen Begutachtung durch Peer Review - ein Pilotprojekt der Deutschen Rentenversicherung.

BACKGROUND: The sociomedical evaluation by the German Pension Insurance serves the purpose of determining entitlement to disability pensions. A quality assurance concept for the sociomedical evaluation was developed, which is based on a peer Review process. Peer review is an established process of external quality assurance in health care. The review is based on a hierarchically constructed manual that was evaluated in this pilot project. METHODS: The database consists of 260 medical reports for disability pension of 12 pension insurance agencies. 771 reviews from 19 peers were included in the evaluation of the inter-rater reliability. Kendall's coefficient of concordance W for more than 2 raters is used as primary measure of inter-rater reliability. RESULTS: Reliability appeared to be heterogeneous. Kendalls W varies for the particular criteria from 0.09 to 0.88 and reached for primary criterion reproducibility a value of 0.37. CONCLUSION: The reliability of the manual seemed acceptable in the context of existing research data and is in line with existing peer review research outcomes. Nevertheless, the concordance is limited and requires optimisation. Starting points for improvement can be seen in a systematic training and regular user meetings of the peers involved.

Daily peer review of abnormal cervical smears in the assessment of individual practice as an additional method of internal quality control.

OBJECTIVE: An important internal quality control system used in the Cancer Prevention and Research Institute cytopathology laboratory in Florence is the peer review procedure, based on the review of all abnormal cytological smears which routinely emerge. Peer review is an important training opportunity for all cytologists, especially for those with less experience. This article shows the results of the peer review procedure. METHODS: Of the 63 754 Papanicolaou (Pap) smears screened in 2011, 1086 were considered to be abnormal [at least atypical squamous cells of undetermined significance (ASC-US+)] on primary screening (selected by a single cytologist) and were subjected to the peer review procedure. The overall performance of the laboratory's cytologists was evaluated using a multiple rater analysis and the comparison of each cytologist with the final diagnosis. Further, the agreement was assessed by means of Cohen's kappa and weighted kappa statistics. RESULTS: In general, a moderate/substantial level of agreement between the ten cytologists and the final diagnoses was evident. Kappa values for each reader compared with the final diagnosis ranged from 0.54 to 0.69. The overall kappa value was 0.62 [95% confidence interval (CI), 0.58-0.66] and overall weighted kappa value was 0.76 (95% CI, 0.74-0.79). The category-specific agreement showed the lowest values for atypical squamous cells, cannot exclude high-grade squamous intraepithelial lesion (ASC-H). CONCLUSION: In summary, peer review represents an important internal quality control in the evaluation and improvement of inter-observer agreement and of the functioning of the laboratory as a whole. Multi-head microscope sessions may improve particularly the reproducibility of borderline diagnoses and, above all, can be an important training contribution for cytologists.

Meaningful Peer Review in Radiology: A Review of Current Practices and Potential Future Directions.

The current practice of peer review within radiology is well developed and widely implemented compared with other medical specialties. However, there are many factors that limit current peer review practices from reducing diagnostic errors and improving patient care. The development of "meaningful peer review" requires a transition away from compliance toward quality improvement, whereby the information and insights gained facilitate education and drive systematic improvements that reduce the frequency and impact of diagnostic error. The next generation of peer review requires significant improvements in IT functionality and integration, enabling features such as anonymization, adjudication by multiple specialists, categorization and analysis of errors, tracking, feedback, and easy export into teaching files and other media that require strong partnerships with vendors. In this article, the authors assess various peer review practices, with focused discussion on current limitations and future needs for meaningful peer review in radiology.

Peer Review: Lessons Learned in A Pediatric Radiology Department.

The purpose of this article is to illustrate types of diagnostic errors and feedback given to radiologists, using cases to support and clarify these categories. A comment-enhanced peer review system may be leveraged to generate a comprehensive feedback categorization scheme. These include errors of observation, errors of interpretation, inadequate patient data gathering, errors of communication, interobserver variability, informational feedback, and compliments. Much of this feedback is captured through comments associated with interpretative agreements.