Poorly performing physicians: does the Script Concordance Test detect bad clinical reasoning?

INTRODUCTION: Evaluation of poorly performing physicians is a worldwide concern for licensing bodies. The Collège des Médecins du Québec currently assesses the clinical competence of physicians previously identified with potential clinical competence difficulties through a day-long procedure called the Structured Oral Interview (SOI). Two peer physicians produce a qualitative report. In view of remediation activities and the potential for legal consequences, more information on the clinical reasoning process (CRP) and quantitative data on the quality of that process is needed. This study examines the Script Concordance Test (SCT), a tool that provides a standardized and objective measure of a specific dimension of CRP, clinical data interpretation (CDI), to determine whether it could be useful in that endeavor. METHODS: Over a 2-year period, 20 family physicians took, in addition to the SOI, a 1-hour paper-and-pencil SCT. Three evaluators, blind as to the purpose of the experiment, retrospectively reviewed SOI reports and were asked to estimate clinical reasoning quality. Subjects were classified into 2 groups (below and above median of the score distribution) for the 2 assessment methods. Agreement between classifications is estimated with the use of the Kappa coefficient. RESULTS: Intraclass correlation for SOI was 0.89. Cronbach alpha coefficient for the SCT was 0.90. Agreement between methods was found for 13 participants (Kappa: 0.30, P = 0.18), but 7 out of 20 participants were classified differently in both methods. All participants but 1 had SCT scores below 2 SD of panel mean, thus indicating serious deficiencies in CDI. DISCUSSION: The finding that the majority of the referred group did so poorly on CDI tasks has great interest for assessment as well as for remediation. In remediation of prescribing skills, adding SCT to SOI is useful for assessment of cognitive reasoning in poorly performing physicians. The structured oral interview should be improved with more precise reporting by those who assess the clinical reasoning process of examinees, and caution is recommended in interpreting SCT scores; they reflect only a part of the reasoning process.

Does accreditation stimulate change? A study of the impact of the accreditation process on Canadian healthcare organizations.

BACKGROUND: One way to improve quality and safety in healthcare organizations (HCOs) is through accreditation. Accreditation is a rigorous external evaluation process that comprises self-assessment against a given set of standards, an on-site survey followed by a report with or without recommendations, and the award or refusal of accreditation status. This study evaluates how the accreditation process helps introduce organizational changes that enhance the quality and safety of care. METHODS: We used an embedded multiple case study design to explore organizational characteristics and identify changes linked to the accreditation process. We employed a theoretical framework to analyze various elements and for each case, we interviewed top managers, conducted focus groups with staff directly involved in the accreditation process, and analyzed self-assessment reports, accreditation reports and other case-related documents. RESULTS: The context in which accreditation took place, including the organizational context, influenced the type of change dynamics that occurred in HCOs. Furthermore, while accreditation itself was not necessarily the element that initiated change, the accreditation process was a highly effective tool for (i) accelerating integration and stimulating a spirit of cooperation in newly merged HCOs; (ii) helping to introduce continuous quality improvement programs to newly accredited or not-yet-accredited organizations; (iii) creating new leadership for quality improvement initiatives; (iv) increasing social capital by giving staff the opportunity to develop relationships; and (v) fostering links between HCOs and other stakeholders. The study also found that HCOs' motivation to introduce accreditation-related changes dwindled over time. CONCLUSIONS: We conclude that the accreditation process is an effective leitmotiv for the introduction of change but is nonetheless subject to a learning cycle and a learning curve. Institutions invest greatly to conform to the first accreditation visit and reap the greatest benefits in the next three accreditation cycles (3 to 10 years after initial accreditation). After 10 years, however, institutions begin to find accreditation less challenging. To maximize the benefits of the accreditation process, HCOs and accrediting bodies must seek ways to take full advantage of each stage of the accreditation process over time.