Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System.

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

The development and evaluation of a provider-focused educational intervention about alcohol use disorder in patients with cirrhosis.

Background: Alcohol use disorder (AUD) is a leading cause of cirrhosis. Insufficient clinician knowledge and comfort managing AUD impacts access to treatment. Using Kern's Framework for Curriculum Development, we aimed to (i) develop and evaluate the effect of an "AUD in cirrhosis" educational intervention on clinicians' knowledge, attitudes, comfort, preparedness, and intention (practice habits) to integrate AUD management into their practice, and (ii) assess clinicians' motivation using Self Determination Theory. Methods: Kern's approach was used for curriculum development. Pilot session feedback informed a three-part flipped-classroom series conducted by interdisciplinary clinicians in hepatology, psychiatry, primary care, and addiction psychology. Participants watched a video followed by a live session focused on (a) withdrawal, (b) screening and brief intervention, and (c) prescribing pharmacotherapy. Questionnaires assessing knowledge and practice habits were adapted from the literature. Attitudes were evaluated using the Short Alcohol and Alcohol Problems Perception Questionnaire (SAAPPQ). Self Determination Theory informed motivation questions. Results: Paired sample t-tests on pre-post questionnaires (n = 229 clinicians; 95 completed questionnaires) revealed significant improvements in preparedness and comfort screening, providing a brief intervention, prescribing pharmacotherapy, and SAAPPQ domains. No significant changes were observed in the intention to prescribe pharmacotherapy. Effect size analysis showed medium to large effects across most topic areas. Conclusions: The developed sessions improved knowledge, attitudes, and practice habits of clinicians caring for this patient population. Given the rise in AUD and significant consequences in cirrhosis, this data offers promise that interactive education may improve practice habits of clinicians interfacing with this patient population.

Clinicians' perspectives and perceived barriers to caring for patients with alcohol use disorder and cirrhosis.

BACKGROUND: Alcohol use disorder (AUD) is one of the leading etiologies for liver cirrhosis and liver transplantation. Few individuals with AUD receive guideline-based care in the form of screening, brief intervention, referral to treatment, or prescription of pharmacotherapy for relapse prevention. We interviewed clinicians across Alberta to assess the current experience and perceived barriers to managing AUD in people who have cirrhosis. The aim of this paper is to summarize these findings to inform the development of an educational intervention. METHODS: We used a qualitative descriptive approach to explore the experiences of clinicians who care for patients with cirrhosis and AUD in Alberta. We conducted semi-structured interviews directed by an interview guide. Interviews were recorded and transcribed verbatim. We used an inductive thematic analysis approach whereby transcripts were coded, with codes grouped into larger categories, then themes. RESULTS: Sixteen clinicians participated in this study. Many participants acknowledged that they do not use a standardized approach to screening, brief intervention, and referral to treatment. Through thematic analysis we identified four themes surrounding barriers to managing AUD in patients with cirrhosis: (i) Practicing within knowledge constraints, (ii) Navigating limited resources and system challenges, (iii) Balancing the complexity of cirrhosis and AUD, and (iv) Acknowledging the influence of provider perceptions on care. CONCLUSION: This article presents the perspectives of clinicians who care for people who have AUD and cirrhosis. Significant barriers exist, including limited knowledge and resources, systemic challenges, and patient complexity. The information gathered will be used to develop an educational intervention that will delve deeper into these issues in order to have the greatest impact on clinicians who routinely interface with this patient population.

Twelve tips for developing an OSCE that measures what you want.

The Objective Structured Clinical Examination (OSCE) is used globally for both high and low stakes assessment. Despite its extensive use, very few published articles provide a set of best practices for developing an OSCE, and of those that do, none apply a modern understanding of validity. This article provides 12 tips for developing an OSCE guided by Kane's validity framework to ensure the OSCE is assessing what it purports to measure. The 12 tips are presented in the order they would be operationalized during OSCE development.

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE.

OBJECTIVE: Although previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills. METHODS: The University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training. RESULTS: For analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales. CONCLUSION: A combined analytic-holistic rating scale increased score reliability and was sensitive to level of training. PRACTICE IMPLICATIONS: Given increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.