Promotion in Place: A Model for Competency-Based, Time-Variable Graduate Medical Education.

PROBLEM: Competency-based medical education is increasingly regarded as a preferred framework for physician training, but implementation is limited. U.S. residency programs remain largely time based, with variable assessments and limited opportunities for individualization. Gaps in graduates' readiness for unsupervised care have been noted across specialties. Logistical barriers and regulatory requirements constrain movement toward competency-based, time-variable (CBTV) graduate medical education (GME), despite its theoretical benefits. APPROACH: The authors describe a vision for CBTV-GME and an implementation model that can be applied across specialties. Termed "Promotion in Place" (PIP), the model relies on enhanced assessment, clear criteria for advancement, and flexibility to adjust individuals' responsibilities and time in training based on demonstrated competence. PIP allows a resident's graduation to be advanced or delayed accordingly. Residents deemed competent for early graduation can transition to attending physician status within their training institution and benefit from a period of "sheltered independence" until the standard graduation date. Residents who need extended time to achieve competency have graduation delayed to incorporate additional targeted education. OUTCOMES: A proposal to pilot the PIP model of CBTV-GME received funding through the American Medical Association's "Reimagining Residency" initiative in 2019. Ten of 46 residency programs in a multihospital system expressed interest and pursued initial planning. Seven programs withdrew for reasons including program director transitions, uncertainty about resident reactions, and the COVID-19 pandemic. Three programs petitioned their specialty boards for exemptions from time-based training. One program was granted the needed exemption and launched a PIP pilot, now in year 4, demonstrating the feasibility of implementing this model. Implementation tools and templates are described. NEXT STEPS: Larger-scale implementation with longer-term assessment is needed to evaluate the impact and generalizability of this CBTV-GME model.

A pilot clinical skills coaching program to reimagine remediation: a cohort study.

Background New approaches are needed to improve and destigmatize remediation in undergraduate medical education (UME).  The COVID-19 pandemic magnified the need to support struggling learners to ensure competency and readiness for graduate medical education (GME).  Clinical skills (CS) coaching is an underutilized approach that may mitigate the stigma of remedial learning. Methods A six-month CS coaching pilot was conducted at Harvard Medical School (HMS) as a destigmatized remedial learning environment for clerkship and post-clerkship students identified as 'at risk' based on objective structured clinical examinations (OSCE).  The pilot entailed individual and group coaching with five faculty, direct bedside observation of CS, and standardized patient encounters with video review. Strengths-based coaching principles and appreciative inquiry were emphasized.  Results Twenty-three students participated in the pilot: 14 clerkship students (cohort 1) and 9 post-clerkship students (cohort 2).  All clerkship students (cohort 1) demonstrated sustained improvement in CS across three OSCEs compared to baseline: at pilot close, at 6-months post pilot, and at 21-24 months post-pilot all currently graduating students (10/10, 100%) passed the summative OSCE, an HMS graduation requirement. All post-clerkship students (cohort 2) passed the HMS graduation OSCE (9/9,100%). Feedback survey results included clerkship students (9/14; 64%) and post-clerkship students (7/9; 78%); all respondents unanimously agreed that individual coaching was "impactful to my clinical learning and practice". Faculty and leadership fully supported the pilot as a destigmatized and effective approach to remediation.  Conclusion Remediation has an essential and growing role in medical schools.  CS coaching for remedial learning can reduce stigma, foster a growth mindset, and support sustained progress for 'at risk' early clerkship through final year students. An "implementation template" with suggested tools and timelines can be locally adapted to guide CS coaching for UME remediation. The CS coaching pilot model is feasible and can be generalized to many UME programs.

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement.

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Use of an objective structured clinical exam (OSCE) for early identification of communication skills deficits in interns.

INTRODUCTION: There is limited information about whether OSCE during GME orientation can identify trainee communication deficits before these become evident via clinical performance evaluations. METHODS: Ninety-seven interns matriculating to eight residency programs in six specialties at four hospitals participated in a nine-station communication skills OSCE. Ratings were based on the "Kalamazoo, adapted" communication skills checklist. Possible association with intern performance evaluations was assessed by repeated-measures logistic regression and ROC curves were generated. RESULTS: The mean OSCE score was 4.08 ± 0.27 with a range of 3.3-4.6. Baseline OSCE scores were associated with subsequent communication concerns recorded by faculty, based on 1591 evaluations. A 0.1-unit decrease in the OSCE communication score was associated with an 18% higher odds of being identified with a communication concern by faculty evaluation (odds ratio 1.18, 95% CI 1.01-1.36, p = 0.034). ROC curves did not demonstrate a "cut-off" score (AUC= 0.558). Non-faculty evaluators were 3-5 times more likely than faculty evaluators to identify communication deficits, based on 1900 evaluations. CONCLUSIONS: Lower OSCE performance was associated with faculty communication concerns on performance evaluations; however, a "cut-off" score was not demonstrated that could identify trainees for potential early intervention. Multi-source evaluation also identified trainees with communication skills deficits.

Evaluating the Evaluators: Implementation of a Multi-Source Evaluation Program for Graduate Medical Education Program Directors.

BACKGROUND: Multi-source evaluation has demonstrated value for trainees, but is not generally provided to residency or fellowship program directors (PDs). OBJECTIVE: To develop, implement, and evaluate a PD multi-source evaluation process. METHODS: Tools were developed for PD evaluation by trainees, department chairs, and graduate medical education (GME) leadership. Evaluation questions were based on PD responsibilities, including Accreditation Council for Graduate Medical Education (ACGME) requirements. A follow-up survey assessed the process. RESULTS: Evaluation completion rates were as follows: trainees in academic year 2012-2013, 53% (958 of 1824), and in academic year 2013-2014, 42% (800 of 1898); GME directors in 2013-2014, 100% (95 of 95); and chairs/chiefs in 2013-2014, 92% (109 of 118). Results of a follow-up survey of PDs (66%, 59 of 90) and chairs (74%, 48 of 65) supports the evaluations' value, with 45% of responding PDs (25 of 56) and 50% of responding chairs (21 of 42) characterizing them as "extremely" or "quite" useful. Most indicated this was the first written evaluation they had received (PDs 78%, 46 of 59) or provided (chairs 69%, 33 of 48) regarding the PD role. More than 60% of PD (30 of 49) and chair respondents (24 of 40) indicated trainee feedback was "extremely" or "quite" useful, and nearly 50% of PDs (29 of 59) and 21% of chairs (10 of 48) planned changes based on the results. Trainee response rates improved in 2014-2015 (52%, 971 of 1872) and 2015-2016 (69%, 1276 of 1837). CONCLUSIONS: In our institution, multi-source evaluation of PDs was sustained over 4 years with acceptable and improving evaluation completion rates. The process and assessment tools are potentially transferrable to other institutions.

Protecting an endangered species: training physicians to conduct clinical research.

PURPOSE: The Program in Clinical Effectiveness (PCE) at Harvard School of Public Health is a postgraduate program emphasizing clinical research. The authors sought to evaluate the research careers of physician graduates and to determine correlates of National Institutes of Health (NIH) grant funding. METHOD: In 2006, all 1,489 graduates from 1986-2005 were sent a 48-item survey that collected information on demographics, program experience, chosen career path, grant awards, and research pursued postprogram. Reported NIH grants were verified on the NIH Computer Retrieval of Information on Scientific Projects Web site. Cox proportional hazard regression was used to determine participant and program features associated with NIH grant funding. RESULTS: Overall, 994 of the 1,365 located graduates (73%) responded to the survey. Graduates pursued research in the following areas: 437 respondents (44%) pursued clinical trials, 537 (54%) pursued epidemiology, and 408 (41%) pursued health services research. A total of 156 respondents (24%) were principal investigators on an NIH grant. Correlates of receiving NIH grant funding included age less than 40 years at time of program enrollment (hazard ratio [HR] 1.87, CI 1.03, 3.41), generalist status (HR 1.57, CI 1.14, 2.16), and publishing research begun as course projects (HR 1.65, CI 1.19, 2.31). Gender, academic status at enrollment, ethnicity, tuition sponsorship, and earning an advanced degree were not associated with receipt of NIH grant funding. CONCLUSIONS: Physicians who enrolled in the PCE at an early age and generalist physicians were particularly successful in establishing careers as clinician-investigators. Programs such as the PCE can help to sustain the workforce of physician-investigators.