How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

It Takes a Village: An Interdisciplinary Approach to Preparing Internal Medicine Residents to Care for Patients at the Intersection of Women's Health, Gender-Affirming Care, and Health Disparities.

Objective: To create an interdisciplinary curriculum to teach key topics at the intersection of women's health, gender-affirming care, and health disparities to internal medicine (IM) residents. Materials and Methods: A core team of faculty from IM, Obstetrics and Gynecology, and Surgery partnered with faculty and fellows from other disciplines and with community experts to design and deliver the curriculum. The resulting curriculum consisted of themed half-day modules, each consisting of three to four inter-related topics, updated and repeated on an ∼3-year cycle. Health equity was a focus of all topics. Module delivery used diverse interactive learning strategies. Modules have been presented to ∼175 residents annually, beginning in 2015. To assess the curriculum, we used formative evaluation methods, using primarily anonymous, electronic surveys, and collected quantitative and qualitative data. Most surveys assessed resident learning by quantifying residents' self-reported comfort with skills taught in the module pre- and postsession. Results: Of 131 residents who completed an evaluation in 2022/23, 121 (90%) "somewhat" or "strongly" agreed with their readiness to perform a range of skills taught in the module. In all previous years where pre- and postsurveys were used to evaluate modules, we observed a consistent meaningful increase in the proportion of residents reporting high levels of comfort with the material. Residents particularly valued interactive teaching methods, and direct learning from community members and peers. Conclusion: Our interdisciplinary curriculum was feasible, valued by trainees, and increased resident learning. The curriculum provides a template to address equity issues across a spectrum of women's and gender-affirming care conditions that can be used by other institutions in implementing similar curricula.

A Qualitative Study of Drivers for Use of the Primary Care Exception Among Internal Medicine Teaching Faculty.

BACKGROUND: The Primary Care Exception (PCE) is a billing rule from the Centers for Medicare and Medicaid Services (CMS) that allows supervising physicians to bill for ambulatory care provided by a resident without their direct supervision. There has been increased focus on entrustment as a method to assess readiness for unsupervised practice. OBJECTIVE: To understand the factors influencing attending physicians' use of the PCE in ambulatory settings and identify common themes defining what motivates faculty preceptors to use the PCE. APPROACH: This was a qualitative exploratory study. Participants were interviewed one-on-one using a semi-structured template informed by the entrustment literature. Analysis was conducted using a thematically framed, grounded theory-based approach to identify major themes and subthemes. PARTICIPANTS: Twenty-seven internal medicine teaching faculty took part in a multi-institutional study representing four residency training programs across two academic medical centers in Connecticut. KEY RESULTS: Four predominant categories of themes influencing PCE use were identified: (1) clinical environment factors, (2) attending attitudes, (3) resident characteristics, and (4) patient attributes. An attending's "internal rules" drawn from prior experiences served as a significant driver of PCE non-use regardless of the trainee, patient, or clinical context. A common conflict existed between using the PCE to promote resident autonomy versus waiving the PCE to promote safety. CONCLUSIONS: The PCE can serve as a tool to support resident autonomy, confidence, and overall clinical efficiency. Choice of PCE use by attendings involved complex internal decision-making schema balancing internal, patient, resident, and environmental-related factors. The lack of standardized processes in competency evaluation may increase susceptibility to biases, which could be mitigated by applying standardized modes of assessment that encompass shared principles.

Twelve tips for integrating podcasts into medical education curricula.

Engagement with medical education podcasts among health professions learners has been increasing steadily over the last several years. Prior '12 Tips' publications have focused on helping medical educators create and publish their own podcasts. However, there is a gap in available resources to help educators incorporate an already existing and growing library of medical education podcasts into their curriculum rather than create their own. In this paper, the authors provide medical educators '12 Tips' on how to integrate podcasts into their teaching sessions grounded in fundamental principles of curriculum development, cognitive science, and accessibility.

"Zoom"ing to the Kitchen: A Novel Approach to Virtual Nutrition Education for Medical Trainees.

While nutritional interventions are first-line therapy for many chronic diseases, most medical trainees receive minimal nutrition education, leaving them unprepared to address nutritional issues with patients. An interactive, single-session, virtual nutrition curriculum was taught online to 80 physician assistant (PA) students. Topics included plant-based nutrition, dietary history-taking and counseling, and culinary medicine. Students were surveyed before, immediately after, and four weeks after the curriculum to assess changes to nutrition-related knowledge, attitudes, confidence, and personal dietary behaviors. Seventy-three PA students (91%) completed the pre-survey, 76 (95%) completed the post-survey, and 42 (52.5%) completed the delayed post-survey. Knowledge scores increased immediately post-intervention (48.9% to 78.9%; p < 0.001) and persisted four weeks later (78.9% to 75.8%; p = 0.54). Post-intervention, students felt more confident in dietary history-taking (55% vs. 95%; p = 0.001) and nutrition counseling (53% vs. 84%; p = 0.003) and agreed that dietary changes alone could reverse type 2 diabetes (74% vs. 97%; p = 0.027) and coronary artery disease (66% vs. 92%; p = 0.039). Curricula using virtual teaching kitchens may be a scalable approach to nutrition education for medical trainees.

A Qualitative Study on the Outcomes of Participation in a Longitudinal Clinician Educator Curriculum During Internal Medicine Residency.

Background: Despite increasing prevalence of longitudinal clinician educator tracks (CETs) within graduate medical education (GME) programs, the outcomes of these curricula and how participation in these tracks affects early career development remains incompletely understood. Objective: To describe the experience and outcomes of participating in a CET and its effects on recent internal medicine residency graduates' perceived educator skills and early career development. Methods: We conducted a qualitative study between July 2019 and January 2020 using in-depth semi-structured interviews of recently graduated physicians from 3 internal medicine residencies at one academic institution who had participated in a CET, the Clinician Educator Distinction (CED). Iterative interviews and data analysis was performed via an inductive, constructionist, thematic analysis approach by 3 researchers to develop a coding and thematic structure. Results were sent electronically to participants for member checking. Results: From 21 (out of 29 eligible) participants, thematic sufficiency was reached at 17 interviews. Four themes related to the CED experience were identified: (1) motivation to go beyond the expectations of residency; (2) educator development outcomes from Distinction participation; (3) factors enabling curricular efficacy; and (4) opportunities for program improvement. A flexible curriculum with experiential learning, observed teaching with feedback, and mentored scholarship allowed participants to enhance teaching and education scholarship skills, join a medical education community, transform professional identities from teachers to educators, and support clinician educator careers. Conclusions: This qualitative study of internal medicine graduates identified key themes surrounding participation in a CET during training, including positively perceived educator development outcomes and themes surrounding educator identity formation.

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities.

Background: Existing assessment tools for competence in critical care ultrasound (CCUS) have limited scope and interrupt clinical workflow. The framework of entrustable professional activities (EPAs) is well suited to developing an assessment tool that is comprehensive and readily integrated into the intensive care unit (ICU) training environment. Objective: This study sought to design an EPA-based tool to assess competence in CCUS for pulmonary and critical care fellows and to assess the validity and reliability of the tool. Methods: Eight experts in CCUS met to define the core EPAs for CCUS. A nominal group technique was used to reach consensus. An assessment tool was created based on the EPAs with a modified Ottawa entrustability scale. Trained faculty evaluated pulmonary and critical care fellows using this tool in the ICU over a 6-month study period at a single institution. An assessment of validity of the EPA-based tool is made with four sources of validity evidence: content, response process, reliability, and relation to other variables. Reliability and response process data were generated using generalizability theory analysis to estimate sources of variance in entrustment scores. Analysis of response process validity and validity by relation to other variables was performed using regression models. Results: Fifty-four assessments were recorded during the study period, conducted on 23 trainees by 13 faculty. Content validity of the tool was demonstrated using expert consensus and published guidelines from critical care societies to define the EPAs. Response process validity was demonstrated by the low variance in entrustment scores due to evaluators (0.086 or 6%) and high agreement between score and trainee self-assessment (regression coefficient, 0.82; P < 0.0001). Reliability was demonstrated by the high "true" variance in entrustment score attributable to the trainee: 0.674 or 45%. Validity by relation to other variables was demonstrated using regression analysis to show correlation between entrustment score and the number of times a fellow has performed an EPA (regression coefficient, 0.023; P < 0.0001). Conclusion: An EPA-based assessment tool for competence in CCUS was created. We obtained sufficient validity evidence on three of the diagnostic EPAs. Procedural EPAs were infrequently assessed, limiting generalizability in this subgroup.

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students' Self-Assessed Skills by Specialty.

BACKGROUND: The Association of American Medical Colleges described 13 Core Entrustable Professional Activities (EPAs) that graduating students should be prepared to perform under indirect supervision on day one of residency. Surgery program directors recently recommended entrustability in these Core EPAs for incoming surgery interns. We sought to determine if graduating students intending to enter surgery agreed they had the skills to perform these Core EPAs. STUDY DESIGN: Using de-identified, individual-level data collected from and about 2019 Association of American Medical Colleges Graduation Questionnaire respondents, latent profile analysis was used to group respondents based on their self-assessed Core EPAs skills' response patterns. Associations between intended specialty, among other variables, and latent profile analysis group were assessed using independent sample t -tests and chi-square tests and multivariable logistic regression methods. RESULTS: Among 12,308 Graduation Questionnaire respondents, latent profile analysis identified 2 respondent groups: 7,863 (63.9%) in a high skill acquisition agreement (SAA) group and 4,445 (36.1%) in a moderate SAA group. Specialty was associated with SAA group membership (p < 0.001), with general surgery, orthopaedic surgery, and emergency medicine respondents (among others) overrepresented in the high SAA group. In the multivariable logistic regression models, each of anesthesiology, ophthalmology, pediatrics, psychiatry, and radiology (vs general surgery) specialty intention was associated with a lower odds of high SAA group membership. CONCLUSION: Graduating students' self-assessed Core EPAs skills were higher for those intending general surgery than for those intending some other specialties. Our findings can inform collaborative efforts to ensure graduates' acquisition of the skills expected of them at the start of residency.

Anthropomorphic Character Animations Versus Digital Chalk Talks in a Resident Diabetes Pharmacotherapy Curriculum: a Randomized Controlled Trial.

BACKGROUND: Animation in medical education has boomed over the past two decades, and demand for distance learning technologies will likely continue in the context of the COVID-19 pandemic. However, experimental data guiding best practices for animation in medical education are scarce. OBJECTIVE: To compare the efficacy of two animated video styles in a diabetes pharmacotherapy curriculum for internal medicine residents. DESIGN: Learners were randomized to receive one of two versions of the same multimodal didactic curriculum. They received identical lectures, group activities, and quizzes, but were randomized to either digital chalk talk (DCT) videos or Sugar-Coated Science (SCS). SCS is an animated series using anthropomorphic characters, stories, and mnemonics to communicate knowledge. PARTICIPANTS: Ninety-two internal medicine residents at a single academic medical center received the curriculum within ambulatory medicine didactics. MAIN MEASURES: Knowledge was measured at multiple time points, as was residents' self-reported comfort using each medication class covered. Surveys assessed video acceptability and telepresence. Key themes were identified from open-ended feedback. KEY RESULTS: Baseline knowledge was low, consistent with prior needs assessments. On immediate posttest, mean scores were higher with SCS than DCT (74.8% versus 68.4%), but the difference was not statistically significant, p = 0.10. Subgroup analyses revealed increased knowledge in the SCS group for specific medication classes. Delayed posttest showed significant knowledge gains averaging 17.6% across all participants (p < 0.05); these gains were similar between animation types. SCS achieved significantly higher telepresence, entertainment, and acceptability scores than DCT. Qualitative data suggested that residents prioritize well-designed, multimodal curricula over specific animation characteristics. CONCLUSION: SCS and DCTs both led to learning within a multimodal curriculum, but SCS significantly enhanced learner experience. Animation techniques exemplified by both SCS and DCTs have roles in the medical educator toolkit. Selection between them should incorporate context, learner factors, and production resources.

Structure and impact of longitudinal Graduate Medical Education curricula designed to prepare future clinician-educators: A systematic scoping review: BEME Guide No. 74.

BACKGROUND: Many Graduate Medical Education (GME) programs offer clinician-educator curricula. The specific instructional methods employed and current best practices for clinician-educator curricula are unknown. We aimed to characterize the structure, curriculum content, instructional methods, and outcomes of longitudinal GME clinician-educator curricula. METHODS: We conducted a scoping review, registered with BEME, by comprehensively searching health science databases and related grey literature from January 2008 to January 2021 for studies involving longitudinal GME curricula aimed to train future clinician-educators. RESULTS: From 9437 articles, 36 unique curricula were included in our review. Most curricula were designed for residents (n = 26) but were heterogeneous in structure, instructional methods, and content. Several curricular themes emerged, including: 1) duration ≥ 12 months, 2) application of theory-based didactics with experiential activities, 3) independent projects, 4) exposure to faculty mentorship and educator communities, 5) strengthening competencies beyond teaching and scholarship, and 6) protected time and funding. Most outcomes were positive and focused on learner satisfaction or behavior change related to scholarly output and career tracking. CONCLUSIONS: Curricula in our review included important skills including experiential teaching, scholarly projects, and exposure to educator communities. Future curricula should build on these competencies and include more assessment of learner and program outcomes.

WBAs in UME-How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship.

BACKGROUND: Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces. DESIGN: A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold. KEY RESULTS: A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity. CONCLUSIONS: Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.

Thriving among Primary Care Physicians: a Qualitative Study.

BACKGROUND: Burnout is high in primary care physicians and negatively impacts the quality of patient care. While many studies have evaluated burnout, there have been few which investigate those physicians who are satisfied with their careers and life-a phenomenon we term "thriving." OBJECTIVE: To identify factors contributing to both career and life satisfaction through qualitative interviews. PARTICIPANTS: The subjects were primary care physicians. APPROACH: Qualitative interviews were performed between July 2018 and March 2020. Physicians were identified by snowball sampling and were asked to complete validated instruments to identify job/life satisfaction and lack of burnout. Semi-structured interviews were conducted, focused on aspects of participants' career and life which contributed to their thriving, including work environment, social networks, family life, institutional support, coping strategies, and extracurricular activities. Transcripts were analyzed using thematic content analysis using a grounded theory approach. MAIN MEASURES: Personal, professional, and life factors that contributed to achieving career and life satisfaction in primary care physicians and potential solutions for burnout. RESULTS: Thirty-two physicians were interviewed (9.4% family physicians, 9.4% combined internists-pediatricians, 40.6% internists, and 40.6% pediatricians) with a mean age 54.7 years and 23.8 years in practice. No physicians included met the criteria for burnout. All met the criteria for career and life satisfaction. Five themes were identified as critical to thriving: an intrinsic love for the work, a rich social network, a fulfilling doctor-patient relationship, a value-oriented belief system, and agency in the work environment. CONCLUSIONS: Several factors contribute to professional fulfillment and life satisfaction among primary care physicians, which we propose as a model for physicians thriving. Some factors were intrinsic, such as having value-oriented beliefs and inherent love for medicine, while others were extrinsic, such as having a fulfilling social network. Barriers and opportunities to apply these lessons for the wider physician community are discussed.

Making Sense of Trainee Performance: Entrustment Decision-Making in Internal Medicine Program Directors.

Background: Competency-based assessment is an important but challenging aspect of residency education but determines trainees' progression towards the ultimate goal of graduation. Entrustment decision making has been proposed as a supplementary metric to assess trainee competence. This study explores the process by which Program Directors (PDs) make entrustment decisions in Internal Medicine (IM) training programs. Study Design: Purposive sampling was used to recruit PDs from ACGME-accredited IM training programs to participate in a semi-structured interview. We analyzed interviews using an iterative, grounded theory-based approach to allow identification of themes that define the process of trainee entrustment. Results: Sixteen PDs were interviewed. Qualitative analysis showed that PDs use a dynamic process to understand trainee entrustability and progression towards competence, including construction of assessment networks, comparing performance to expected trajectory of trainee competence development, and bidirectional filtering and weighing of assessment data. Conclusions: PDs serve as a central processor by which assessment data on trainees is filtered, weighted, and compared an expected trajectory, all to gain understanding of trainee performance. Assessment networks are crucial to understanding trainee competence. While expected trajectory is an important tool to determine how trainees are progressing, its continued use may inject bias into the assessment process and slow transition to true competency-based assessment.

Understanding Entrustment Decision-Making by Surgical Program Directors.

BACKGROUND: The process of entrustment-placing trust in a trainee to independently execute a task-has been proposed as a complementary metric to assess competence. However, entrustment decision-making by trainee supervisors is not well understood in surgical training. We aim to explore processes underlying entrustment decision-making (EDM) by general surgery program directors. MATERIALS AND METHODS: Purposive sampling was used to recruit 20 program directors from Accreditation Council for Graduate Medical Education-accredited general surgery training programs to participate in a one-hour semistructured interview. We analyzed interviews using an iterative and inductive approach to identify novel themes associated with the process of trainee entrustment. RESULTS: Qualitative analysis identified that program directors rely on a network of faculty to make entrustment decisions regarding trainees. Perceived trainee competence to perform independent clinical tasks varies significantly in and out of the operating room (OR), with a strong emphasis on entrustment for technical competencies to the exclusion of cognitive competencies. In the OR, entrustment is informed by an attending's reflexive trust and physical presence, trainee labels, and presumed discernment. Outside of the OR, trainee labels, presumed discernment, and transference of competence were identified as critical themes. CONCLUSIONS: Modifiable components of entrustment are equally dependent on trainee and faculty behavior. Entrustment is more heavily informed by trainee performance in the OR, despite program directors uniformly stating that judgment outside of the OR is the most critical component of resident training. The inclusion of EDM to evaluate trainee progression should be considered as an important adjunct to established Accreditation Council for Graduate Medical Education milestones.

A resident-led project to improve documentation of overweight and obesity in a primary care clinic.

Background: Although the prevalence of overweight and obesity (OW/OB) has increased in the last three decades, studies show that these conditions are sub-optimally documented by physicians. Health information technology tools have varying effects on improving documentation of OW/OB but often have to be complemented with other interventions to be effective. Objective: Upon identifying low rates of documentation of diagnoses of overweight and obesity by resident and attending physicians, despite the use of an electronic health record (EHR) with automated BMI calculations, we performed a quality improvement (QI) project to improve documentation of these diagnoses for patients in our community hospital primary care clinic. Methods: The EHR was reviewed to determine documentation rates by resident and attending physicians between 1 March 2018 and 31 September 2018. We collected pre-intervention data, developed interventions, and implemented tests of change using Plan-Do-Study-Act (PDSA) cycles to improve documentation of OW/OB. Results: Documentation of overweight and obesity diagnoses increased from a baseline of 46% to 79% over a 20-week period after initiation of our project. Conclusion: We demonstrate the successful implementation of resident-led, multi-faceted interventions in a team-based QI project to optimize documentation of OW/OB in the EHR.

Using an Entrustable Professional Activity to Assess Consultation Requests Called on an Internal Medicine Teaching Service.

Introduction: The Accreditation Council for Graduate Medical Education's milestones require internal medicine residents to have competency in calling consults. Based on a literature review, we developed an Entrustable Professional Activity (EPA) to delineate the knowledge, skills, and attitudes required for a consultation request and, building on the EPA, implemented an assessment instrument to provide feedback to interns calling consultation requests and assess the quality of their consult questions and the level of supervision required in performing this milestone. Methods: Assessments were done on internal medicine inpatient teaching services. Consultation requests were performed by interns and observed by residents using the assessment instrument. Feedback was provided to the interns. Interns then completed a self-reflection instrument based on the feedback. Results: Twenty-six paired observations were collected over three 1-month rotations. There was a moderate positive correlation (r = .43) comparing resident and intern responses to how they felt about the intern's ability to make a consultation request. There was a strong positive correlation (r = .65) comparing resident opinion of how strong the intern's ability in calling a consult to how well the consult question used the PICO (patient, intervention, comparators, outcomes of interest) framework. Twenty-five out of 28 interns (89%) said they would make a change during their next consultation request due to feedback from their resident. Discussion: Our EPA-based assessment instrument provided an opportunity to give interns feedback and to assess the quality of the consultation requests they made.