Developing an educational "hub": impact of a distance-learning curriculum in a multinational cohort.

PURPOSE: To address a gap in radiation oncology education in low- and middle-income countries (LMICs), we sought to evaluate the effectiveness and generalizability of a refined curriculum on intensity modulated radiotherapy (IMRT) offered to existing radiation therapy (RT) clinics across Africa and Latin America (LATAM) at no cost. METHODS: A curriculum was created based on prior needs assessments and adapted for participating medical physicists, radiation oncologists, radiation therapists, and trainees in LMICs. English-speaking and Spanish-speaking teams of volunteer educators delivered 27 hour-long sessions 1-2 times weekly for 4 months using video conferencing to African and LATAM cohorts, respectively. Pre- and post-course multiple-choice examinations were administered to LATAM participants, and pre- and post-course self-confidence (1-5 Likert-scale) and open-ended feedback were collected from all participants. RESULTS: Twenty-five centers across Africa (13) and LATAM (12) participated, yielding a total of 332 enrolled participants (128 African, 204 LATAM). Sessions were delivered with a mean of 44 (22.5) and 85 (25.4) participants in the African and LATAM programs, respectively. Paired pre and post-course data demonstrated significant (p < 0.001) improvement in knowledge from 47.9 to 89.6% and self-confidence across four domains including foundations (+ 1.1), commissioning (+ 1.3), contouring (+ 1.7), and treatment planning (+ 1.0). Attendance was a significant predictor of change in self-confidence in "high attendance" participants only, suggesting a threshold effect. Qualitative data demonstrates that participants look forward to applying their knowledge in the clinical setting. CONCLUSION: A specialized radiation oncology curriculum adapted for LMIC audiences was effective for both African and LATAM participants. Participant feedback suggests that the refined IMRT course empowered clinics with knowledge and confidence to help train others. This feasible "Hub and Spokes" approach in which a distance-learning course establishes a hub to be leveraged by spokes (learners) may be generalizable to others aiming to reduce global health care disparities through training efforts.

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School.

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

Master clinical teachers and personalised learning.

Clinical history taking and physical examination are two of the most important competencies of physicians. In addition to informing diagnoses, these activities build rapport and establish relationships between caregivers and patients. Despite this, emphasis on the assessment of bedside clinical skills is declining. To prepare our students for clinical work, we began a clinical competency, personalised teaching programme in which students perform a history and physical examination in front of a master clinical teacher (MCT) approximately every 2 weeks throughout their core clerkship year. The MCT works with the student in a clinical encounter, providing personalised bedside instruction on all features of being a clinician including bedside manner, history-taking skills, physical examination skills, and clinical reasoning. The MCT then provides an assessment of student's competency development and gives feedback to the student about what they do well and where they have opportunities for growth. Assessment data are collected and tracked longitudinally across the clerkship phase to ensure that each student is progressing developmentally. With over 6000 observations of student performance, we are able to discern competency development and growth over time. We can identify if a student is not improving as expected during their clerkship phase and intervene by providing extra practice and training. This core clerkship teaching programme has been well received by both students and instructors and has led us to pilot this approach during the post-clerkship phase of our medical training.

Educating for adaptive expertise: case examples along the medical education continuum.

Adaptive expertise represents the combination of both efficient problem-solving for clinical encounters with known solutions, as well as the ability to learn and innovate when faced with a novel challenge. Fostering adaptive expertise requires careful approaches to instructional design to emphasize deeper, more effortful learning. These teaching strategies are time-intensive, effortful, and challenging to implement in health professions education curricula. The authors are educators whose missions encompass the medical education continuum, from undergraduate through to organizational learning. Each has grappled with how to promote adaptive expertise development in their context. They describe themes drawn from educational experiences at these various learner levels to illustrate strategies that may be used to cultivate adaptive expertise.At Vanderbilt University School of Medicine, a restructuring of the medical school curriculum provided multiple opportunities to use specific curricular strategies to foster adaptive expertise development. The advantage for students in terms of future learning had to be rationalized against assessments that are more short-term in nature. In a consortium of emergency medicine residency programs, a diversity of instructional approaches was deployed to foster adaptive expertise within complex clinical learning environments. Here the value of adaptive expertise approaches must be balanced with the efficiency imperative in clinical care. At Mayo Clinic, an existing continuous professional development program was used to orient the entire organization towards an adaptive expertise mindset, with each individual making a contribution to the shift.The different contexts illustrate both the flexibility of the adaptive expertise conceptualization and the need to customize the educational approach to the developmental stage of the learner. In particular, an important benefit of teaching to adaptive expertise is the opportunity to influence individual professional identity formation to ensure that clinicians of the future value deeper, more effortful learning strategies throughout their careers.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020.

Importance: Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency. Objective: To assess progress in developing an entrustment process in the Core EPAs framework. Design, Setting, and Participants: In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database. Interventions: Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations. Main Outcomes and Measures: On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded. Results: Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs. Conclusions and Relevance: These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.

Measuring the Master Adaptive Learner: Development and Internal Structure Validity Evidence for a New Instrument.

BACKGROUND: The master adaptive learner (MAL) uses self-regulated learning skills to develop adaptive, efficient, and accurate skills in practice. Given rapid changes in healthcare, it is essential that medical students develop into MALs. There is a need for an instrument that can capture MAL behaviors and characteristics. The objective of this study was to develop an instrument for measuring the MAL process in medical students and evaluate its psychometric properties. METHODS: As part of curriculum evaluation, 818 students completed previously developed instruments with validity evidence including the Self-Regulated Learning Perception Scale, Brief Resilience Scale, Goal Orientation Scale, and Jefferson Scale of Physician Lifelong Learning. The authors performed exploratory factor analysis to examine underlying relationships between items. Items with high factor loadings were retained. Cronbach's alpha was computed. In parallel, the multi-institutional research team rated the same items to provide content validity evidence of the items to MAL model. RESULTS: The original 67 items were reduced to 28 items loading onto four factors: Planning, Learning, Resilience, and Motivation. Each subscale included the following number of items and Cronbach's alpha: Planning (10 items, alpha = 0.88), Learning (6 items, alpha = 0.81), Resilience (6 items, alpha = 0.89), and Motivation (6 items, alpha = 0.81). The findings from the factor analyses aligned with the research team ratings of linkage to the components of MAL. CONCLUSION: These findings serve as a starting point for future work measuring master adaptive learning to identify and support learners. To fully measure the MAL construct, additional items may need to be developed.

Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study.

PURPOSE: In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD: For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS: The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS: TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

Precision Cancer Medicine: Dynamic Learning of Cancer Biology in a Clinically Meaningful Context.

PURPOSE: Precision medicine is revolutionizing healthcare practices, most notably in oncology. With cancer being the second leading cause of death in the USA, it is important to integrate precision oncology content in undergraduate medical education. METHODS: In 2015, we launched a Clinical Cancer Medicine Integrated Science Course (ISC) for post-clerkship medical students at Vanderbilt University School of Medicine (VUSM). In this ISC, students learned cancer biology and clinical oncology concepts through a combination of classroom and patient care activities. Student feedback from mid- and end-of-course surveys and student match data were analyzed and used to develop ongoing course improvements. RESULTS: To date, 72 medical students have taken the Clinical Cancer Medicine ISC. Over 90% of students who completed end-of-course surveys agreed or strongly agreed that this course advanced their foundational science knowledge in clinical cancer medicine, that clinical relevance was provided during non-clinical foundational science learning activities, and that foundational science learning was embedded in course clinical experiences. Students who took this course most commonly matched in Internal Medicine, Pathology, Pediatrics, and Radiation Oncology. VUSM students who matched into Pathology and Radiation Oncology were more likely to take this ISC than students who matched in other specialties. CONCLUSION: The Clinical Cancer Medicine ISC serves as a model for incorporating precision oncology, cancer biology foundational science, and oncology patient care activities in undergraduate medical education. The course prepares students to care for oncology patients in their fields of interests during their future career in medicine.

Exploiting the power of information in medical education.

The explosion of medical information demands a thorough reconsideration of medical education, including what we teach and assess, how we educate, and whom we educate. Physicians of the future will need to be self-aware, self-directed, resource-effective team players who can synthesize and apply summarized information and communicate clearly. Training in metacognition, data science, informatics, and artificial intelligence is needed. Education programs must shift focus from content delivery to providing students explicit scaffolding for future learning, such as the Master Adaptive Learner model. Additionally, educators should leverage informatics to improve the process of education and foster individualized, precision education. Finally, attributes of the successful physician of the future should inform adjustments in recruitment and admissions processes. This paper explores how member schools of the American Medical Association Accelerating Change in Medical Education Consortium adjusted all aspects of educational programming in acknowledgment of the rapid expansion of information.

Revisiting the Core Entrustable Professional Activities for Entering Residency.

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

The educators' experience: Learning environments that support the master adaptive learner.

BACKGROUND: The Master Adaptive Learner (MAL) theoretical framework describes an integrated approach to learning that combines features of educational theory on self-regulated learning and aspects of quality improvement. In order to develop MAL students, it is important to pay attention to the learning environment. PURPOSE: To describe educators' perspectives about the learning environment needed to promote the development of master adaptive learners. METHODS: Thematic analysis of reports by medical educators who were workshop participants at a national presentation on creating effective learning environment to develop MAL in undergraduate medical education. RESULTS: Three themes educators considered important in the development of the Master Adaptive Learner were Adaptive Educator, Support for Learning, and Institutional Commitment. These findings suggest that in order to support the MAL, an educational setting should provide faculty who can be flexible and adapt to the developing MAL, learning experiences that support active learning, focused on groups as well as developing individual learners. Leaders in the educational setting should demonstrate a commitment to creating a culture to support learning and provide appropriate resources to that end. CONCLUSION: Learning environments to develop master adaptive learners need to have adaptive educators, teaching, learning, and institutional culture to support challenge and grow Master Adaptive Learners.

Embedding Ethics Education in Clinical Clerkships by Identifying Clinical Ethics Competencies: The Vanderbilt Experience.

The clinical clerkships in medical school are the first formal opportunity for trainees to apply bioethics concepts to clinical encounters. These clerkships are also typically trainees' first sustained exposure to the "reality" of working in clinical teams and the full force of the challenges and ethical tensions of clinical care. We have developed a specialized, embedded ethics curriculum for Vanderbilt University medical students during their second (clerkship) year to address the unique experience of trainees' first exposure to clinical care. Our embedded curriculum is centered around core "ethics competencies" specific to the clerkship: for Medicine, advanced planning and end-of-life discussions; for Surgery, informed consent; for Pediatrics, the patient-family-provider triad; for Obstetrics and Gynecology, women's autonomy, unborn child's interests, and partner's rights; and for Neurology/Psychiatry, decision-making capacity. In this paper, we present the rationale for these competencies, how we integrated them into the clerkships, and how we assessed these competencies. We also review the additional ethical issues that have been identified by rotating students in each clerkship and discuss our strategies for continued evolution of our ethics curriculum.

Developing an Entrustment Process: Insights from the AAMC CoreEPA Pilot.

One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.

Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales.

Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.

Too Many Hats? Conflicts of Interest in Learning Community Faculty Roles.

PURPOSE: Many US medical schools have adopted learning communities to provide a framework for advising and teaching functions. Faculty who participate in learning communities often have additional educator roles. Defining potential conflicts of interest (COIs) among these roles is an important consideration for schools with existing learning communities and those looking to develop them, both for transparency with students and also to comply with regulatory requirements. METHODS: A survey was sent to the institutional contact for each of the 42 Learning Communities Institute (LCI) member medical schools to assess faculty opinions about what roles potentially conflict. The survey asked the role of learning community faculty in summative and formative assessment of students and whether schools had existing policies around COIs in medical education. RESULTS: In all, 35 (85%) LCI representatives responded; 30 (86%) respondents agreed or strongly agreed that learning community faculty should be permitted to evaluate their students for formative purposes, while 19 (54%) strongly agreed or agreed that learning community faculty should be permitted to evaluate their students in a way that contributes to a grade; 31 (89%) reported awareness of the accreditation standard ensuring "that medical students can obtain academic counseling from individuals who have no role in making assessment or promotion decisions about them," but only 10 (29%) had a school policy about COIs in education. There was a wide range of responses about what roles potentially conflict with being a learning community faculty. CONCLUSION: The potential for COIs between learning community faculty and other educator roles concerns faculty at schools with learning communities, but most schools have not formally addressed these concerns.

Learning to balance efficiency and innovation for optimal adaptive expertise.

It is critical for health professionals to continue to learn and this must be supported by health professions education (HPE). Adaptive expert clinicians are not only expert in their work but have the additional capacity to learn and improve in their practices. The authors review a selective aspect of learning to become an adaptive expert: the capacity to optimally balance routine approaches that maximize efficiency with innovative ones where energy and resources are used to customize actions for novel or difficult situations. Optimal transfer of learning, and hence the design of instruction, differs depending on whether the goal is efficient or innovative practice. However, the task is necessarily further complicated when the aspiration is an adaptive expert practitioner who can fluidly balance innovation with efficiency as the situation requires. Using HPE examples at both the individual and organizational level, the authors explore the instructional implications of learning to shift from efficient to innovative expert functioning, and back. They argue that the efficiency-innovation tension is likely to endure deep into the future and therefore warrants important consideration in HPE.