Artificial intelligence education: An evidence-based medicine approach for consumers, translators, and developers.

Current and future healthcare professionals are generally not trained to cope with the proliferation of artificial intelligence (AI) technology in healthcare. To design a curriculum that caters to variable baseline knowledge and skills, clinicians may be conceptualized as "consumers", "translators", or "developers". The changes required of medical education because of AI innovation are linked to those brought about by evidence-based medicine (EBM). We outline a core curriculum for AI education of future consumers, translators, and developers, emphasizing the links between AI and EBM, with suggestions for how teaching may be integrated into existing curricula. We consider the key barriers to implementation of AI in the medical curriculum: time, resources, variable interest, and knowledge retention. By improving AI literacy rates and fostering a translator- and developer-enriched workforce, innovation may be accelerated for the benefit of patients and practitioners.

The Clinical Microsystems Clerkship at University of California, San Francisco: Integrating Clinical Skills and Health Systems Improvement for Early Medical Students.

PROBLEM: Medical educators recognize that partnering actively with health system leaders closes significant health care experience, quality, and outcomes gaps. Medical schools have explored innovations training physicians to care for both individual patients and populations while improving systems of care. Yet, early medical student education fails to include systems improvement as foundational skills. When health systems science is taught, it is often separated from core clinical skills. APPROACH: The Clinical Microsystems Clerkship at the University of California, San Francisco School of Medicine, launched in 2016, integrates clinical skills training with health systems improvement from the start of medical school. Guided by communities of practice and workplace learning principles, it embeds first-year and second-year students in longitudinal clinical microsystems with physician coaches and interprofessional clinicians one day per week. Students learn medical history, physical examination, patient communication, interprofessional teamwork, and health systems improvement. Assessments include standardized patient examinations and improvement project reports. Program outcome measures include student satisfaction and attitudes, clinical skills performance, and evidence of systems improvement learning, including dissemination and scholarship. OUTCOMES: Students reported high satisfaction (first-year, 4.10; second-year, 4.29, on a scale of 1-5) and value (4.14) in their development as physicians. Clinical skills assessment accuracy was high (70%-96%). Guided by interprofessional clinicians across 15 departments, students completed 258 improvement projects in 3 health systems (academic, safety net, Veterans Affairs). Sample projects reduced disparities in hypertension, improved opiate safety, and decreased readmissions. Graduating students reported both clinical skills and health systems knowledge as important to physician success, patient experience, and clinical outcomes (4.73). Most graduates discussed their projects in residency applications (85%) and disseminated related papers and presentations (54%). NEXT STEPS: Integrating systems improvement, interprofessional teamwork, and clinical skills training can redefine early medical student education. Health system perspectives, long-term outcomes, and sustainability merit further exploration.

"It's All About the Trust And Building A Foundation:" Evaluation of a Longitudinal Medical Student Coaching Program.

Coaching is increasingly implemented in medical education to support learners' growth, learning, and wellbeing. Data demonstrating the impact of longitudinal coaching programs are needed.We developed and evaluated a comprehensive longitudinal medical student coaching program designed to achieve three aims for students: fostering personal and professional development, advancing physician skills with a growth mindset, and promoting student wellbeing and belonging within an inclusive learning community. We also sought to advance coaches' development as faculty through satisfying education roles with structured training. Students meet with coaches weekly for the first 17 months of medical school for patient care and health systems skills learning, and at least twice yearly throughout the remainder of medical school for individual progress and planning meetings and small-group discussions about professional identity. Using the developmental evaluation framework, we iteratively evaluated the program over the first five years of implementation with multiple quantitative and qualitative measures of students' and coaches' experiences related to the three aims.The University of California, San Francisco, School of Medicine, developed a longitudinal coaching program in 2016 for medical students alongside reform of the four-year curriculum. The coaching program addressed unmet student needs for a longitudinal, non-evaluative relationship with a coach to support their development, shape their approach to learning, and promote belonging and community.In surveys and focus groups, students reported high satisfaction with coaching in measures of the three program aims. They appreciated coaches' availability and guidance for the range of academic, personal, career, and other questions they had throughout medical school. Students endorsed the value of a longitudinal relationship and coaches' ability to meet their changing needs over time. Students rated coaches' teaching of foundational clinical skills highly. Students observed coaches learning some clinical skills with them - skills outside a coach's daily practice. Students also raised some concerns about variability among coaches. Attention to wellbeing and belonging to a learning community were program highlights for students. Coaches benefited from relationships with students and other coaches and welcomed the professional development to equip them to support all student needs.Students perceive that a comprehensive medical student coaching program can achieve aims to promote their development and provide support. Within a non-evaluative longitudinal coach relationship, students build skills in driving their own learning and improvement. Coaches experience a satisfying yet challenging role. Ongoing faculty development within a coach community and funding for the role seem essential for coaches to fulfill their responsibilities.

We Have No Choice but to Transform: The Future of Medical Education After the COVID-19 Pandemic.

Medical education exists to prepare the physician workforce that our nation needs, but the COVID-19 pandemic threatened to disrupt that mission. Likewise, the national increase in awareness of social justice gaps in our country pointed out significant gaps in health care, medicine, and our medical education ecosystem. Crises in all industries often present leaders with no choice but to transform-or to fail. In this perspective, the authors suggest that medical education is at such an inflection point and propose a transformational vision of the medical education ecosystem, followed by a 10-year, 10-point plan that focuses on building the workforce that will achieve that vision. Broad themes include adopting a national vision; enhancing medicine's role in social justice through broadened curricula and a focus on communities; establishing equity in learning and processes related to learning, including wellness in learners, as a baseline; and realizing the promise of competency-based, time-variable training. Ultimately, 2020 can be viewed as a strategic inflection point in medical education if those who lead and regulate it analyze and apply lessons learned from the pandemic and its associated syndemics.

Medical Education's Wicked Problem: Achieving Equity in Assessment for Medical Learners.

Despite a lack of intent to discriminate, physicians educated in U.S. medical schools and residency programs often take actions that systematically disadvantage minority patients. The approach to assessment of learner performance in medical education can similarly disadvantage minority learners. The adoption of holistic admissions strategies to increase the diversity of medical training programs has not been accompanied by increases in diversity in honor societies, selective residency programs, medical specialties, and medical school faculty. These observations prompt justified concerns about structural and interpersonal bias in assessment. This manuscript characterizes equity in assessment as a "wicked problem" with inherent conflicts, uncertainty, dynamic tensions, and susceptibility to contextual influences. The authors review the underlying individual and structural causes of inequity in assessment. Using an organizational model, they propose strategies to achieve equity in assessment and drive institutional and systemic improvement based on clearly articulated principles. This model addresses the culture, systems, and assessment tools necessary to achieve equitable results that reflect stated principles. Three components of equity in assessment that can be measured and evaluated to confirm success include intrinsic equity (selection and design of assessment tools), contextual equity (the learning environment in which assessment occurs), and instrumental equity (uses of assessment data for learner advancement and selection and program evaluation). A research agenda to address these challenges and controversies and demonstrate reduction in bias and discrimination in medical education is presented.

Designing Well-Being: Using Design Thinking to Engage Residents in Developing Well-Being Interventions.

PROBLEM: Improving well-being in residency requires solutions that focus on organizational factors and the individual needs of residents, yet there are few examples of successful strategies to address this challenge. Design thinking (DT), or human-centered design, is an approach to problem-solving that focuses on understanding emotions and human dynamics and may be ideally suited to tackling well-being as a complex problem. The authors taught residents to use DT techniques to identify, analyze, and address organizational well-being challenges. APPROACH: Internal medicine residents at the University of California, San Francisco completed an 8-month DT program in 2016-2017. The program consisted of four 2-hour workshops with small group project work between sessions. In each session, resident teams shared their progress and analyzed emerging themes to solve well-being problems. At the conclusion of the program, they summarized the final design principles and recommendations that emerged from their work and were interviewed about DT as a strategy for developing well-being interventions for residents. OUTCOMES: Eighteen residents worked in teams to design solutions to improve: community and connection, space for reflection, peer support, and availability of individualized wellness. The resulting recommendations led to new interventions to improve well-being through near-peer communities. Residents emphasized how DT enhanced their creative thinking and trust in the residency program. They reported that not having enough time to work on projects between sessions and losing momentum during their clinical rotations were their biggest challenges. NEXT STEPS: Residents found DT useful for completing needs assessments, piloting interventions, and outlining essential design principles to improve well-being in residency. DT's focus on human values may be particularly suited to developing well-being interventions to enhance institutional community and culture. One outcome-that DT promoted creativity and trust for participants-may have applications in other spheres of medical education.

Toward Creating Equity in Awards Received During Medical School: Strategic Changes at One Institution.

Membership in the Alpha Omega Alpha Honor Medical Society (AΩA) is a widely recognized achievement valued by residency selection committees and employers. Yet research has shown selection favors students from racial/ethnic groups not underrepresented in medicine (not-UIM). The authors describe efforts to create equity in AΩA selection at the University of California, San Francisco, School of Medicine, through implementation of a holistic selection process, starting with the class of 2017, and present outcomes.Informed by the definition of holistic review, medical school leaders applied strategic changes grounded in evidence on inclusion, mitigating bias, and increasing opportunity throughout the AΩA selection process. These addressed increasing selection committee diversity, revising selection criteria and training committee members to review applications using a new instrument, broadening student eligibility and inviting applications, reviewing blinded applications, and making final selection decisions based on review and discussion of a rank-ordered list of students that equally weighted academic achievement and professional contributions.The authors compared AΩA eligibility and selection outcomes for 3 classes (2014-2016) during clerkship metric-driven selection, which prioritized academic achievement, and 3 classes (2017-2019) during holistic selection. During clerkship metric-driven selection, not-UIM students were 4 times more likely than UIM students to be eligible for AΩA (P = .001) and 3 times more likely to be selected (P = .001). During holistic selection, not-UIM students were 2 times more likely than UIM students to be eligible for AΩA (P = .001); not-UIM and UIM students were similarly likely to be selected (odds ratio = .7, P = .12)This new holistic selection process created equity in representation of UIM students among students selected for AΩA. Centered on equity pedagogy, which advocates dismantling structures that create inequity, this holistic selection process has implications for creating equity in awards selection during medical education.

The Consequences of Structural Racism on MCAT Scores and Medical School Admissions: The Past Is Prologue.

Those in medical education have a responsibility to prepare a physician workforce that can serve increasingly diverse communities, encourage healthy changes in patients, and advocate for the social changes needed to advance the health of all. The authors of this Perspective discuss many of the likely causes of the observed differences in mean Medical College Admission Test (MCAT) scores between students from groups well represented in medicine and those from groups underrepresented in medicine. The lower mean MCAT scores of underrepresented groups can present challenges to diversifying the physician workforce if medical schools only admit those applicants with the highest MCAT scores. The authors review the psychometric literature, which showed no evidence of bias in the exam, and note that the differences in mean MCAT scores between racial and ethnic groups are similar to those in other measures of academic achievement and performance on high-stakes tests.The authors then describe the ways in which structural racism in the United States has contributed to differences in achievement for underrepresented students compared with well-represented students. These differences are not due to differences in aptitude but to differences in opportunities. The authors describe the widespread consequences of structural racism on economic success, educational opportunity, and bias in the educational environment. They close with 3 recommendations for medical schools that may mitigate the consequences of structural racism while maintaining academic standards and admitting students likely to succeed. Adopting these recommendations may help the medical profession build the diverse physician workforce needed to serve communities today.

Differences in Narrative Language in Evaluations of Medical Students by Gender and Under-represented Minority Status.

BACKGROUND: In varied educational settings, narrative evaluations have revealed systematic and deleterious differences in language describing women and those underrepresented in their fields. In medicine, limited qualitative studies show differences in narrative language by gender and under-represented minority (URM) status. OBJECTIVE: To identify and enumerate text descriptors in a database of medical student evaluations using natural language processing, and identify differences by gender and URM status in descriptions. DESIGN: An observational study of core clerkship evaluations of third-year medical students, including data on student gender, URM status, clerkship grade, and specialty. PARTICIPANTS: A total of 87,922 clerkship evaluations from core clinical rotations at two medical schools in different geographic areas. MAIN MEASURES: We employed natural language processing to identify differences in the text of evaluations for women compared to men and for URM compared to non-URM students. KEY RESULTS: We found that of the ten most common words, such as "energetic" and "dependable," none differed by gender or URM status. Of the 37 words that differed by gender, 62% represented personal attributes, such as "lovely" appearing more frequently in evaluations of women (p < 0.001), while 19% represented competency-related behaviors, such as "scientific" appearing more frequently in evaluations of men (p < 0.001). Of the 53 words that differed by URM status, 30% represented personal attributes, such as "pleasant" appearing more frequently in evaluations of URM students (p < 0.001), and 28% represented competency-related behaviors, such as "knowledgeable" appearing more frequently in evaluations of non-URM students (p < 0.001). CONCLUSIONS: Many words and phrases reflected students' personal attributes rather than competency-related behaviors, suggesting a gap in implementing competency-based evaluation of students. We observed a significant difference in narrative evaluations associated with gender and URM status, even among students receiving the same grade. This finding raises concern for implicit bias in narrative evaluation, consistent with prior studies, and suggests opportunities for improvement.

Core Clerkship Grading: The Illusion of Objectivity.

Core clerkship grading creates multiple challenges that produce high stress for medical students, interfere with learning, and create inequitable learning environments. Students and faculty alike succumb to the illusion of objectivity-that quantitative ratings converted to grades convey accurate measures of the complexity of clinical performance.Clerkship grading is the first high-stakes assessment within medical school and occurs just as students are newly immersed full-time in an environment in which patient care supersedes their needs as learners. Students earning high marks situate themselves to earn entry into competitive residency programs and selective specialties. However, there is no commonly accepted standard for how to assign clerkship grades, and the process is vulnerable to imprecision and bias. Rewarding learners for the speed with which they adapt inherently favors students who bring advantages acquired before medical school and discounts the goal of all learners achieving competence.The authors propose that, rather than focusing on assigning core clerkship grades, assessment of student performance should incorporate expert judgment of learning progress. Competency-based medical education is predicated on the articulation of stepwise expectations for learners, with the support and time allocated for each learner to meet those expectations. Concurrently, students should ideally review their own performance data with coaches to self-assess areas of relative strength and areas for further growth. Eliminating grades in favor of competency-based assessment for learning holds promise to engage learners in developing essential patient care and teamwork skills and to foster their development of lifelong learning habits.

How Small Differences in Assessed Clinical Performance Amplify to Large Differences in Grades and Awards: A Cascade With Serious Consequences for Students Underrepresented in Medicine.

While students entering medical schools are becoming more diverse, trainees in residency programs in competitive specialties and academic medicine faculty have not increased in diversity. As part of an educational continuous quality improvement process at the University of California, San Francisco, School of Medicine, the authors examined data for the classes of 2013-2016 to determine whether differences existed between underrepresented in medicine (UIM) and not-UIM students' clinical performance (clerkship director ratings and number of clerkship honors grades awarded) and honor society membership-all of which influence residency selection and academic career choices.This analysis demonstrated differences that consistently favored not-UIM students. Whereas the size and magnitude of differences in clerkship director ratings were small, UIM students received approximately half as many honors grades as not-UIM students and were three times less likely to be selected for honor society membership.The authors use these findings to illustrate the amplification cascade, a phenomenon in which small differences in assessed performance lead to larger differences in grades and selection for awards. The amplification cascade raises concerns about opportunities for UIM students to compete successfully for competitive residency programs and potentially enter academic careers. Using a fishbone diagram, a continuous quality improvement root cause analysis tool, the authors contextualize their institutional results. They describe potential causes of group differences, drawing from the education disparities literature, and propose interventions and future research. They also share countermeasures adopted at their institution and encourage other medical schools to consider similar exploration of their institutional data.

Competency-Based, Time-Variable Education in the Health Professions: Crossroads.

Health care systems around the world are transforming to align with the needs of 21st-century patients and populations. Transformation must also occur in the educational systems that prepare the health professionals who deliver care, advance discovery, and educate the next generation of physicians in these evolving systems. Competency-based, time-variable education, a comprehensive educational strategy guided by the roles and responsibilities that health professionals must assume to meet the needs of contemporary patients and communities, has the potential to catalyze optimization of educational and health care delivery systems. By designing educational and assessment programs that require learners to meet specific competencies before transitioning between the stages of formal education and into practice, this framework assures the public that every physician is capable of providing high-quality care. By engaging learners as partners in assessment, competency-based, time-variable education prepares graduates for careers as lifelong learners. While the medical education community has embraced the notion of competencies as a guiding framework for educational institutions, the structure and conduct of formal educational programs remain more aligned with a time-based, competency-variable paradigm.The authors outline the rationale behind this recommended shift to a competency-based, time-variable education system. They then introduce the other articles included in this supplement to Academic Medicine, which summarize the history of, theories behind, examples demonstrating, and challenges associated with competency-based, time-variable education in the health professions.