Using Natural Language Processing to Visualize Narrative Feedback in a Medical Student Performance Dashboard.

PROBLEM: Clinical competency committees rely on narrative feedback for important insight into learner performance, but reviewing comments can be time-consuming. Techniques such as natural language processing (NLP) could create efficiencies in narrative feedback review. In this study, the authors explored whether using NLP to create a visual dashboard of narrative feedback to preclerkship medical students would improve the competency review efficiency. APPROACH: Preclerkship competency review data collected at the Northwestern University Feinberg School of Medicine from 2014 to 2021 were used to identify relevant features of narrative data associated with review outcome (ready or not ready) and draft visual summary reports of the findings. A user needs analysis was held with experienced reviewers to better understand work processes in December 2019. Dashboards were designed based on this input to help reviewers efficiently navigate large amounts of narrative data. The dashboards displayed the model's prediction of the review outcome along with visualizations of how narratives in a student's portfolio compared with previous students' narratives. Excerpts of the most relevant comments were also provided. Six faculty reviewers who comprised the competency committee in spring 2023 were surveyed on the dashboard's utility. OUTCOMES: Reviewers found the predictive component of the dashboard most useful. Only 1 of 6 reviewers (17%) agreed that the dashboard improved process efficiency. However, 3 (50%) thought the visuals made them more confident in decisions about competence, and 3 (50%) thought they would use the visual summaries for future reviews. The outcomes highlight limitations of visualizing and summarizing narrative feedback in a comprehensive assessment system. NEXT STEPS: Future work will explore how to optimize the dashboards to meet reviewer needs. Ongoing advancements in large language models may facilitate these efforts. Opportunities to collaborate with other institutions to apply the model to an external context will also be sought.

Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship Performance: Lessons Learned.

Background: Natural language processing is a promising technique that can be used to create efficiencies in the review of narrative feedback to learners. The Feinberg School of Medicine has implemented formal review of pre-clerkship narrative feedback since 2014 through its portfolio assessment system but this process requires considerable time and effort. This article describes how natural language processing was used to build a predictive model of pre-clerkship student performance that can be utilized to assist competency committee reviews. Approach: The authors took an iterative and inductive approach to the analysis, which allowed them to identify characteristics of narrative feedback that are both predictive of performance and useful to faculty reviewers. Words and phrases were manually grouped into topics that represented concepts illustrating student performance. Topics were reviewed by experienced reviewers, tested for consistency across time, and checked to ensure they did not demonstrate bias. Outcomes: Sixteen topic groups of words and phrases were found to be predictive of performance. The best-fitting model used a combination of topic groups, word counts, and categorical ratings. The model had an AUC value of 0.92 on the training data and 0.88 on the test data. Reflection: A thoughtful, careful approach to using natural language processing was essential. Given the idiosyncrasies of narrative feedback in medical education, standard natural language processing packages were not adequate for predicting student outcomes. Rather, employing qualitative techniques including repeated member checking and iterative revision resulted in a useful and salient predictive model.

We Have No Choice but to Transform: The Future of Medical Education After the COVID-19 Pandemic.

Medical education exists to prepare the physician workforce that our nation needs, but the COVID-19 pandemic threatened to disrupt that mission. Likewise, the national increase in awareness of social justice gaps in our country pointed out significant gaps in health care, medicine, and our medical education ecosystem. Crises in all industries often present leaders with no choice but to transform-or to fail. In this perspective, the authors suggest that medical education is at such an inflection point and propose a transformational vision of the medical education ecosystem, followed by a 10-year, 10-point plan that focuses on building the workforce that will achieve that vision. Broad themes include adopting a national vision; enhancing medicine's role in social justice through broadened curricula and a focus on communities; establishing equity in learning and processes related to learning, including wellness in learners, as a baseline; and realizing the promise of competency-based, time-variable training. Ultimately, 2020 can be viewed as a strategic inflection point in medical education if those who lead and regulate it analyze and apply lessons learned from the pandemic and its associated syndemics.

Establishing Trust When Assessing Learners: Barriers and Opportunities.

Trust plays a critical role in the assessment of learners in the clinical setting. In an ideal system, learners can be vulnerable and share their limitations and areas for improvement, while faculty possess the time and skill to provide specific feedback that enables learners to achieve competency in clinical skills. For medical students, a number of threats to the establishment of trust in the learning environment exist, including the interplay between feedback and grades, the existence of bias, and competing demands for faculty time. However, several strategies can help institutions to overcome these threats and foster a culture of trust related to assessment and assessment systems: Provide ungraded environments where learners are able to be vulnerable, cocreate assessments and assessment systems with faculty and learners, acknowledge and address bias, and provide faculty with adequate time and resources to employ best practices in assessment. By intentionally employing these strategies, our institutions can support trust in assessment systems and further learner growth and achievement.

Incorporating Physician Input Into a Maintenance of Certification Examination: A Content Validity Tool.

PURPOSE: As part of the American Board of Internal Medicine's (ABIM's) continuing effort to update its Maintenance of Certification (MOC) program, a content validity tool was used to conduct structured reviews of MOC exam blueprints (i.e., test specification tables) by the physician community. Results from the Cardiovascular Disease MOC blueprint review are presented to illustrate the process ABIM conducted for several internal medicine disciplines. METHOD: Ratings of topic frequency and importance were collected from cardiologists in 2016 using a three-point scale (low, medium, high). The web-based survey instrument presented 188 blueprint topic descriptions, each combined with five patient-related tasks (e.g., diagnosis, treatment). Descriptive statistics and chi-square analysis were employed. RESULTS: Responses from 441 review participants were analyzed. Frequency and importance ratings were aggregated as a composite statistic representing clinical relevance, and exam assembly criteria were modified to select questions, or items, addressing clinically relevant content only. Specifically, ≥ 88% of exam items now address high-importance topics, including ≤ 15% on topics that are also low frequency; and ≤ 12% of exam items now address medium-importance topics, including ≤ 3% on topics that are also low frequency. The updated blueprint has been published for test takers and provides enhanced information on content that would and would not be tested in subsequent examinations. It is linked to more detailed feedback that examinees receive on items answered incorrectly. CONCLUSIONS: The blueprint review garnered valuable feedback from the physician community and provided new evidence for the content validity of the Cardiovascular Disease MOC exam.

A Qualitative Analysis of Narrative Preclerkship Assessment Data to Evaluate Teamwork Skills.

Construct: Students entering the health professions require competency in teamwork. Background: Although many teamwork curricula and assessments exist, studies have not demonstrated robust longitudinal assessment of preclerkship students' teamwork skills and attitudes. Assessment portfolios may serve to fill this gap, but it is unknown how narrative comments within portfolios describe student teamwork behaviors. Approach: We performed a qualitative analysis of narrative data in 15 assessment portfolios. Student portfolios were randomly selected from 3 groups stratified by quantitative ratings of teamwork performance gathered from small-group and clinical preceptor assessment forms. Narrative data included peer and faculty feedback from these same forms. Data were coded for teamwork-related behaviors using a constant comparative approach combined with an identification of the valence of the coded statements as either "positive observation" or "suggestion for improvement." Results: Eight codes related to teamwork emerged: attitude and demeanor, information facilitation, leadership, preparation and dependability, professionalism, team orientation, values team member contributions, and nonspecific teamwork comments. The frequency of codes and valence varied across the 3 performance groups, with students in the low-performing group receiving more suggestions for improvement across all teamwork codes. Conclusions: Narrative data from assessment portfolios included specific descriptions of teamwork behavior, with important contributions provided by both faculty and peers. A variety of teamwork domains were represented. Such feedback as collected in an assessment portfolio can be used for longitudinal assessment of preclerkship student teamwork skills and attitudes.

Description and Early Outcomes of a Comprehensive Curriculum Redesign at the Northwestern University Feinberg School of Medicine.

In 2012, the Northwestern University Feinberg School of Medicine launched a redesigned curriculum addressing the four primary recommendations in the 2010 Carnegie Foundation for the Advancement of Teaching report on reforming medical education. This new curriculum provides a more standardized evaluation of students' competency achievement through a robust portfolio review process coupled with standard evaluations of medical knowledge and clinical skills. It individualizes learning processes through curriculum flexibility, enabling students to take electives earlier and complete clerkships in their preferred order. The new curriculum is integrated both horizontally and vertically, combining disciplines within organ-based modules and deliberately linking elements (science in medicine, clinical medicine, health and society, professional development) and threads (medical decision making, quality and safety, teamwork and leadership, lifestyle medicine, advocacy and equity) across the three phases that replaced the traditional four-year timeline. It encourages students to conduct research in an area of interest and commit to lifelong learning and self-improvement. The curriculum formalizes the process of professional identity formation and requires students to reflect on their experiences with the informal and hidden curricula, which strongly shape their identities.The authors describe the new curriculum structure, explain their approach to each Carnegie report recommendation, describe early outcomes and challenges, and propose areas for further work. Early data from the first cohort to progress through the curriculum show unchanged United States Medical Licensing Examination Step 1 and 2 scores, enhanced student research engagement and career exploration, and improved student confidence in the patient care and professional development domains.

What Is the Relationship Between a Preclerkship Portfolio Review and Later Performance in Clerkships?

PURPOSE: Medical educators struggle to find effective ways to assess essential competencies such as communication, professionalism, and teamwork. Portfolio-based assessment provides one method of addressing this problem by allowing faculty reviewers to judge performance, as based on a longitudinal record of student behavior. At the Feinberg School of Medicine, the portfolio system measures behavioral competence using multiple assessments collected over time. This study examines whether a preclerkship portfolio review is a valid method of identifying problematic student behavior affecting later performance in clerkships. METHOD: The authors divided students into two groups based on a summative preclerkship portfolio review in 2014: students who had concerning behavior in one or more competencies and students progressing satisfactorily. They compared how students in these groups later performed on two clerkship outcomes as of October 2015: final grades in required clerkships, and performance on a clerkship clinical composite score. They used Mann-Whitney tests and multiple linear regression to examine the relationship between portfolio review results and clerkship outcomes. They used USMLE Step 1 to control for knowledge acquisition. RESULTS: Students with concerning behavior preclerkship received significantly lower clerkship grades than students progressing satisfactorily (P = .002). They also scored significantly lower on the clinical composite score (P < .001). Regression analysis indicated concerning behavior was associated with lower clinical composite scores, even after controlling for knowledge acquisition. CONCLUSIONS: The results show a preclerkship portfolio review can identify behaviors that impact clerkship performance. A comprehensive portfolio system is a valid way to measure behavioral competencies.

Comparison of Content on the American Board of Internal Medicine Maintenance of Certification Examination With Conditions Seen in Practice by General Internists.

Importance: Success on the internal medicine (IM) examination is a central requirement of the American Board of Internal Medicine's (ABIM's) Maintenance of Certification program (MOC). Therefore, it is important to understand the degree to which this examination reflects conditions seen in practice, one dimension of content validity, which focuses on the match between content in the discipline and the topics on the examination questions. Objective: To assess whether the frequency of questions on IM-MOC examinations were concordant with the frequency of conditions seen in practice. Design, Setting, and Participants: The 2010-2013 IM-MOC examinations were used to calculate the percentage of questions for 186 medical condition categories from the examination blueprint, which balances examination content by considering importance and frequency of conditions seen in practice. Nationally representative estimates of conditions seen in practice by general internists were estimated from the primary diagnosis for 13 832 office visits (2010-2013 National Ambulatory Medical Care Surveys) and 108 472 hospital stays (2010 National Hospital Discharge Survey). Exposures: Prevalence of conditions included on the IM-MOC examination questions. Main Outcomes and Measures: The outcome measure was the concordance between the percentages of IM-MOC examination questions and the percentages of conditions seen in practice during either office visits or hospital stays for each of 186 condition categories (eg, diabetes mellitus, ischemic heart disease, liver disease). The concordance thresholds were 0.5 SD of the weighted mean percentages of the applicable 186 conditions seen in practice (0.74% for office visits; 0.51% for hospital stays). If the absolute differences between the percentages of examination questions and the percentages of office visit conditions or hospital stay conditions seen were less than the applicable concordance threshold, then the condition category was judged to be concordant. Results: During the 2010-2013 IM-MOC examination periods, 3600 questions (180 questions per examination form) were administered and 3461 questions (96.1%) were mapped into the 186 study conditions (mean, 18.6 questions per condition). Comparison of the percentages of 186 categories of medical conditions seen in 13 832 office visits and 108 472 hospital stays with the percentages of 3461 questions on IM-MOC examinations revealed that 2389 examination questions (69.0%; 95% CI, 67.5%-70.6% involving 158 conditions) were categorized as concordant. For concordance between questions and office visits only, 2010 questions (58.08%; 95% CI, 56.43%-59.72% of all examination questions) involving 145 conditions were categorized as concordant. For concordance between questions and hospital stays only, 1456 questions (42.07%; 95% CI, 40.42%-43.71% of all examination questions) involving 122 conditions were categorized as concordant. Conclusions and Relevance: Among questions on IM-MOC examinations from 2010-2013, 69% were concordant with conditions seen in general internal medicine practices, although some areas of discordance were identified.

Feasibility and Outcomes of Implementing a Portfolio Assessment System Alongside a Traditional Grading System.

PURPOSE: Portfolios are a powerful tool to collect and evaluate evidence of medical students' competence across time. However, comprehensive portfolio assessment systems that are implemented alongside traditional graded curricula at medical schools in the United States have not been described in the literature. This study describes the development and implementation of a longitudinal competency-based electronic portfolio system alongside a graded curriculum at a relatively large U.S. medical school. METHOD: In 2009, the authors developed a portfolio system that served as a repository for all student assessments organized by competency domain. Five competencies were selected for a preclerkship summative portfolio review. Students submitted reflections on their performance. In 2014, four clinical faculty members participated in standard-setting activities and used expert judgment and holistic review to rate students' competency achievement as "progressing toward competence," "progressing toward competence with some concern," or "progressing toward competence pending remediation." Follow-up surveys measured students' and faculty members' perceptions of the process. RESULTS: Faculty evaluated 156 portfolios and showed high levels of agreement in their ratings. The majority of students achieved the "progressing toward competence" benchmark in all competency areas. However, 31 students received at least one concerning rating, which was not reflected in their course grades. Students' perceptions of the system's ability to foster self-assessment were mixed. CONCLUSIONS: The portfolio review process allowed faculty to identify students with a concerning rating in a behavioral competency who would not have been identified in a traditional grading system. Identification of these students allows for intervention and early remediation.

Academic Performance of Students in an Accelerated Baccalaureate/MD Program: Implications for Alternative Physician Education Pathways.

PURPOSE: Over one-third of U.S. medical schools offer combined baccalaureate/MD (BA/MD) degree programs. A subset of these truncate the premedical phase, reducing total time to the MD degree. Data comparing educational outcomes of these programs with those of conventional pathways are limited. METHOD: The authors reviewed demographic characteristics and medical school performance of all 2,583 students entering Northwestern University Feinberg School of Medicine from 1999 to 2013, comparing students in the Honors Program in Medical Education (HPME), an accelerated seven-year program, versus non-HPME medical students. They evaluated Alpha Omega Alpha (AOA) selection, quintile performance distribution from the Medical Student Performance Evaluation, United States Medical Licensing Examination (USMLE) scores, and Match outcomes. RESULTS: A total of 560 students (21.7%) entered through the HPME. HPME students were on average 2.2 years younger and less likely (15/537 [2.8%] versus 285/1,833 [15.5%]) to belong to a racial/ethnic group underrepresented in medicine. There were no significant differences in AOA selection, quintile performance distribution, or USMLE scores. More HPME students entered internal medicine (161/450 [35.8%] versus 261/1,265 [20.6%]), and fewer chose emergency medicine (25/450 [5.6%] versus 110/1,265 [8.7%]) and obstetrics-gynecology (9/450 [2.0%] versus 67/1,265 [5.3%]). CONCLUSIONS: The academic performances of medical students in the two programs studied were equivalent. Accelerated BA/MD programs might play a role in ameliorating the length and cost of a medical education. The academic success of these students absent the usual emphasis on undergraduate GPA and Medical College Admission Test scores supports efforts to redefine medical student selection criteria.

Status of portfolios in undergraduate medical education in the LCME accredited US medical school.

AIM: We sought to investigate the number of US medical schools utilizing portfolios, the format of portfolios, information technology (IT) innovations, purpose of portfolios and their ability to engage faculty and students. METHODS: A 21-question survey regarding portfolios was sent to the 141 LCME-accredited, US medical schools. The response rate was 50% (71/141); 47% of respondents (33/71) reported that their medical school used portfolios in some form. Of those, 7% reported the use of paper-based portfolios and 76% use electronic portfolios. Forty-five percent reported portfolio use for formative evaluation only; 48% for both formative and summative evaluation, and 3% for summative evaluation alone. RESULTS: Seventy-two percent developed a longitudinal, competency-based portfolio. The most common feature of portfolios was reflective writing (79%). Seventy-three percent allow access to the portfolio off-campus, 58% allow usage of tablets and mobile devices, and 9% involve social media within the portfolio. Eighty percent and 69% agreed that the portfolio engaged students and faculty, respectively. Ninety-seven percent reported that the portfolios used at their institution have room for improvement. CONCLUSION: While there is significant variation in the purpose and structure of portfolios in the medical schools surveyed, most schools using portfolios reported a high level of engagement with students and faculty.

The scope and variety of combined baccalaureate-MD programs in the United States.

The landscape of combined baccalaureate-MD programs has changed substantially in the last two decades but has not been documented in detail. The authors review the current state of these programs and discuss opportunities for future study of their evolving role and potential impact.In 2011, using a definition of baccalaureate-MD program built on prior research, the authors reviewed Association of American Medical Colleges sources and medical school Web sites to identify and characterize 81 active programs. In addition, they surveyed the 57 medical schools offering those programs; 31 schools with 39 programs responded. The resulting database inventories the number and distribution of programs; institutional affiliations; missions or goals; length; size; admissions criteria; curricula; and retention requirements.Since the inception of combined programs in 1961, their number and curricular length have increased. Pressures that spurred earlier programs remain evident in the goals of today's programs: attract talented high school or early college students, especially from diverse backgrounds; prepare physicians to meet societal needs; and offer an enriched premedical environment. Baccalaureate educational activities achieve program goals through special courses, medical experiences, community service, and learning communities tailored to students' needs. Admission and retention criteria are comparable to those of traditional medical schools.Combined baccalaureate-MD programs have evolved along several paths during the last half century and have enriched the baccalaureate experiences of medical students. Shifting expectations for the selection and education of future physicians warrant focused research on these programs to document their effectiveness in addressing those expectations.