Evaluating the Role of Competency-Based Behavioral Interviewing in Holistic Medical School Admissions.

PROBLEM: Holistic review is a multifaceted concept that aims to increase diversity and applicant fit with program needs by complementing traditional academic requirements with appraisal of a wider range of personal characteristics and experiences. Behavioral interviewing has been practiced and studied in human resources, business, and organizational psychology for over 50 years. Its premise is that future performance can be anticipated from past actions. However, many of the interview approaches within the holistic framework are resource intensive and logistically challenging. APPROACH: The Vanderbilt University School of Medicine instituted a competency-based behavioral interview (CBBI) to augment the selection process in 2012. Behavioral interviews are based on key competencies needed for entering students and require applicants to reflect on their actual experiences and what they learned from them. The authors reviewed 5 years of experience (2015-2019) to evaluate how CBBI scores contributed to the overall assessment of applicants for admission. OUTCOMES: The final admission committee decision for each applicant was determined by reviewing multiple factors, with no single assessment determining the final score. The CBBI and summary interview scores showed a strong association (P < .005), suggesting that the summary interviewer, who had access to the full applicant file, and the CBBI interviewer, who did not, assessed similar strengths despite the 2 different approaches, or that the strengths assessed tracked in the same direction. Students whose 2 interview scores were not aligned were less likely to be accepted to the school. NEXT STEPS: The review raised awareness about the cultural aspects of interpreting the competencies and the need to expand our cultural framework throughout interviewer training. Findings indicate that CBBIs have the potential to reduce bias related to over-reliance on standardized metrics; however, additional innovation and research are needed.

Clinical use of artificial intelligence requires AI-capable organizations.

Artificial intelligence-based algorithms are being widely implemented in health care, even as evidence is emerging of bias in their design, problems with implementation, and potential harm to patients. To achieve the promise of using of AI-based tools to improve health, healthcare organizations will need to be AI-capable, with internal and external systems functioning in tandem to ensure the safe, ethical, and effective use of AI-based tools. Ideas are starting to emerge about the organizational routines, competencies, resources, and infrastructures that will be required for safe and effective deployment of AI in health care, but there has been little empirical research. Infrastructures that provide legal and regulatory guidance for managers, clinician competencies for the safe and effective use of AI-based tools, and learner-centric resources such as clear AI documentation and local health ecosystem impact reviews can help drive continuous improvement.

Competencies for the Use of Artificial Intelligence-Based Tools by Health Care Professionals.

PURPOSE: The expanded use of clinical tools that incorporate artificial intelligence (AI) methods has generated calls for specific competencies for effective and ethical use. This qualitative study used expert interviews to define AI-related clinical competencies for health care professionals. METHOD: In 2021, a multidisciplinary team interviewed 15 experts in the use of AI-based tools in health care settings about the clinical competencies health care professionals need to work effectively with such tools. Transcripts of the semistructured interviews were coded and thematically analyzed. Draft competency statements were developed and provided to the experts for feedback. The competencies were finalized using a consensus process across the research team. RESULTS: Six competency domain statements and 25 subcompetencies were formulated from the thematic analysis. The competency domain statements are: (1) basic knowledge of AI: explain what AI is and describe its health care applications; (2) social and ethical implications of AI: explain how social, economic, and political systems influence AI-based tools and how these relationships impact justice, equity, and ethics; (3) AI-enhanced clinical encounters: carry out AI-enhanced clinical encounters that integrate diverse sources of information in creating patient-centered care plans; (4) evidence-based evaluation of AI-based tools: evaluate the quality, accuracy, safety, contextual appropriateness, and biases of AI-based tools and their underlying data sets in providing care to patients and populations; (5) workflow analysis for AI-based tools: analyze and adapt to changes in teams, roles, responsibilities, and workflows resulting from implementation of AI-based tools; and (6) practice-based learning and improvement regarding AI-based tools: participate in continuing professional development and practice-based improvement activities related to use of AI tools in health care. CONCLUSIONS: The 6 clinical competencies identified can be used to guide future teaching and learning programs to maximize the potential benefits of AI-based tools and diminish potential harms.

Considering Clinician Competencies for the Implementation of Artificial Intelligence-Based Tools in Health Care: Findings From a Scoping Review.

BACKGROUND: The use of artificial intelligence (AI)-based tools in the care of individual patients and patient populations is rapidly expanding. OBJECTIVE: The aim of this paper is to systematically identify research on provider competencies needed for the use of AI in clinical settings. METHODS: A scoping review was conducted to identify articles published between January 1, 2009, and May 1, 2020, from MEDLINE, CINAHL, and the Cochrane Library databases, using search queries for terms related to health care professionals (eg, medical, nursing, and pharmacy) and their professional development in all phases of clinical education, AI-based tools in all settings of clinical practice, and professional education domains of competencies and performance. Limits were provided for English language, studies on humans with abstracts, and settings in the United States. RESULTS: The searches identified 3476 records, of which 4 met the inclusion criteria. These studies described the use of AI in clinical practice and measured at least one aspect of clinician competence. While many studies measured the performance of the AI-based tool, only 4 measured clinician performance in terms of the knowledge, skills, or attitudes needed to understand and effectively use the new tools being tested. These 4 articles primarily focused on the ability of AI to enhance patient care and clinical decision-making by improving information flow and display, specifically for physicians. CONCLUSIONS: While many research studies were identified that investigate the potential effectiveness of using AI technologies in health care, very few address specific competencies that are needed by clinicians to use them effectively. This highlights a critical gap.

Effect of Smartphone App-Based Education on Clinician Prescribing Habits in a Learning Health Care System: A Randomized Cluster Crossover Trial.

Importance: Effective methods for engaging clinicians in continuing education for learning-based practice improvement remain unknown. Objective: To determine whether a smartphone-based app using spaced education with retrieval practice is an effective method to increase evidence-based practice. Design, Setting, and Participants: A prospective, unblinded, single-center, crossover randomized clinical trial was conducted at a single academic medical center from January 6 to April 24, 2020. Vanderbilt University Medical Center clinicians prescribing intravenous fluids were invited to participate in this study. Interventions: All clinicians received two 4-week education modules: 1 on prescribing intravenous fluids and 1 on prescribing opioid and nonopioid medications (counterbalancing measure), over a 12-week period. The order of delivery was randomized 1:1 such that 1 group received the fluid management module first, followed by the pain management module after a 4-week break, and the other group received the pain management module first, followed by the fluid management module after a 4-week break. Main Outcomes and Measures: The primary outcome was evidence-based clinician prescribing behavior concerning intravenous fluids in the inpatient setting and pain medication prescribing on discharge from the hospital. Results: A total of 354 participants were enrolled and randomized, with 177 in group 1 (fluid then pain management education) and 177 in group 2 (pain management then fluid education). During the overall study period, 16 868 questions were sent to 349 learners, with 11 783 (70.0%) being opened: 10 885 (92.4%) of those opened were answered and 7175 (65.9%) of those answered were answered correctly. The differences between groups changed significantly over time, indicated by the significant interaction between educational intervention and time (P = .002). Briefly, at baseline evidence-concordant IV fluid ordered 7.2% less frequently in group 1 than group 2 (95% CI, -19.2% to 4.9%). This was reversed after training at 4% higher (95% CI, -8.2% to 16.0%) in group 1 than group 2, a more than doubling in the odds of evidence-concordant ordering (OR, 2.56, 95% CI, 0.80-8.21). Postintervention, all gains had been reversed with less frequent ordering in group 1 than group 2 (-9.5%, 95% CI, -21.6% to 2.7%). There was no measurable change in opioid prescribing behaviors at any time point. Conclusions and Relevance: In this randomized clinical trial, use of smartphone app learning modules resulted in statistically significant short-term improvement in some prescribing behaviors. However, this effect was not sustained over the long-term. Additional research is needed to understand how to sustain improvements in care delivery as a result of continuous professional development at the institutional level. Trial Registration: ClinicalTrials.gov Identifier: NCT03771482.

Comparison of Two Learning Modalities on Continuing Medical Education Consumption and Knowledge Acquisition: A Pilot Randomized Controlled Trial.

BACKGROUND: Research has demonstrated that active learning, spaced education, and retrieval-based practice can improve knowledge acquisition, knowledge retention, and clinical practice. Furthermore, learners prefer active learning modalities that use the testing effect and spaced education as compared to passive, lecture-based education. However, most research has been performed with students and residents rather than practicing physicians. To date, most continuing medical education (CME) opportunities use passive learning models, such as face-to-face meetings with lecture-style didactic sessions. The aim of this study was to investigate learner engagement, as measured by the number of CME credits earned, via two different learning modalities. METHODS: Diplomates of the American Board of Anesthesiology or candidates for certification through the board (referred to colloquially and for the remainder of this article as board certified or board eligible) were provided an opportunity to enroll in the study. Participants were recruited via email. Once enrolled, they were randomized into 1 of 2 groups: web-app-based CME (Webapp CME) or an online interface that replicated online CME (Online CME). The intervention period lasted 6 weeks and participants were provided educational content using one of the two approaches. As an incentive for participation, CME credits could be earned (without cost) during the intervention period and for completion of the postintervention quiz. The same number of CME credits was available to each group. RESULTS: Fifty-four participants enrolled and completed the study. The mean number of CME credits earned was greater in the Webapp group compared to the Online group (12.3 ± 1.4 h versus 4.5 ± 2.3 h, P < .001). Concerning knowledge acquisition, the difference in postintervention quiz scores was not statistically significant (Webapp 70% ± 7% versus Online 60% ± 11%, P = .11). However, only 29% of the Online group completed the postintervention quiz, versus 77% of the Webapp group (P < .001), possibly showing a greater rate of learner engagement in the Webapp group. CONCLUSION: In this prospective, randomized controlled pilot study, we demonstrated that daily spaced education delivered to learners through a smartphone web app resulted in greater learner engagement than an online modality. Further research with larger trials is needed to confirm our findings.

Quiz-Based Microlearning at Scale: a Rapid Educational Response to COVID-19.

Virtual quizzing is a viable model for continuing education at a large scale, particularly during the COVID-19 pandemic. By leveraging technology, microlearning encourages mobile education that is engaging, flexible, and accessible. Learners reported that this format was effective and preferable to traditional methods of education, suggesting further opportunity for innovation.

Embedding Learning in a Learning Health Care System to Improve Clinical Practice.

PROBLEM: In an ideal learning health care system (LHS), clinicians learn from what they do and do what they learn, closing the evidence-to-practice gap. In operationalizing an LHS, great strides have been made in knowledge generation. Yet, considerable challenges remain to the broad uptake of identified best practices. To bridge the gap from generating actionable knowledge to applying that knowledge in clinical practice, and ultimately to improving outcomes, new information must be disseminated to and implemented by frontline clinicians. To date, the dissemination of this knowledge through traditional avenues has not achieved meaningful practice change quickly. APPROACH: Vanderbilt University Medical Center (VUMC) developed QuizTime, a smartphone application learning platform, to provide a mechanism for embedding workplace-based clinician learning in the LHS. QuizTime leverages spaced education and retrieval-based practice to facilitate practice change. Beginning in January 2020, clinician-researchers and educators at VUMC designed a randomized, controlled trial to test whether the QuizTime learning system influenced clinician behavior in the context of recent evidence supporting the use of balanced crystalloids rather than saline for intravenous fluid management and new regulations around opioid prescribing. OUTCOMES: Whether spaced education and retrieval-based practice influence clinician behavior and patient outcomes at the VUMC system level will be tested using the data currently being collected. NEXT STEPS: These findings will inform future directions for developing and deploying learning approaches at scale in an LHS, with the goal of closing the evidence-to-practice gap.

The Vanderbilt Program in Interprofessional Learning: Sustaining a Longitudinal, Clinical Experience That Aligns Practice With Education.

PROBLEM: Designing and sustaining a longitudinal, clinic-based interprofessional learning experience is logistically challenging, which has limited the educational opportunities available in health professions schools. The authors discuss the Vanderbilt Program in Interprofessional Learning (VPIL), which addresses some of the challenges facing clinic-based interprofessional experiences. APPROACH: VPIL places first- and second-year students from 4 professional degree programs (medicine, nursing, pharmacy, social work) in Nashville, Tennessee, on teams where they work and learn together in authentic clinical environments over a 2-year period. The program was implemented in 2010 and includes 3 components: a summer immersion experience, seminar-based classroom and simulation sessions, and a weekly clinical experience. Students also complete a capstone quality improvement project. VPIL administrators have set up structures at the institutional, clinic, faculty, and student levels that have contributed to the sustainability of the program. OUTCOMES: Between 2010 and 2019, VPIL admitted 398 students who participated on 91 clinical teams. In addition, 55 clinical preceptors and 12 core faculty trained students for future collaborative practice. The program has received consistently high ratings from students, who have produced 69 quality improvement projects at their clinics. These projects have addressed aspects of the care delivery process and produced durable materials, showing that the program has contributed to important innovations in the health system. NEXT STEPS: VPIL faculty continue to improve the curriculum and administrative structures and work to expand the program to reach a wider variety of health professions students. Going forward, lessons from the program could assist educators in creating opportunities for students to learn interprofessionally and deliver high value health care in increasingly complex delivery systems.

Development of an interprofessional clinical learning environment report card.

Multiple factors in the learning environment can encourage or impede student learning. Unanswered questions regarding the shared learning environment for graduate nursing and medical education and the desire for an ongoing improvement process drove creation of an interprofessional collaborative and development of an Interprofessional Clinical Learning Environment Report Card (I-CLERC) at one U.S. academic medical center. The I-CLERC offers a process and a product for institutionalizing a shared assessment tool to inform improvement efforts, track progress and promote accountability. In addition, it enhances interprofessional collaboration, with students and faculty from both nursing and medicine working together to define excellence, monitor performance, and identify areas for improvement in the shared clinical learning environment. The purpose of this manuscript is to describe development and implementation of an interdisciplinary, institutional collaborative for ongoing evaluation of the shared clinical learning environment.

Copy-and-Paste in Medical Student Notes: Extent, Temporal Trends, and Relationship to Scholastic Performance.

BACKGROUND: Medical students may observe and subsequently perpetuate redundancy in clinical documentation, but the degree of redundancy in student notes and whether there is an association with scholastic performance are unknown. OBJECTIVES: This study sought to quantify redundancy, defined generally as the proportion of similar text between two strings, in medical student notes and evaluate the relationship between note redundancy and objective indicators of student performance. METHODS: Notes generated by medical students rotating through their medicine clerkship during a single academic year at our institution were analyzed. A student-patient interaction (SPI) was defined as a history and physical and at least two contiguous progress notes authored by the same student during a single patient's hospitalization. For some students, SPI pairs were available from early and late in the clerkship. Redundancy between analogous sections of consecutive notes was calculated on a 0 to 100% scale and was derived from edit distance, the number of changes needed to transform one text string into another. Indicators of student performance included United States Medical Licensing Exam (USMLE) scores. RESULTS: Ninety-four single SPIs and 58 SPI pairs were analyzed. Redundancy in the assessment/plan section was high (40%) and increased within individual SPIs (to 60%; p < 0.001) and between SPI pairs over the course of the clerkship (by 30-40%; p < 0.001). Students in the lowest tertile of USMLE step II clinical knowledge scores had higher redundancy in the assessment/plan section than their classmates (67 ± 24% vs. 38 ± 22%; p = 0.002). CONCLUSION: During the medicine clerkship, the assessment/plan section of medical student notes became more redundant over a patient's hospital course and as students gained clinical experience. These trends may be indicative of deficiencies in clinical knowledge or reasoning, as evidenced by performance on some standardized evaluations.

Imprinting on Clinical Rotations: Multisite Survey of High- and Low-Value Medical Student Behaviors and Relationship with Healthcare Intensity.

BACKGROUND: Physician behaviors are important to high-value care, and the learning environment medical students encounter on clinical clerkships may imprint their developing practice patterns. OBJECTIVES: To explore potential imprinting on clinical rotations by (a) describing high- and low-value behaviors among medical students and (b) examining relationships with regional healthcare intensity (HCI). DESIGN: Multisite cross-sectional survey PARTICIPANTS: Third- and fourth-year students at nine US medical schools MAIN MEASURES: Survey items measured high-value (n = 10) and low-value (n = 9) student behaviors. Regional HCI was measured using Dartmouth Atlas End-of-Life Chronic Illness Care data (ratio of physician visits per decedent compared with the US average, hospital care intensity index, ratio of medical specialty to primary care physician visits per decedent). Associations between regional HCI and student behaviors were examined using unadjusted and adjusted (controlling for age, sex, and year in school) logistic regression analyses, using median item ratings to summarize reported engagement in high- and low-value behaviors. KEY RESULTS: Of 2623 students invited, 1304 (50%) responded. Many reported trying to determine healthcare costs (1085/1234, 88%), but only 45% (571/1257) reported including cost details in case presentations. Students acknowledged suggesting tests solely to anticipate what their supervisor would want (1143/1220, 94%), show off their ability to generate a broad differential diagnosis (1072/1218, 88%), satisfy curiosity (958/1217, 79%), protect the team from liability (938/1215, 77%), and build clinical experience (533/1217, 44%). Students in higher intensity regions reported significantly more low-value behaviors: each one-unit increase in the ratio of physician visits per decedent increased the odds of reporting low-value behaviors by 20% (OR 1.20, 95% CI 1.04-1.38; P = 0.01). CONCLUSIONS: Third- and fourth-year medical students report engaging in both high- and low-value behaviors, which are related to regional HCI. This underscores the importance of the clinical learning environment and suggests imprinting is already underway during medical school.

Attitudes toward cost-conscious care among U.S. physicians and medical students: analysis of national cross-sectional survey data by age and stage of training.

BACKGROUND: The success of initiatives intended to increase the value of health care depends, in part, on the degree to which cost-conscious care is endorsed by current and future physicians. This study aimed to first analyze attitudes of U.S. physicians by age and then compare the attitudes of physicians and medical students. METHODS: A paper survey was mailed in mid-2012 to 3897 practicing physicians randomly selected from the American Medical Association Masterfile. An electronic survey was sent in early 2015 to all 5,992 students at 10 U.S. medical schools. Survey items measured attitudes toward cost-conscious care and perceived responsibility for reducing healthcare costs. Physician responses were first compared across age groups (30-40 years, 41-50 years, 51-60 years, and > 60 years) and then compared to student responses using Chi square tests and logistic regression analyses (controlling for sex). RESULTS: A total of 2,556 physicians (65%) and 3395 students (57%) responded. Physician attitudes generally did not differ by age, but differed significantly from those of students. Specifically, students were more likely than physicians to agree that cost to society should be important in treatment decisions (p < 0.001) and that physicians should sometimes deny beneficial but costly services (p < 0.001). Students were less likely to agree that it is unfair to ask physicians to be cost-conscious while prioritizing patient welfare (p < 0.001). Compared to physicians, students assigned more responsibility for reducing healthcare costs to hospitals and health systems (p < 0.001) and less responsibility to lawyers (p < 0.001) and patients (p < 0.001). Nearly all significant differences persisted after controlling for sex and when only the youngest physicians were compared to students. CONCLUSIONS: Physician attitudes toward cost-conscious care are similar across age groups. However, physician attitudes differ significantly from medical students, even among the youngest physicians most proximate to students in age. Medical student responses suggest they are more accepting of cost-conscious care than physicians and attribute more responsibility for reducing costs to organizations and systems rather than individuals. This may be due to the combined effects of generational differences, new medical school curricula, students' relative inexperience providing cost-conscious care within complex healthcare systems, and the rapidly evolving U.S. healthcare system.

Framing Medicine as a Moral Practice: An Introductory Medical School Course.

PROBLEM: The transition into medical school represents a time of profound professional development for medical students. Many medical schools manage this transition with brief orientations followed by abrupt moves into the anatomy laboratory. Recognizing that early introduction of key humanistic concepts could have a lasting impact on students' attitudes, faculty at the Vanderbilt University School of Medicine created the Foundations of the Profession (FoP) course to frame medicine as a moral practice. APPROACH: The FoP course, offered annually since 2012, occurs during the first week of medical school. Using coronary artery disease with a chief complaint of angina as a core example, teams of students create variations of five hypothetical patients and walk them through potential care episodes. This allows students to compare the impact of many factors on a provider's ability to uphold fundamental professional obligations. Students engage in readings, lectures, small-group discussions, clinic visits, and research on insurance plans. Faculty engage with students in small groups and establish a safe environment for discussion of challenging moral dilemmas. OUTCOMES: From 2013 to 2016, 356 (97%) of the 368 first-year medical students who took the course submitted summative course evaluations. Of the respondents, 349 (98%) indicated they believed the course contributed to their professional development and supported their learning. NEXT STEPS: Future iterations of this course may include increased exposure to key educational faculty to solidify the formation of a moral scaffold on which to build subsequent knowledge.

Value-Added Activities in Medical Education: A Multisite Survey of First- and Second-Year Medical Students' Perceptions and Factors Influencing Their Potential Engagement.

PURPOSE: To describe attitudes of first- and second-year U.S. medical students toward value-added medical education, assess their self-reported desire to participate in value-added activities, and identify potentially modifiable factors influencing their engagement. METHOD: The authors conducted a cross-sectional survey of first- and second-year students at nine U.S. medical schools in 2017. Survey items measured students' attitudes toward value-added medical education (n = 7), desire to participate in value-added activities (n = 20), and factors influencing potential engagement (n = 18). RESULTS: Of 2,670 students invited to participate, 1,372 (51%) responded. Seventy-six percent (1,043/1,368) moderately or strongly agreed they should make meaningful contributions to patient care. Students' desire to participate was highest for patient care activities approximating those traditionally performed by physicians, followed by systems improvement activities and lowest for activities not typically performed by physicians. Factors increasing desire to participate included opportunities to interact with practicing physicians (1,182/1,244; 95%), patients (1,177/1,246; 95%), and residents or fellows (1,166/1,246; 94%). Factors decreasing desire to participate included making changes to the health care system (365/1,227; 30%), interacting with patients via phone or electronic communication (410/1,243; 33%), and lack of curricular time (634/1,233; 51%). CONCLUSIONS: First- and second-year medical students agree they should add value to patient care, but their desire to participate in value-added activities varies depending on the nature of the tasks. Medical schools may be able to increase students' desire to participate by enabling face-to-face interactions with patients, embedding students in health care teams, and providing dedicated curricular time.