Fidelity to Best Practices in EPA Implementation: Outcomes Supporting Use of the Core Components Framework From the University of Virginia Entrustable Professional Activity Program.

PROBLEM: The rapid expansion of entrustable professional activity (EPA) assessment programs has led to calls to ensure fidelity in implementation and integrity in meeting the goals of competency-based medical education. Initiated in July 2017, in advance of the articulated core components of EPA implementation, this article describes the structure and outcomes of the University of Virginia (UVA) EPA Program and provides support for the identified essential components. APPROACH: The UVA EPA Program includes workplace assessments by residents/fellows, attending faculty, and master assessors (MAs), experienced clinicians who assess students across disciplines and clinical settings. All assessors participate in formal professional development and provide verbal and written comments to support their supervision ratings. The Entrustment Committee, composed of 12 MAs, uses a shared mental model and aggregates all assessor data to make a high-stakes summative entrustment decision about students' readiness to assume the role of an acting intern. OUTCOMES: Since 2017, over 2,000 assessors have completed 56,969 EPA assessments for 1,479 students. Ninety-four percent of assessments have been done during the clerkship phase. Residents/fellows have completed a mean of 18 assessments, attending faculty a mean of 27, and MAs a mean of 882. Seventy-four percent of observed encounters involved patients with acute concerns with or without a co-morbid condition. Fifty percent of assessments occurred in inpatient and 32% in ambulatory settings. Eighty-seven percent of assessments contained narrative comments with more than 100 characters. NEXT STEPS: Planned next steps will include earlier identification of students who require individualized learning to promote the development of skills related to EPAs, expansion of the remediation program to enable more students to engage in a clinical performance mastery elective, and creation of targeted professional development for assessors to reinforce the tenets of the EPA program.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment.

INTRODUCTION: The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated. METHODS: In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding. RESULTS: Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout. CONCLUSIONS: Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned.

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

Using Instrument-Guided Team Reflection and Debriefing to Cultivate Teamwork Knowledge, Skills, and Attitudes in Pre-Clerkship Learning Teams.

Drawing on the science of teamwork and the science of learning, we designed an instrument-guided team reflection and debriefing activity to foster teamwork knowledge, skills, and attitudes (KSAs) in medical students. We then embedded this activity within and between a biweekly series of pre-clerkship Team-Based Learning sessions with the goal of encouraging medical students to cultivate a practical and metacognitive appreciation of eight foundational teamwork KSAs that are applicable to both healthcare teams and classroom learning teams. On evaluations, 144 learners from a class of 156 reported increased appreciation for and team improvement with these teamwork KSAs.

The Cadaveric Skin Biopsy Project: description and student evaluation of an innovative approach to dermatology instruction in the preclerkship medical school curriculum.

BACKGROUND: Dermatology can develop creative ways of participating in the preclerkship medical school curriculum. OBJECTIVE: We sought to describe and report student survey results of a novel collaborative learning activity for medical students, directed by dermatology, histology, and gross anatomy faculty, which used cadavers to replicate the process of skin lesion biopsy and provided a realistic setting in which to learn normal-appearing and abnormal skin histology. METHODS: First-year medical students were surveyed regarding the impact of this activity on their understanding of skin histology and their appreciation of dermatology and dermatologic procedures. RESULTS: Students were appreciative of the opportunity to perform biopsies and discover the link between the clinical presentation of a lesion and its underlying histopathology. They were less impressed with the ability of the activity to improve their understanding of the characteristics of benign versus malignant lesions. LIMITATIONS: This is an early feasibility trial at 1 institution. CONCLUSIONS: This project represents one approach to introducing students to dermatology and dermatologic procedures and achieves institutional, Liaison Committee on Medical Education, and Association of American Medical Colleges educational goals. Overall, students highly valued the opportunities to practice clinical procedures and found it aided their understanding and appreciation of dermatology.

Responding to a changing nation: are faculty prepared for cross-cultural conversations and care?

BACKGROUND AND OBJECTIVES: The United States is becoming increasingly diverse. Health disparities continue with little improvement despite national policies and standards. Medical institutions are modifying their curricula; however, little is known about faculty attitudes and comfort in addressing cultural issues. The purpose of this study was to determine faculty attitudes, self-perceived levels of comfort and skill, and future knowledge needs pertaining to cultural competence. METHODS: A survey was administered to all clinical faculty at the University of Virginia School of Medicine. Survey questions addressed faculty attitudes and self-perceived levels of comfort and skill in dealing with cultural issues, as well as perceived need and interest in future cultural competence training. RESULTS: When considering each phase of education (medical school, residency, continuing medical education [CME]), fewer than 25% of the respondents reported receiving formal instruction in cultural competency in any given phase, although 93% felt that cultural competency training was important. Fifty-eight percent felt "very comfortable" caring for diverse patients, although this dropped to 30% when specifying limited English proficiency. The situation in which the highest percentage of respondents felt "not particularly comfortable" or "not at all comfortable" was breaking bad news to a patient's family first if this was more culturally appropriate (47%). Respondents felt most skilled in working with medical interpreters, apologizing for cross-cultural misunderstandings, and eliciting the patients' perspectives about their health and illness. Respondents felt the least skilled providing culturally sensitive end-of-life care and dealing with cross-cultural conflicts. CONCLUSIONS: Clinical faculty have received limited instruction on cultural competency, and the reported levels of comfort and skill in many challenging areas of multicultural health leave much room for improvement. Until faculty become more comfortable and are able to model and teach these behaviors to learners, positive responses to national policies in culturally competent care are likely to be limited.

Sharpening the eye of the OSCE with critical action analysis.

PURPOSE: When interpreting performance scores on an objective structured clinical examination (OSCE), are all checklist items created equal? Although assigning priority through checklist item weighting is often used to emphasize the clinical importance of selected checklist items, the authors propose the use of critical action analysis as an additional method for analyzing and discriminating clinical performance in clinical skill assessment exercises. A critical action is defined as an OSCE checklist item whose performance is critical to ensure an optimal patient outcome and avoid medical error. In this study, the authors analyzed a set of clerkship OSCE performance outcome data and compared the results of critical action analysis versus traditional checklist item performance scores. METHOD: OSCE performance scores of 398 third-year clerkship students from 2003 to 2006 at the University of Virginia School of Medicine were analyzed using descriptive statistics and a logistic regression model. Through a consensus process, 10 of 25 OSCE cases were identified as containing critical actions. RESULTS: Students who scored above the median correctly performed the critical actions more often than those scoring lower. However, for 9 of 10 cases, 6% to 46% of higher-scoring students failed to perform the critical action correctly. CONCLUSIONS: Failure to address this skill assessment outcome is a missed opportunity to more fully understand and apply the results of such examinations to the clinical performance development of medical students. Including critical action analysis in OSCE data interpretation sharpens the eye of the OSCE and enhances its value in clinical skill assessment.

Enhancing clinical skills education: University of Virginia School of Medicine's Clerkship Clinical Skills Workshop Program.

In 1993, the University of Virginia School of Medicine began a clinical skills workshop program in an effort to improve the preparation of all clerkship students to participate in clinical care. This program involved the teaching of selected basic clinical skills by interested faculty to small groups of third-year medical students. Over the past 14 years, the number of workshops has increased from 11 to 31, and they now involve clerkship faculty from family medicine, internal medicine, and pediatrics. Workshops include a variety of common skills from the communication, physical examination, and clinical test and procedure domains such as pediatric phone triage, shoulder examination, ECG interpretation, and suturing. Workshop sessions allow students to practice skills on each other, with standardized patients, or with models, with the goal of improving competence and confidence in the performance of basic clinical skills. Students receive direct feedback from faculty on their skill performance. The style and content of these workshops are guided by an explicit set of educational criteria.A formal evaluation process ensures that faculty receive regular feedback from student evaluation comments so that adherence to workshop criteria is continuously reinforced. Student evaluations confirm that these workshops meet their skill-learning needs. Preliminary outcome measures suggest that workshop teaching can be linked to student assessment data and may improve students' skill performance. This program represents a work-in-progress toward the goal of providing a more comprehensive and developmental clinical skills curriculum in the school of medicine.