Viewing Readiness-for-Residency through Binoculars: Mapping Competency-Based Assessments to the AAMC's 13 Core Entrustable Professional Activities (EPAs).

Construct: The construct being assessed is readiness-for-residency of graduating medical students, as measured through two assessment frameworks. Background: Readiness-for-residency of near-graduate medical students should be but is not consistently assessed. To address this, the Association of American Medical Colleges (AAMC), in 2014, identified and described 13 core Entrustable Professional Activities (EPAs), which are tasks that all residents should be able to perform unsupervised upon entering residency. However, the AAMC did not initially provide measurement guidelines or propose standardized assessments. We designed Night-onCall (NOC), an immersive simulation for our near-graduating medical students to assess and address their readiness-for-residency, framed around tasks suggested by the AAMC's core EPAs. In adopting this EPA assessment framework, we began by building upon an established program of competency-based clinical skills assessments, repurposing competency-based checklists to measure components of the EPAs where possible, and designing new checklists, when necessary. This resulted in a blended suite of 14 checklists, which theoretically provide substantive assessment of all 13 core EPAs. In this paper, we describe the consensus-based mapping process conducted to ensure we understood the relationship between competency and EPA-based assessment lenses and could therefore report meaningful feedback on both to transitioning students in the NOC exercise. Approach: Between January-November 2017, five clinician and two non-clinician health professions educators at NYU Grossman School of Medicine conducted a rigorous consensus-based mapping process, which included each rater mapping each of the 310 NOC competency-based checklist items to lists of entrustable behaviors expected of learners according to the AAMC 13 core EPAs. Findings: All EPAs were captured to varying degrees by the 14 NOC checklists (overall Intraclass Correlation Coefficient (ICC) = 0.77). Consensus meetings resolved discrepancies and improved ICC values for three (EPA-9, EPA-10, EPA-12) of the four EPAs that initially showed poor reliability. Conclusions: Findings suggest that with some limitations (e.g., EPA-7 "form clinical questions/retrieve evidence") established competency-based assessments can be repurposed to measure readiness-for-residency through an EPA lens and both can be reported to learners and faculty.

Improving the Clinical Skills Performance of Graduating Medical Students Using "WISE OnCall," a Multimedia Educational Module.

INTRODUCTION: "Transitions to residency" programs are designed to maximize quality and safety of patient care, as medical students become residents. However, best instructional or readiness assessment practices are not yet established. We sought to study the impact of a screen-based interactive curriculum designed to prepare interns to address common clinical coverage issues (WISE OnCall) on the clinical skills demonstrated in simulation and hypothesize that performance would improve after completing the module. METHODS: Senior medical students were recruited to participate in this single group prestudy/poststudy. Students responded to a call from a standardized nurse (SN) and assessed a standardized patient (SP) with low urine output, interacted with a 45-minute WISE OnCall module on the assessment and management of oliguria, and then evaluated a different SP with low urine output of a different underlying cause. Standardized patients assessed clinical skills with a 37-item, behaviorally anchored checklist measuring clinical skills (intraclass correlation coefficient [ICC], 0.55-0.81). Standardized nurses rated care quality and safety and collaboration and interprofessional communication using a 33-item literature-based, anchored checklist (ICC, 0.47-0.52). Standardized patient and SN ratings of the same student performance were correlated (r, 0.37-0.62; P < 0.01). Physicians assessed clinical reasoning quality based on the students' patient encounter note (ICC, 0.55-0.68), ratings that did not correlate with SP and SN ratings. We compared pre-post clinical skills performance and clinical reasoning. Fifty-two medical students (31%) completed this institutional review board -approved study. RESULTS: Performance as measured by the SPs, SNs, and the postencounter note all showed improvement with mostly moderate to large effect sizes (range of Cohen's d, 0.30-1.88; P < 0.05) after completion of the online module. Unexpectedly, professionalism as rated by the SP was poorer after the module (Cohen's d, -0.93; P = 0.000). DISCUSSION: A brief computer-based educational intervention significantly improved graduating medical students' clinical skills needed to be ready for residency.

A simulated "Night-onCall" to assess and address the readiness-for-internship of transitioning medical students.

Transitioning medical students are anxious about their readiness-for-internship, as are their residency program directors and teaching hospital leadership responsible for care quality and patient safety. A readiness-for-internship assessment program could contribute to ensuring optimal quality and safety and be a key element in implementing competency-based, time-variable medical education. In this paper, we describe the development of the Night-onCall program (NOC), a 4-h readiness-for-internship multi-instructional method simulation event. NOC was designed and implemented over the course of 3 years to provide an authentic "night on call" experience for near graduating students and build measurements of students' readiness for this transition framed by the Association of American Medical College's Core Entrustable Professional Activities for Entering Residency. The NOC is a product of a program of research focused on questions related to enabling individualized pathways through medical training. The lessons learned and modifications made to create a feasible, acceptable, flexible, and educationally rich NOC are shared to inform the discussion about transition to residency curriculum and best practices regarding educational handoffs from undergraduate to graduate education.

Adapting a weight management tool for Latina women: a usability study of the Veteran Health Administration's MOVE!23 tool.

BACKGROUND: Obesity disproportionately affects Latina women, but few targeted, technology-assisted interventions that incorporate tailored health information exist for this population. The Veterans Health Administration (VHA) uses an online weight management tool (MOVE!23) which is publicly available, but was not designed for use in non-VHA populations. METHODS: We conducted a qualitative study to determine how interactions between the tool and other contextual elements impacted task performance when the target Latina users interacted with MOVE!23. We sought to identify and classify specific facilitators and barriers that might inform design changes to the tool and its context of use, and in turn promote usability. Six English-speaking, adult Latinas were recruited from an inner city primary care clinic and a nursing program at a local university in the United States to engage in a "Think-Aloud" protocol while using MOVE!23. Sessions were recorded, transcribed, and coded to identify interactions between four factors that contribute to usability (Tool, Task, User, Context). RESULTS: Five themes influencing usability were identified: Technical Ability and Technology Preferences; Language Confusion and Ambiguity; Supportive Tool Design and Facilitator Guidance; Relevant Examples; and Personal Experience. Features of the tool, task, and other contextual factors failed to fully support participants at times, impeding task completion. Participants interacted with the tool more readily when its language was familiar and content was personally relevant. When faced with ambiguity and uncertainty, they relied on the tool's visual cues and examples, actively sought relevant personal experiences, and/or requested facilitator support. CONCLUSIONS: The ability of our participants to successfully use the tool was influenced by the interaction of individual characteristics with those of the tool and other contextual factors. We identified both tool-specific and context-related changes that could overcome barriers to the use of MOVE!23 among Latinas. Several general considerations for the design of eHealth tools are noted.

Twelve tips for improving the effectiveness of web-based multimedia instruction for clinical learners.

Using educational technology does not necessarily make medical education more effective. There are many different kinds of technology available to the contemporary medical teacher and what constitutes effective use may depend on the technology, the learning situation and many other factors. Web-based multimedia instruction (WBMI) provides learners with self-directed independent learning opportunities based on didactic material enhanced with multimedia features such as video and animations. WBMI may be used to replace other didactic events (e.g. lectures) or it may be provided in addition to other learning opportunities. Clinical educators looking to use WBMI need to make sure that it will meet both their learners' needs and the program's needs, and it has to align to the contexts in which it is used. The following 12 tips have been developed to help guide faculty through some of the key features of the effective use of WBMI in clinical teaching programs. These tips are based on more than a decade developing, using and appraising WBMI in support of surgical clerkship education across the USA and beyond and they are intended both to inform individual uses of WBMI in clinical training and to guide the strategic use of WBMI in clinical clerkship curricula.

WISE-MD usage among millennial medical students.

BACKGROUND: E-learning is increasingly common in undergraduate medical education. Internet-based multimedia materials should be designed with millennial learner utilization preferences in mind for maximal impact. METHODS: Medical students used all 20 Web Initiative for Surgical Education of Medical Doctors modules from July 1, 2013 to October 1, 2013. Data were analyzed for topic frequency, time and week day, and access to questions. RESULTS: Three thousand five hundred eighty-seven students completed 35,848 modules. Students accessed modules for average of 51 minutes. Most frequent use occurred on Sunday (23.1%), Saturday (15.4%), and Monday (14.3%). Friday had the least use (8.2%). A predominance of students accessed the modules between 7 and 10 PM (34.4%). About 80.4% of students accessed questions for at least one module. They completed an average of 40 ± 30 of the questions. Only 827 students (2.3%) repeated the questions. CONCLUSIONS: Web Initiative for Surgical Education of Medical Doctors has peak usage during the weekend and evenings. Most frequently used modules reflect core surgical problems. Multiple factors influence the manner module questions are accessed.

The cognitive impact of interactive design features for learning complex materials in medical education.

To identify the most effective way for medical students to interact with a browser-based learning module on the symptoms and neurological underpinnings of stroke syndromes, this study manipulated the way in which subjects interacted with a graphical model of the brain and examined the impact of functional changes on learning outcomes. It was hypothesized that behavioral interactions that were behaviorally more engaging and which required deeper consideration of the model would result in heightened cognitive interaction and better learning than those whose manipulation required less deliberate behavioral and cognitive processing. One hundred forty four students were randomly assigned to four conditions whose model controls incorporated features that required different levels of behavioral and cognitive interaction: Movie (low behavioral/low cognitive, n = 40), Slider (high behavioral/low cognitive, n = 36), Click (low behavioral/high cognitive, n = 30), and Drag (high behavioral/high cognitive, n = 38). Analysis of Covariates (ANCOVA) showed that students who received the treatments associated with lower cognitive interactivity (Movie and Slider) performed better on a transfer task than those receiving the module associated with high cognitive interactivity (Click and Drag, partial eta squared = .03). In addition, the students in the high cognitive interactivity conditions spent significantly more time on the stroke locator activity than other conditions (partial eta squared = .36). The results suggest that interaction with controls that were tightly coupled with the model and whose manipulation required deliberate consideration of the model's features may have overtaxed subjects' cognitive resources. Cognitive effort that facilitated manipulation of content, though directed at the model, may have resulted in extraneous cognitive load, impeding subjects in recognizing the deeper, global relationships in the materials. Instructional designers must, therefore, keep in mind that the way in which functional affordances are integrated with the content can shape both behavioral and cognitive processing, and has significant cognitive load implications.

Medical students as human subjects in educational research.

INTRODUCTION: Special concerns often arise when medical students are themselves the subjects of education research. A recently completed large, multi-center randomized controlled trial of computer-assisted learning modules for surgical clerks provided the opportunity to explore the perceived level of risk of studies where medical students serve as human subjects by reporting on: 1) the response of Institutional Review Boards (IRBs) at seven institutions to the same study protocol; and 2) the thoughts and feelings of students across study sites about being research subjects. METHODS: From July 2009 to August 2010, all third-year medical students at seven collaborating institutions were eligible to participate. Patterns of IRB review of the same protocol were compared. Participation burden was calculated in terms of the time spent interacting with the modules. Focus groups were conducted with medical students at each site. Transcripts were coded by three independent reviewers and analyzed using Atlas.ti. RESULTS: The IRBs at the seven participating institutions granted full (n=1), expedited (n=4), or exempt (n=2) review of the WISE Trial protocol. 995 (73% of those eligible) consented to participate, and 207 (20%) of these students completed all outcome measures. The average time to complete the computer modules and associated measures was 175 min. Common themes in focus groups with participant students included the desire to contribute to medical education research, the absence of coercion to consent, and the low-risk nature of the research. DISCUSSION: Our findings demonstrate that risk assessment and the extent of review utilized for medical education research vary among IRBs. Despite variability in the perception of risk implied by differing IRB requirements, students themselves felt education research was low risk and did not consider themselves to be vulnerable. The vast majority of eligible medical students were willing to participate as research subjects. Participants acknowledged the time demands of their participation and were readily able to withdraw when those burdens became unsustainable.