Learner Assessment and Program Evaluation: Supporting Precision Education.

ABSTRACT: Precision education (PE) systematically leverages data and advanced analytics to inform educational interventions that, in turn, promote meaningful learner outcomes. PE does this by incorporating analytic results back into the education continuum through continuous feedback cycles. These data-informed sequences of planning, learning, assessing, and adjusting foster competence and adaptive expertise. PE cycles occur at individual (micro), program (meso), or system (macro) levels. This article focuses on program- and system-level PE.Data for PE come from a multitude of sources, including learner assessment and program evaluation. The authors describe the link between these data and the vital role evaluation plays in providing evidence of educational effectiveness. By including prior program evaluation research supporting this claim, the authors illustrate the link between training programs and patient outcomes. They also describe existing national reports providing feedback to programs and institutions, as well as 2 emerging, multiorganization program- and system-level PE efforts. The challenges encountered by those implementing PE and the continuing need to advance this work illuminate the necessity for increased cross-disciplinary collaborations and a national cross-organizational data-sharing effort.Finally, the authors propose practical approaches for funding a national initiative in PE as well as potential models for advancing the field of PE. Lessons learned from successes by others illustrate the promise of these recommendations.

Social network analysis of publication collaboration of accelerating change in MedEd consortium.

INTRODUCTION: The American Medical Association formed the Accelerating Change in Medical Education Consortium through grants to effect change in medical education. The dissemination of educational innovations through scholarship was a priority. The objective of this study was to explore the patterns of collaboration of educational innovation through the consortium's publications. METHOD: Publications were identified from grantee schools' semi-annual reports. Each publication was coded for the number of citations, Altmetric score, domain of scholarship, and collaboration with other institutions. Social network analysis explored relationships at the midpoint and end of the grant. RESULTS: Over five years, the 32 Consortium institutions produced 168 publications, ranging from 38 papers from one institution to no manuscripts from another. The two most common domains focused on health system science (92 papers) and competency-based medical education (30 papers). Articles were published in 54 different journals. Forty percent of publications involved more than one institution. Social network analysis demonstrated rich publishing relationships within the Consortium members as well as beyond the Consortium schools. In addition, there was growth of the network connections and density over time. CONCLUSION: The Consortium fostered a scholarship network disseminating a broad range of educational innovations through publications of individual school projects and collaborations.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education.

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

Realizing the vision of the Lancet Commission on Education of Health Professionals for the 21st Century: Transforming medical education through the Accelerating Change in Medical Education Consortium.

In the last two decades, prompted by the anticipated arrival of the 21st Century and on the centenary of the publication of the Flexner Report, many in medical education called for change to address the expanding chasm between the requirements of the health care system and the educational systems producing the health care workforce. Calls were uniform. Curricular content was missing. There was a mismatch in where people trained and where they were needed to practice, legacy approaches to pedagogical methods that needed to be challenged, an imbalance in diversity of trainees, and a lack of research on educational outcomes, resulting in a workforce that was described as ill-equipped to provide health care in the current and future environment. The Lancet Commission on Education of Health Professionals for the 21st Century published a widely acclaimed report in 2010 that called for a complete and authoritative re-examination of health professional education. This paper describes the innovations of the American Medical Association Accelerating Change in Medical Education Consortium schools as they relate to the recommendations of the Lancet Commission. We outline the successes, challenges, and lessons learned in working to deeply reform medical education.

Revisiting the Core Entrustable Professional Activities for Entering Residency.

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

A Responsible Educational Handover: Improving Communication to Improve Learning.

An important tenet of competency-based medical education is that the educational continuum should be seamless. The transition from undergraduate medical education (UME) to graduate medical education (GME) is far from seamless, however. Current practices around this transition drive students to focus on appearing to be competitively prepared for residency. A communication at the completion of UME-an educational handover-would encourage students to focus on actually preparing for the care of patients. In April 2018, the American Medical Association's Accelerating Change in Medical Education consortium meeting included a debate and discussion on providing learner performance measures as part of a responsible educational handover from UME to GME. In this Perspective, the authors describe the resulting 5 recommendations for developing such a handover: (1) The purpose of the educational handover should be to provide medical school performance data to guide continued improvement in learner ability and performance, (2) the process used to create an educational handover should be philosophically and practically aligned with the learner's continuous quality improvement, (3) the educational handover should be learner driven with a focus on individualized learning plans that are coproduced by the learner and a coach or advisor, (4) the transfer of information within an educational handover should be done in a standardized format, and (5) together, medical schools and residency programs must invest in adequate infrastructure to support learner improvement. These recommendations are shared to encourage implementation of the educational handover and to generate a potential research agenda that can inform policy and best practices.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency.

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Assessing learning in the adaptive curriculum.

BACKGROUND: Medical education is a dynamic process that will continuously evolve to respond to changes in the foundations of medicine, the clinical practice of medicine and in health systems science. PURPOSE: In this paper, we review how assessing learning in such a dynamic environment requires comprehensive flexible and adaptable methodological approaches designed to assess knowledge attainment and transfer, clinical skills/competency development, and ethical/professional behavior. Adaptive assessments should measure the learner's ability to observe where changes in health care delivery are needed and how to implement them. Balancing formative and summative assessments will promote reflective learning so that each student will reach her/his highest potential. From the programmatic perspective, measuring the design and delivery of instruction in relation to students? efforts to achieve competency will improve learning and foster continuous professional development of faculty and advance the science of learning. APPROACH: We describe how two medical schools are approaching adaptive assessment, including using portfolio systems that encompass teaching and learning experiences while offering real-time longitudinal tracking of digital data toward improving learning and provide curricula continuous improvement cycles. Using latest technologies, portfolios produce actionable data displays with precise guidance for learning and program development.

Describing the Journey and Lessons Learned Implementing a Competency-Based, Time-Variable Undergraduate Medical Education Curriculum.

Oregon Health & Science University School of Medicine launched a completely new undergraduate medical education curriculum in 2014. This initiative dramatically transformed the MD degree program, changing the instructional content taught, the pedagogical methods used by the faculty, and the methods of assessment, and it added new elements such as academic coaching and programmatic entrustment to the program. One of the most exciting and impactful aspects to date of this curricular transformation has been the deliberate implementation of a competency-based framework that incorporates frequent assessment, tracking of student progression using an electronic portfolio, and academic coaching to optimize learning and customize curricular elements for each student. The next major step in this process-the implementation of time-variable progression-is currently ongoing as a planning group at the school works through the conceptual, logistical, legal, and regulatory issues related to implementing such a system. When implementation is complete, MD students will graduate only once they have earned entrustment for all 13 Core Entrustable Professional Activities for Entering Residency. This article describes the school's progress to date in its curricular transformation and articulates lessons learned thus far in driving substantive and dramatic institutional changes that profoundly impact students, faculty, and administrators in one academic health center.

Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot.

In 2014, the Association of American Medical Colleges (AAMC) published a list of 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs) that medical school graduates might be expected to perform, without direct supervision, on the first day of residency. Soon after, the AAMC commissioned a five-year pilot with 10 medical schools across the United States, seeking to implement the Core EPA framework to improve the transition from undergraduate to graduate medical education.In this article, the pilot team presents the organizational structure and early results of collaborative efforts to provide guidance to other institutions planning to implement the Core EPA framework. They describe the aims, timeline, and organization of the pilot as well as findings to date regarding the concepts of entrustment, assessment, curriculum development, and faculty development. On the basis of their experiences over the first two years of the pilot, the authors offer a set of guiding principles for institutions intending to implement the Core EPA framework. They also discuss the impact of the pilot, its limitations, and next steps, as well as how the pilot team is engaging the broader medical education community. They encourage ongoing communication across institutions to capitalize on the expertise of educators to tackle challenges related to the implementation of this novel approach and to generate common national standards for entrustment. The Core EPA pilot aims to better prepare medical school graduates for their professional duties at the beginning of residency with the ultimate goal of improving patient care.

Applying the institutional review board data repository approach to manage ethical considerations in evaluating and studying medical education.

ISSUE: Medical educators and educational researchers continue to improve their processes for managing medical student and program evaluation data using sound ethical principles. This is becoming even more important as curricular innovations are occurring across undergraduate and graduate medical education. Dissemination of findings from this work is critical, and peer-reviewed journals often require an institutional review board (IRB) determination. APPROACH: IRB data repositories, originally designed for the longitudinal study of biological specimens, can be applied to medical education research. The benefits of such an approach include obtaining expedited review for multiple related studies within a single IRB application and allowing for more flexibility when conducting complex longitudinal studies involving large datasets from multiple data sources and/or institutions. In this paper, we inform educators and educational researchers on our analysis of the use of the IRB data repository approach to manage ethical considerations as part of best practices for amassing, pooling, and sharing data for educational research, evaluation, and improvement purposes. IMPLICATIONS: Fostering multi-institutional studies while following sound ethical principles in the study of medical education is needed, and the IRB data repository approach has many benefits, especially for longitudinal assessment of complex multi-site data.

Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education.

Physicians in the 21st century will increasingly interact in diverse ways with information systems, requiring competence in many aspects of clinical informatics. In recent years, many medical school curricula have added content in information retrieval (search) and basic use of the electronic health record. However, this omits the growing number of other ways that physicians are interacting with information that includes activities such as clinical decision support, quality measurement and improvement, personal health records, telemedicine, and personalized medicine. We describe a process whereby six faculty members representing different perspectives came together to define competencies in clinical informatics for a curriculum transformation process occurring at Oregon Health & Science University. From the broad competencies, we also developed specific learning objectives and milestones, an implementation schedule, and mapping to general competency domains. We present our work to encourage debate and refinement as well as facilitate evaluation in this area.

Factors contributing to successful interorganizational collaboration: the case of CS2day.

Continuing medical education's transition from an emphasis on dissemination to changing clinical practice has made it increasingly necessary for CME providers to develop effective interorganizational collaborations. Although interorganizational collaboration has become commonplace in most sectors of government, business, and academia, our review of the literature and experience as practitioners and researchers suggest that the practice is less widespread in the CME field. The absence of a rich scholarly literature on establishing and maintaining interorganizational collaborations to provide continuing education to health professionals means there is little information about how guidelines and principles for effective collaboration developed in other fields might apply to continuing professional development in health care and few models of successful collaboration. The purpose of this article is to address this gap by describing a successful interorganizational CME collaboration-Cease Smoking Today (CS2day)-and summarizing what was learned from the experience, extending our knowledge by exploring and illustrating points of connection between our experience and the existing literature on successful interorganizational collaboration. In this article, we describe the collaboration and the clinical need it was organized to address, and review the evidence that led us to conclude the collaboration was successful. We then discuss, in the context of the literature on effective interorganizational collaboration, several factors we believe were major contributors to success. The CS2day collaboration provides an example of how guidelines for collaboration developed in various contexts apply to continuing medical education and a case example providing insight into the pathways that lead to a collaboration's success.

Advancing public health through continuing education of health care professionals.

This article describes how the CS2day (Cease Smoking Today) initiative positioned continuing education (CE) in the intersection between medicine and public health. The authors suggest that most CE activities address the medical challenges that clinicians confront, often to the neglect of the public health issues that are key risk factors for the onset and exacerbation of diseases. The authors further suggest that the educational activities of the CS2day initiative functioned as Type III translational science in that it facilitated the use of research-derived practice guidelines in clinical practice and in the community. The article concludes by stating that the successful results of the CS2day initiative illustrate what can happen when continuing education efforts develop from a public health problem rather than just a practice gap identified in a clinical practice setting.

Clinical decision support systems use in Wisconsin.

CONTEXT: Clinical decision support systems (CDSS) are becoming increasingly common in medical practice. OBJECTIVE: To assess utilization, level of interest, and potential barriers to implementation of CDSS among physicians providing inpatient care in Wisconsin. DESIGN AND PARTICIPANTS: A Web-based survey consisting of 20 questions e-mailed to 5783 members of the Wisconsin Medical Society. RESULTS: Of those contacted, 496 (9%) responded and 356 (72%) were eligible for the survey. According to 38% of respondents, CDSS were in place in their facility; less than a third were computer-based. Few existing users of CDSS reported being dissatisfied (2%) although 38% of the respondents were unfamiliar with CDSS or their use in medical practice. Most (79%) described themselves as receptive to new decision support tools, though the most commonly anticipated barrier to implementation was physician acceptance. CONCLUSIONS: CDSS are used in limited capacity in Wisconsin and existing systems are not likely to be computer-based. Despite physicians expressing a generally favorable interest in CDSS, a knowledge gap persists.

Learning to use learning resources during medical school and residency.

OBJECTIVES: This study (1) examined the natural history of learning to use learning resources by medical students and residents and (2) considered whether that history is consistent with the ways in which physicians approach their learning tasks. METHODS: The authors conducted and analyzed thirty-two open-ended interviews of first-year and third-year medical students and first-year and senior residents in internal medicine, family medicine, or pediatrics. RESULTS AND DISCUSSION: Learning to use learning resources occurs at the same time as learning done to address instructional and clinical problems that physicians-in-training face, with all kinds of learning following well-documented stages. Skills for using resources are developed gradually and by overcoming barriers such as time constraints and existing habits. CONCLUSIONS: Implications of the natural history of learning to use learning resources can be employed by librarians and medical teachers to facilitate self-directed learning for physicians-in-training. Specific recommendations are provided.