"PLEASE Complete Your Evaluations!" Strategies to Engage Faculty in Competency-Based Assessments.

Competency-based assessments (CBAs) have gained traction in graduate medical education and inform important learner outcomes through the continuum of medical training. Active participation in new CBAs presents challenges to faculty working in a busy clinical environment. As such, the implementation of new CBAs can be approached with intention to foster acceptance and engagement with new evaluations. This paper describes strategies utilized to implement CBAs among clinician educators during a national assessment pilot. Our methods are grounded in educational, psychological, business, ecological, communication, and information technology theory. Our primary interventions included creating a multilevel vision, engaging a dedicated work group, incorporating quality improvement methodology, and integrating technology to successfully implement the assessments. These practical and effective interventions may also be applied to the implementation of other educational innovations.

The Emergence and Spread of Practice-Based Medical Education Research Networks.

Educational research networks leverage shared goals and common infrastructure to overcome traditional barriers to medical education research, including small sample sizes, lack of generalizability, need for expertise in statistical analysis, and limitations on data sharing. The diversity of extant network models today is exciting and provides a set of common options and challenges that newly emerging networks can expect. These include decisions about network focus, organization of data, sampling strategies, funding, and governance. Common challenges include managing authorship, human subjects protection rules, data use agreements, and statistical disclosure control. Medical education research networks both advance the field and develop the researchers who participate in them. The authors repeat the call that they and others have made for the development of networks to promulgate best practices and coordinate multinetwork (multinational, multispecialty, and cross-curriculum) studies.

Reported Pediatrics Milestones (Mostly) Measure Program, Not Learner Performance.

PURPOSE: Semiannually, U.S. pediatrics residency programs report resident milestone levels to the Accreditation Council for Graduate Medical Education (ACGME). The Pediatrics Milestones Assessment Collaborative (PMAC, consisting of the National Board of Medical Examiners, American Board of Pediatrics, and Association of Pediatric Program Directors) developed workplace-based assessments of 2 inferences: readiness to serve as an intern with a supervisor present (D1) and readiness to care for patients with a supervisor nearby in the pediatric inpatient setting (D2). The authors compared learner and program variance in PMAC scores with ACGME milestones. METHOD: The authors examined sources of variance in PMAC scores and milestones between November 2015 and May 2017 of 181 interns at 8 U.S. pediatrics residency programs using random effects models with program, competency, learner, and program × competency components. RESULTS: Program-related milestone variance was substantial (54% D1, 68% D2), both in comparison to learner milestone variance (22% D1, 14% D2) and program variance in the PMAC scores (12% D1, 10% D2). In contrast, learner variance represented 44% (D1) or 26% (D2) of variance in PMAC scores. Within programs, PMAC scores were positively correlated with milestones for all but one competency. CONCLUSIONS: PMAC assessments provided scores with little program-specific variance and were more sensitive to differences in learners within programs compared with milestones. Milestones reflected greater differences by program than by learner. This may represent program-based differences in intern performance or in use of milestones as a reporting scale. Comparing individual learner milestones without adjusting for programs is problematic.

ARAF recurrent mutation causes central conducting lymphatic anomaly treatable with a MEK inhibitor.

The treatment of lymphatic anomaly, a rare devastating disease spectrum of mostly unknown etiologies, depends on the patient manifestations1. Identifying the causal genes will allow for developing affordable therapies in keeping with precision medicine implementation2. Here we identified a recurrent gain-of-function ARAF mutation (c.640T>C:p.S214P) in a 12-year-old boy with advanced anomalous lymphatic disease unresponsive to conventional sirolimus therapy and in another, unrelated, adult patient. The mutation led to loss of a conserved phosphorylation site. Cells transduced with ARAF-S214P showed elevated ERK1/2 activity, enhanced lymphangiogenic capacity, and disassembly of actin skeleton and VE-cadherin junctions, which were rescued using the MEK inhibitor trametinib. The functional relevance of the mutation was also validated by recreating a lymphatic phenotype in a zebrafish model, with rescue of the anomalous phenotype using a MEK inhibitor. Subsequent therapy of the lead proband with a MEK inhibitor led to dramatic clinical improvement, with remodeling of the patient's lymphatic system with resolution of the lymphatic edema, marked improvement in his pulmonary function tests, cessation of supplemental oxygen requirements and near normalization of daily activities. Our results provide a representative demonstration of how knowledge of genetic classification and mechanistic understanding guides biologically based medical treatments, which in our instance was life-saving.

Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth From Ten Pediatrics Residency Programs.

PURPOSE: This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workload-unadjusted versus workload-adjusted scores. METHOD: Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workload-adjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time. RESULTS: On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workload-unadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time. CONCLUSIONS: Incorporating a measure of observer-reported workload could improve the measurement properties and the ability to interpret WBA scores.

Pathogenic variant in EPHB4 results in central conducting lymphatic anomaly.

Central conducting lymphatic anomaly (CCLA) is one of the complex lymphatic anomalies characterized by dilated lymphatic channels, lymphatic channel dysmotility and distal obstruction affecting lymphatic drainage. We performed whole exome sequencing (WES) of DNA from a four-generation pedigree and examined the consequences of the variant by transfection of mammalian cells and morpholino and rescue studies in zebrafish. WES revealed a heterozygous mutation in EPHB4 (RefSeq NM_004444.4; c.2334 + 1G>C) and RNA-Seq demonstrated that the EPHB4 mutation destroys the normal donor site, which leads to the use of a cryptic splice donor that results in retention of the intervening 12-bp intron sequence. Transient co-expression of the wild-type and mutant EPHB4 proteins showed reduced phosphorylation of tyrosine, consistent with a loss-of-function effect. Zebrafish ephb4a morpholino resulted in vessel misbranching and deformities in the lymphatic vessel development, indicative of possible differentiation defects in lymphatic vessels, mimicking the lymphatic presentations of the patients. Immunoblot analysis using zebrafish lysates demonstrated over-activation of mTORC1 as a consequence of reduced EPHB4 signaling. Strikingly, drugs that inhibit mTOR signaling or RAS-MAPK signaling effectively rescued the misbranching phenotype in a comparable manner. Moreover, knock-in of EPHB4 mutation in HEK293T cells also induced mTORC1 activity. Our data demonstrate the pathogenicity of the identified EPHB4 mutation as a novel cause of CCLA and suggesting that ERK inhibitors may have therapeutic benefits in such patients with complex lymphatic anomalies.

A novel workplace-based assessment for competency-based decisions and learner feedback.

BACKGROUND: Increased recognition of the importance of competency-based education and assessment has led to the need for practical and reliable methods to assess relevant skills in the workplace. METHODS: A novel milestone-based workplace assessment system was implemented in 15 pediatrics residency programs. The system provided: (1) web-based multisource feedback (MSF) and structured clinical observation (SCO) instruments that could be completed on any computer or mobile device; and (2) monthly feedback reports that included competency-level scores and recommendations for improvement. RESULTS: For the final instruments, an average of five MSF and 3.7 SCO assessment instruments were completed for each of 292 interns; instruments required an average of 4-8 min to complete. Generalizability coefficients >0.80 were attainable with six MSF observations. Users indicated that the new system added value to their existing assessment program; the need to complete the local assessments in addition to the new assessments was identified as a burden of the overall process. CONCLUSIONS: Outcomes - including high participation rates and high reliability compared to what has traditionally been found with workplace-based assessment - provide evidence for the validity of scores resulting from this novel competency-based assessment system. The development of this assessment model is generalizable to other specialties.

Validity Evidence From Ratings of Pediatric Interns and Subinterns on a Subset of Pediatric Milestones.

PURPOSE: To investigate evidence for validity of faculty members' pediatric milestone (PM) ratings of interns (first-year residents) and subinterns (fourth-year medical students) on nine subcompetencies related to readiness to serve as a pediatric intern in the inpatient setting. METHOD: The Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN) and the National Board of Medical Examiners collaborated to investigate the utility of assessments of the PMs for trainees' performance. Data from 32 subinterns and 179 interns at 17 programs were collected from July 2012 through April 2013. Observers used several tools to assess learners. At each site, a faculty member used these data to make judgments about the learner's current developmental milestone in each subcompetency. Linear mixed models were fitted to milestone judgments to examine their relationship with learner's rank and subcompetency. RESULTS: On a 5-point developmental scale, mean milestone levels for interns ranged from 3.20 (for the subcompetency Work effectively as a member of a team) to 3.72 (Humanism) and for subinterns from 2.89 (Organize and prioritize care) to 3.61 (Professionalization). Mean milestone ratings were significantly higher for the Professionalism competency (3.59-3.72) for all trainees compared with Patient Care (2.89-3.24) and Personal and Professional Development (3.33-3.51). Mean intern ratings were significantly higher than mean subintern ratings for all nine subcompetencies except Professionalization, Humanism, and Trustworthiness. CONCLUSIONS: The PMs had a coherent internal structure and could distinguish between differing levels of trainees, which supports their validation for documenting developmental progression of pediatric trainees.

The Pediatrics Milestones Assessment Pilot: Development of Workplace-Based Assessment Content, Instruments, and Processes.

PURPOSE: To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). METHOD: One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners' readiness to serve as pediatric interns in the inpatient setting. During the 2012-2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. RESULTS: Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback "helped me understand how those with whom I work perceive my performance," and 85% (117/137) agreed they were "useful for constructing future goals or identifying a developmental path." Site leads identified several benefits and challenges to the assessment process. CONCLUSIONS: PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.

Medical education practice-based research networks: Facilitating collaborative research.

BACKGROUND: Research networks formalize and institutionalize multi-site collaborations by establishing an infrastructure that enables network members to participate in research, propose new studies, and exploit study data to move the field forward. Although practice-based clinical research networks are now widespread, medical education research networks are rapidly emerging. AIMS: In this article, we offer a definition of the medical education practice-based research network, a brief description of networks in existence in July 2014 and their features, and a more detailed case study of the emergence and early growth of one such network, the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN). METHODS: We searched for extant networks through peer-reviewed literature and the world-wide web. RESULTS: We identified 15 research networks in medical education founded since 2002 with membership ranging from 8 to 120 programs. Most focus on graduate medical education in primary care or emergency medicine specialties. CONCLUSIONS: We offer four recommendations for the further development and spread of medical education research networks: increasing faculty development, obtaining central resources, studying networks themselves, and developing networks of networks.

An online evidence based medicine exercise prompts reflection in third year medical students.

BACKGROUND: Reflective practice is a desirable trait in physicians, yet there is little information about how it is taught to or learned by medical students. The purpose of this study was to determine whether an online Evidence Based Medicine (EBM) exercise with a face-to-face debriefing session would prompt third year medical students to reflect on their current skills and lead them to further reflection on clinical decision making in the future. METHODS: All third year medical students at the University Of Maryland School Of Medicine who completed their pediatrics clerkship between 7/1/09 and 2/11/11 were required to complete the EBM exercise. Following completion each student received a personal report (Learning Profile) of their responses and attended a one hour large group debriefing session. Student responses to a survey following the debriefing sessions were analyzed using a post-test survey design with a single experimental cohort. RESULTS: Ninety-five percent of students completing the debriefing survey indicated that the debriefing session helped them better understand their learning profiles; 68% stated that their profiles allowed them to evaluate themselves and their decisions. Sixty-three percent noted that participating in the exercise and the debrief would lead them to either learn more about EBM and use EBM more in the future or reflect more on their own decision making. CONCLUSIONS: The EBM exercise was a successful way to introduce the concept of reflective practice to third year medical students, and the graphic Learning Profiles were effective instigators of discussion and reflection.

Front-line ordering clinicians: matching workforce to workload.

BACKGROUND: Matching workforce to workload is particularly important in healthcare delivery, where an excess of workload for the available workforce may negatively impact processes and outcomes of patient care and resident learning. Hospitals currently lack a means to measure and match dynamic workload and workforce factors. OBJECTIVES: This article describes our work to develop and obtain consensus for use of an objective tool to dynamically match the front-line ordering clinician (FLOC) workforce to clinical workload in a variety of inpatient settings. METHODS: We undertook development of a tool to represent hospital workload and workforce based on literature reviews, discussions with clinical leadership, and repeated validation sessions. We met with physicians and nurses from every clinical care area of our large, urban children's hospital at least twice. RESULTS: We successfully created a tool in a matrix format that is objective and flexible and can be applied to a variety of settings. We presented the tool in 14 hospital divisions and received widespread acceptance among physician, nursing, and administrative leadership. The hospital uses the tool to identify gaps in FLOC coverage and guide staffing decisions. DISCUSSION: Hospitals can better match workload to workforce if they can define and measure these elements. The Care Model Matrix is a flexible, objective tool that quantifies the multidimensional aspects of workload and workforce. The tool, which uses multiple variables that are easily modifiable, can be adapted to a variety of settings.

The pediatrics milestones: initial evidence for their use as learning road maps for residents.

OBJECTIVE: As the next step in competency-based medical education, the Pediatrics Milestone Project seeks to provide a learner-centered approach to training and assessment. To help accomplish this goal, this study sought to determine how pediatric residents understand, interpret, and respond to the Pediatrics Milestones. METHODS: Cognitive interviews with 48 pediatric residents from all training levels at 2 training programs were conducted. Each participant reviewed one Pediatrics Milestone document (PMD). Eight total Pediatrics Milestones, chosen for their range of complexity, length, competency domain, and primary author, were included in this study. Six residents, 2 from each year of residency training, reviewed each PMD. Interviews were transcribed and coded using inductive methods, and codes were grouped into themes that emerged. RESULTS: Four major themes emerged through coding and analysis: 1) the participants' degree of understanding of the PMDs is sufficient, often deep; 2) the etiology of participants' understanding is rooted in their experiences; 3) there are qualities of the PMD that may contribute to or detract from understanding; and 4) participants apply their understanding by noting the PMD describes a developmental progression that can provide a road map for learning. Additionally, we learned that residents are generally comfortable being placed in the middle of a series of developmental milestones. Two minor themes focusing on interest and practicality were also identified. CONCLUSIONS: This study provides initial evidence for the Pediatrics Milestones as learner-centered documents that can be used for orientation, education, formative feedback, and, ultimately, assessment.

Perspective: beyond counting hours: the importance of supervision, professionalism, transitions of care, and workload in residency training.

The medical education community's conversations about residents' duty hours have long focused solely on the number of those hours. In July 2011, the Accreditation Council for Graduate Medical Education (ACGME) enacted its most recent iteration of standards regarding duty hours. Those standards, as well as a 2008 Institute of Medicine report, look beyond the quantity of duty hours to address their quality as well. Indeed, the majority of the 2011 ACGME standards specify requirements for the qualitative components of residents' working and learning environments, including supervision of residents; professionalism, personal responsibility, and patient safety; transitions of care; and clinical responsibilities (including workload). The authors believe that focusing on these qualitative (rather than quantitative) components of the resident's working and learning environment provides the greatest promise for balancing patient care with resident education, thus optimizing the safety and effectiveness of both. For each of the four qualitative components that the authors discuss (enhancing supervision, nurturing professionalism and personal responsibility, ensuring safe transitions of care, and optimizing workloads and cognitive loads), they offer agendas for faculty development, educational program planning, and research. Thus, the authors call on the medical education community to expand its discussion beyond counting duty hours to focus on these critical issues that ensure quality resident education and patient care and to implement necessary strategies to address them.