Investigating the validity of web-enabled mechanistic case diagramming scores to assess students' integration of foundational and clinical sciences.

As medical schools have changed their curricula to address foundational and clinical sciences in a more integrated fashion, teaching methods such as concept mapping have been incorporated in small group learning settings. Methods that can assess students' ability to apply such integrated knowledge are not as developed, however. The purpose of this project was to assess the validity of scores on a focused version of concept maps called mechanistic case diagrams (MCDs), which are hypothesized to enhance existing tools for assessing integrated knowledge that supports clinical reasoning. The data were from the medical school graduating class of 2018 (N = 136 students). In 2014-2015 we implemented a total of 16 case diagrams in case analysis groups within the Mechanisms of Health and Disease (MOHD) strand of the pre-clinical curriculum. These cases were based on topics being taught during the lectures and small group sessions for MOHD. We created an overall score across all 16 cases for each student. We then correlated these scores with performance in the preclinical curriculum [as assessed by overall performance in MOHD integrated foundational basic science courses and overall performance in the Clinical and Professional Skills (CAPS) courses], and standardized licensing exam scores [United States Medical Licensing Exam (USMLE)] Step 1 (following core clerkships) and Step 2 Clinical Knowledge (at the beginning of the fourth year of medical school). MCD scores correlated with students' overall basic science scores (r = .46, p = .0002) and their overall performance in Clinical and Professional Skills courses (r = .49, p < .0001). In addition, they correlated significantly with standardized exam measures, including USMLE Step 1 (r = .33, p ≤ .0001), and USMLE Step 2 CK (r = .39, p < .0001). These results provide preliminary validity evidence that MCDs may be useful in identifying students who have difficulty in integrating foundational and clinical sciences.

Time-variable medical education innovation in context.

BACKGROUND: Medical education is undergoing robust curricular reform with several innovative models emerging. In this study, we examined current trends in 3-year Doctor of Medicine (MD) education and place these programs in context. METHODS: A survey was conducted among Deans of U.S. allopathic medical schools using structured phone interview regarding current availability of a 3-year MD pathway, and/or other variations in curricular innovation, within their institution. Those with 3-year programs answered additional questions. RESULTS: Data from 107 institutions were obtained (75% survey response rate). The most common variation in length of medical education today is the accelerated 3-year pathway. Since 2010, 9 medical schools have introduced parallel 3-year MD programs and another 4 are actively developing such programs. However, the total number of students in 3-year MD tracks remains small (n=199 students, or 0.2% total medical students). Family medicine and general internal medicine are the most common residency programs selected. Benefits of 3-year MD programs generally include reduction in student debt, stability of guaranteed residency positions, and potential for increasing physician numbers in rural/underserved areas. Drawbacks include concern about fatigue/burnout, difficulty in providing guaranteed residency positions, and additional expense in teaching 2 parallel curricula. Four vignettes of alternative innovative and relevant curricular initiatives are also presented in order to place 3-year MD programs in a broader context of medical education reform in the U.S. CONCLUSION: Three-year MD pathways are the most common accelerated alternative available at a small number of medical schools for highly selected students. Long-term evaluation of these programs will be essential to determine if these programs are meeting their goals (e.g., increasing the number of physicians in rural/underserved areas). Benefits and shortcomings of such programs should be carefully examined when considering this approach, or others described, as part of MD curricular options designed to individualize medical education.

Web-Enabled Mechanistic Case Diagramming: A Novel Tool for Assessing Students' Ability to Integrate Foundational and Clinical Sciences.

PROBLEM: As medical schools move from discipline-based courses to more integrated approaches, identifying assessment tools that parallel this change is an important goal. APPROACH: The authors describe the use of test item statistics to assess the reliability and validity of web-enabled mechanistic case diagrams (MCDs) as a potential tool to assess students' ability to integrate basic science and clinical information. Students review a narrative clinical case and construct an MCD using items provided by the case author. Students identify the relationships among underlying risk factors, etiology, pathogenesis and pathophysiology, and the patients' signs and symptoms. They receive one point for each correctly identified link. OUTCOMES: In 2014-2015 and 2015-2016, case diagrams were implemented in consecutive classes of 150 medical students. The alpha reliability coefficient for the overall score, constructed using each student's mean proportion correct across all cases, was 0.82. Discrimination indices for each of the case scores with the overall score ranged from 0.23 to 0.51. In a G study using those students with complete data (n = 251) on all 16 cases, 10% of the variance was true score variance, and systematic case variance was large. Using 16 cases generated a G coefficient (relative score reliability) equal to 0.72 and a Phi equal to 0.65. NEXT STEPS: The next phase of the project will involve deploying MCDs in higher-stakes settings to determine whether similar results can be achieved. Further analyses will determine whether these assessments correlate with other measures of higher-order thinking skills.

Design and rationale of the medical students learning weight management counseling skills (MSWeight) group randomized controlled trial.

Physicians have an important role addressing the obesity epidemic. Lack of adequate teaching to provide weight management counseling (WMC) is cited as a reason for limited treatment. National guidelines have not been translated into an evidence-supported, competency-based curriculum in medical schools. Weight Management Counseling in Medical Schools: A Randomized Controlled Trial (MSWeight) is designed to determine if a multi-modal theoretically-guided WMC educational intervention improves observed counseling skills and secondarily improve perceived skills and self-efficacy among medical students compared to traditional education (TE). Eight U.S. medical schools were pair-matched and randomized in a group randomized controlled trial to evaluate whether a multi-modal education (MME) intervention compared to traditional education (TE) improves observed WMC skills. The MME intervention includes innovative components in years 1-3: a structured web-course; a role play exercise, WebPatientEncounter, and an enhanced outpatient internal medicine or family medicine clerkship. This evidence-supported curriculum uses the 5As framework to guide treatment and incorporates patient-centered counseling to engage the patient. The primary outcome is a comparison of scores on an Objective Structured Clinical Examination (OSCE) WMC case among third year medical students. The secondary outcome compares changes in scores of medical students from their first to third year on an assessment of perceived WMC skills and self-efficacy. MSWeight is the first RCT in medical schools to evaluate whether interventions integrated into the curriculum improve medical students' WMC skills. If this educational approach for teaching WMC is effective, feasible and acceptable it can affect how medical schools integrate WMC teaching into their curriculum.

The Impact of Using Mean Versus Mode When Assessing Resident Competency.

BACKGROUND: The Accreditation Council for Graduate Medical Education Milestone Project was implemented in 2014 to standardize assessments and progression of residents. While it is recommended that milestones not be used as tools for direct assessments of resident competency, many programs have used or adapted milestone tools for this purpose. OBJECTIVE: We sought to explore use of the most frequent milestone level at which a resident was evaluated (ie, the mode), and compared this to the standard practice of using the arithmetic mean for summarizing performance. METHODS: We reviewed all Family Medicine Milestone evaluations from 1 program for the first 2 academic years of milestone implementation. Mean and mode scores were calculated across 24 unique residents, 841 evaluation forms, and 5897 measurements. The proportion of overestimation errors (where the mean is at least 0.5 larger than the mode) and underestimation errors (where the mean is at least 0.5 less than the mode) were then compared across resident training year and subcompetency. RESULTS: For the 24 residents, an estimation error occurred in 175 of 792 of the comparisons (22%). Of these errors, 118 (67%) were overestimation errors. First-year residents accounted for 55% (96 of 175) of all estimation errors. All subcompetencies had some estimation errors, with 6 having greater than 5%. CONCLUSIONS: If the trend for using the milestones as stand-alone assessment tools is to continue, aggregating data by using frequency distributions and mode would be a more stable and appropriate approach given their nominal or, at best, ordinal nature.

Medical Emergency During Flight: A Team-Building Exercise.

INTRODUCTION: While teams are a central component in health care, many professionals who function in them have had little, if any, formal training on how to develop an effective team. Medical educators and trainers have used many different approaches to teach the basic skills and knowledge of team effectiveness and how team members can best interact with each other. To make team training more realistic, experiential exercises have been used. One of the more popular categories of experiential activities is survival exercises in which team members are given a scenario and required to make decisions that ultimately decide whether the team survives the ordeal. METHODS: This activity describes a situation in which a medical professional is traveling on an airliner when a request for medical assistance occurs. Participants can include clinically experienced medical students, residents, fellows, and faculty physicians. The activity can be used as a stand-alone exercise or in conjunction with another team topic, such as communications or decision making. It has also been effective as an icebreaker for teams working together during a workshop. RESULTS: Approximately 100 medical students, residents, and faculty from anesthesia, family medicine, pediatrics, and internal medicine have participated in this activity. It has been very well received and generated a great deal of discussion of both medical knowledge and team-building skills. DISCUSSION: This activity, which can be used to examine team communications, decision making, leadership, and conflict management, is suitable for health care professionals either through intra- or interprofessional training.

An Investigation of the Generalizability of Medical School Grades.

UNLABELLED: Construct/Background: Medical school grades are currently unstandardized, and their level of reliability is unknown. This means their usefulness for reporting on student achievement is also not well documented. This study investigates grade reliability within 1 medical school. APPROACH: Generalizability analyses are conducted on grades awarded. Grades from didactic and clerkship-based courses were treated as 2 levels of a fixed facet within a univariate mixed model. Grades from within the 2 levels (didactic and clerkship) were also entered in a multivariate generalizability study. RESULTS: Grades from didactic courses were shown to produce a highly reliable mean score (G = .79) when averaged over as few as 5 courses. Although the universe score correlation between didactic and clerkship courses was high (r = .80), the clerkship courses required almost twice as many grades to reach a comparable level of reliability. When grades were converted to a Pass/Fail metric, almost all information contained in the grades was lost. CONCLUSIONS: Although it has been suggested that the imprecision of medical school grades precludes their use as a reliable indicator of student achievement, these results suggest otherwise. While it is true that a Pass/Fail system of grading provides very little information about a student's level of performance, a multi-tiered grading system was shown to be a highly reliable indicator of student achievement within the medical school. Although grades awarded during the first 2 didactic years appear to be more reliable than clerkship grades, both yield useful information about student performance within the medical college.

Teaching Medical Students to Help Patients Quit Smoking: Outcomes of a 10-School Randomized Controlled Trial.

BACKGROUND: Early in medical education, physicians must develop competencies needed for tobacco dependence treatment. OBJECTIVE: To assess the effect of a multi-modal tobacco dependence treatment curriculum on medical students' counseling skills. DESIGN: A group-randomized controlled trial (2010-2014) included ten U.S. medical schools that were randomized to receive either multi-modal tobacco treatment education (MME) or traditional tobacco treatment education (TE). SETTING/PARTICIPANTS: Students from the classes of 2012 and 2014 at ten medical schools participated. Students from the class of 2012 (N = 1345) completed objective structured clinical examinations (OSCEs), and 50 % (N = 660) were randomly selected for pre-intervention evaluation. A total of 72.9 % of eligible students (N = 1096) from the class of 2014 completed an OSCE and 69.7 % (N = 1047) completed pre and post surveys. INTERVENTIONS: The MME included a Web-based course, a role-play classroom demonstration, and a clerkship booster session. Clerkship preceptors in MME schools participated in an academic detailing module and were encouraged to be role models for third-year students. MEASUREMENTS: The primary outcome was student tobacco treatment skills using the 5As measured by an objective structured clinical examination (OSCE) scored on a 33-item behavior checklist. Secondary outcomes were student self-reported skills for performing 5As and pharmacotherapy counseling. RESULTS: Although the difference was not statistically significant, MME students completed more tobacco counseling behaviors on the OSCE checklist (mean 8.7 [SE 0.6] vs. mean 8.0 [SE 0.6], p = 0.52) than TE students. Several of the individual Assist and Arrange items were significantly more likely to have been completed by MME students, including suggesting behavioral strategies (11.8 % vs. 4.5 %, p < 0.001) and providing information regarding quitline (21.0 % vs. 3.8 %, p < 0.001). MME students reported higher self-efficacy for Assist, Arrange, and Pharmacotherapy counseling items (ps ≤0.05). LIMITATIONS: Inclusion of only ten schools limits generalizability. CONCLUSIONS: Subsequent interventions should incorporate lessons learned from this first randomized controlled trial of a multi-modal longitudinal tobacco treatment curriculum in multiple U.S. medical schools. NIH Trial Registry Number: NCT01905618.

Dissection and dissection-associated required experiences improve student performance in gross anatomy: Differences among quartiles.

To promote student learning, educational strategies should provide multiple levels of engagement with the subject matter. This study investigated examination data from five first year medical gross anatomy class cohorts (692 students) to determine if enhanced student performance was correlated with learning through dissection in a course that used a rotating dissection schedule coupled with peer teaching and other associated experiences. When students performed two of five weekly dissections for a given unit, their average scores on both laboratory and written examinations tended to increase as compared to when they had completed only one week of dissection (P < 0.01). However, these performance gains differed across the class strata and were related to the amount of dissection completed. Students in the upper quartile (UQS) of the class benefited when they had dissected once (92.8%) or twice (92.4%), and these scores were significantly higher than those attained when learning from peers (90.3%, P < 0.01). Students in the lower quartile (LQS) benefited most from the dissection experiences, where practical examination performance was better (77.8% and 80.5%) than when these students learned material from their peers (73.7%, P < 0.01). Although UQS benefited from dissection, LQS benefited to a greater extent in both the practical and written examinations with dissection. Although limited, these data suggest that dissection, coupled with associated educational activities, is an effective pedagogical strategy for learning. Further investigation is required to evaluate the concomitant benefits of peer teaching that are associated with the dissection experience. Anat Sci Educ 9: 238-246. © 2015 American Association of Anatomists.

Assessing effective teaching: what medical students value when developing evaluation instruments.

PURPOSE: To investigate what criteria medical students would value and use in assessing teaching skills. METHOD: Fourth-year medical students at the University of Iowa Carver College of Medicine enrolled in a teaching elective course are required to design and use an evaluation instrument to assess effective teaching. Each class uses a similar process in developing their instruments. Since the first class in spring 2007, 193 medical students have created 36 different instruments. Three faculty evaluation experts conducted a thematic analysis of the instruments and coded the information according to what was being evaluated and what types of ratings were indicated. The data were submitted to a fourth faculty reviewer, who synthesized the information and adjusted the codes to better capture the data. Common themes and categories were detected. RESULTS: Four themes were identified: content (instructor knowledgeable, teaches at level of learner, practical information), learning environment, teacher personal attributes, and teaching methods. Thirty-two descriptors were distinguished across the 36 instruments. Thirteen descriptors were present in 50% or more of the instruments. The most common rating systems were Likert scales and open comments. CONCLUSIONS: Fourth-year medical students can offer an eclectic resource for evaluating teaching in the classroom and the clinic. Using the descriptors that were identified in greater than 50% of the evaluation instruments will provide effective measures that can be incorporated into medical teacher evaluation instruments.

The integrated curriculum in medical education: AMEE Guide No. 96.

The popularity of the term "integrated curriculum" has grown immensely in medical education over the last two decades, but what does this term mean and how do we go about its design, implementation, and evaluation? Definitions and application of the term vary greatly in the literature, spanning from the integration of content within a single lecture to the integration of a medical school's comprehensive curriculum. Taking into account the integrated curriculum's historic and evolving base of knowledge and theory, its support from many national medical education organizations, and the ever-increasing body of published examples, we deem it necessary to present a guide to review and promote further development of the integrated curriculum movement in medical education with an international perspective. We introduce the history and theory behind integration and provide theoretical models alongside published examples of common variations of an integrated curriculum. In addition, we identify three areas of particular need when developing an ideal integrated curriculum, leading us to propose the use of a new, clarified definition of "integrated curriculum", and offer a review of strategies to evaluate the impact of an integrated curriculum on the learner. This Guide is presented to assist educators in the design, implementation, and evaluation of a thoroughly integrated medical school curriculum.

Do preceptors with more rating experience provide more reliable assessments of medical student performance?

BACKGROUND: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability. PURPOSE: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings. METHODS: Preceptors were divided into two groups based on rater experience. The univariate and multivariate G study designs used were simple rater (r)-nested-within-person (p) [r : p and r(○) : p(•)] models, and in the univariate analysis was applied separately to CEFs completed by high and low experienced raters. RESULTS: The high experienced rater group yielded a substantially higher observed reliability in both the univariate and multivariate analyses. CONCLUSIONS: These results support the hypothesis that high experienced raters produce more reliable ratings of student performance and suggest methods for improving CEF ratings.

Developing future faculty: a program targeting internal medicine fellows' teaching skills.

INTRODUCTION: The increased demand for clinician-educators in academic medicine necessitates additional training in educational skills to prepare potential candidates for these positions. Although many teaching skills training programs for residents exist, there is a lack of reports in the literature evaluating similar programs during fellowship training. AIM: To describe the implementation and evaluation of a unique program aimed at enhancing educational knowledge and teaching skills for subspecialty medicine fellows and chief residents. SETTING: Fellows as Clinician-Educators (FACE) program is a 1-year program open to fellows (and chief residents) in the Department of Internal Medicine at the University of Iowa. PROGRAM DESCRIPTION: The course involves interactive monthly meetings held throughout the academic year and has provided training to 48 participants across 11 different subspecialty fellowships between 2004 and 2009. PROGRAM EVALUATION: FACE participants completed a 3-station Objective Structured Teaching Examination using standardized learners, which assessed participants' skills in giving feedback, outpatient precepting, and giving a mini-lecture. Based on reviews of station performance by 2 independent raters, fellows demonstrated statistically significant improvement on overall scores for 2 of the 3 cases. Participants self-assessed their knowledge and teaching skills prior to starting and after completing the program. Analyses of participants' retrospective preassessments and postassessments showed improved perceptions of competence after training. CONCLUSION: The FACE program is a well-received intervention that objectively demonstrates improvement in participants' teaching skills. It offers a model approach to meeting important training skills needs of subspecialty medicine fellows and chief residents in a resource-effective manner.

Assessing implicit gender bias in Medical Student Performance Evaluations.

For medical schools, the increasing presence of women makes it especially important that potential sources of gender bias be identified and removed from student evaluation methods. Our study looked for patterns of gender bias in adjective data used to inform our Medical Student Performance Evaluations (MSPEs). Multigroup Confirmatory Factor Analysis (CFA) was used to model the latent structure of the adjectives attributed to students (n = 657) and to test for systematic scoring errors by gender. Gender bias was evident in two areas: (a) women were more likely than comparable men to be described as ''compassionate,'' ''sensitive,'' and ''enthusiastic'' and (b) men were more likely than comparable women to be seen as ''quick learners.'' The gender gap in ''quick learner'' attribution grows with increasing student proficiency; men's rate of increase is over twice that of women's. Technical and nontechnical approaches for ameliorating the impact of gender bias on student recommendations are suggested.

Residents' attitudes about the introduction of a surgical skills simulation laboratory.

BACKGROUND: As surgical education programs develop surgical skills laboratories, it will be important to do so in the most efficient, cost-effective manner. METHODS: We distributed a brief written survey to all general surgery residents at the University of Iowa Hospitals and Clinics regarding their perceptions of the usefulness of a surgical skills laboratory in training of both open and laparoscopic techniques. For the initial survey, we used analysis of variance to compare differences across groups. This was followed by a second survey to post-graduate year (PGY)-1 and PGY-2 residents assessing their knowledge, perceived skill, and rank preference of surgical skill sessions. RESULTS: We received 100% responses rate from both surveys. Respondents to the first survey represented all levels of residency (PGY1-PGY5), and the number of respondents per level ranged from 4 to six. Although there was general agreement that surgical skills laboratories were beneficial for both open and laparoscopic procedures and for all levels of training, there was more support for using them to prepare junior residents. In addition, they were seen as especially beneficial for teaching about laparoscopic techniques. Junior residents did not think that residents should be required to demonstrate mastery in the skills laboratory before being allowed to operate, whereas senior residents were neutral about such requirements. A follow-up survey targeted junior level residents (PGY-1 and 2) to assess their perceived skill for various techniques, and their interest in improving skills through simulation. Results showed that perceived skill differed between the two groups, as did attitudes about the priority for different skills. CONCLUSION: As residency programs implement surgical skills laboratories, understanding local opinions about the potential benefits and sequencing may help to design the laboratories for maximal educational benefit.

Defining and describing medical learning communities: results of a national survey.

PURPOSE: To investigate what is meant by learning community in medical education and to identify the most important features of current medical education learning communities. METHOD: After a literature review, the authors surveyed academic deans of all U.S. and Canadian medical schools and colleges (N=124) to identify those that had implemented a learning community. Those with student learning communities (N=18) answered a series of questions about the goals, structure, function, benefits, and challenges of their communities. RESULTS: The most common primary goals included fostering communication among students and faculty; promoting caring, trust, and teamwork; helping students establish academic support networks; and helping students establish social support networks. Most deans said that students remained in the same community for all four years of medical school and that communities were linked to specific faculty and/or peer advisors. For most schools, communities included students from many class years, and participation was mandatory. Curricular purposes included professionalism training, leadership development, and service learning. Almost all schools had social functions related to their communities, and most provided career planning, group mentoring, and personal counseling. CONCLUSIONS: Learning communities in medical education demonstrate diverse approaches to achieving the general goal of enhanced student learning. Medical school leaders considering learning communities should determine the goals they want to accomplish and be open to adopting different approaches based on local needs. Evaluation and effective monitoring of evolution are needed to determine the best approaches for different needs and to assess impact on students and faculty.

Patient-physician communication: informed consent for imaging-guided spinal injections.

PURPOSE: Historically, informed-consent forms have been developed for the purpose of educating patients. However, informed-consent forms can be very difficult to understand. The hypothesis of this study was that a method using diagrams would improve patient-physician communication without increasing the time required to obtain informed consent over the teach-the-teacher method, as well as over current standard informed-consent protocol. METHODS: Ninety-nine of 109 patients undergoing spinal injections agreed to participate and completed this prospective, randomized, controlled study. The patients were randomly assigned to the control group (informed consent obtained in the customary manner at the investigators' institution, with 12 key points of consent and home care discussed conversationally), the teach-the-teacher group (patients had to repeat the 12 key points to the physicians before informed consent was complete), and the diagram group (patients viewed a set of diagrams illustrating the 12 key points before signing the informed-consent form). After the procedure, the patients completed a survey to test knowledge recall, anxiety, and pain during the procedure. RESULTS: Statistically significant results included a lower survey score for the control group, longer time required for the teach-the-teacher group than the control group, and a negative correlation between age and survey score in the teach-the-teacher group. CONCLUSIONS: The diagram method required less time than the teach-the-teacher method, had no negative correlation in survey score results with age, and had improved patient-physician communication over the control group.

Five-Year Experience: Reflective Writing in a Preclinical End-of-Life Care Curriculum.

INTRODUCTION: This paper examines the use of reflective writing in a preclinical end-of-life curriculum including comparison of the role and outcomes of out-of-class (OC) versus in-class (IC) writing. METHODS: Learners were required to complete one-page essays on their experiences and concerns about death and dying after attending a series of end-of-life care lectures. From 2002-2005, essays were completed OC and in 2006 and 2007 essays were completed during the first ten minutes of small group discussion sessions. Essays were collected and analyzed for salient themes. RESULTS: Between 2002-2007, reflection essays were gathered from 829 learners, including 522 OC essays and 307 IC essays. Essay analysis identified four major themes of student concerns related to caring for dying patients, as well as student reactions to specific curricular components and to the use of reflection. IC essays were shorter and less polished than OC essays but utilized a wider variety of formats including poems and bulleted lists. IC essays tended to react to lecture content immediately preceding the writing exercise whereas OC varied in curricular components upon which they focused. OC essays have the advantage of giving learners more time to choose subject matter, whereas IC essays provide a structured time in which to actively reflect. Both formats served as catalysts for small group discussions. DISCUSSION: Writing exercises can effectively provide an important opportunity and motivation for learners to reflect on past experiences and future expectations related to providing end-of-life care.