Mobile Application to Improve Just-in-Time 2-Way Formative Feedback in Graduate Medical Education.

Background An easy-to-use application to facilitate direct observation and allow for 2-way feedback between residents and faculty is needed. Objective To develop a mobile-based application (app) with the goals of (1) providing just-in-time feedback to residents; (2) improving timeliness of feedback by faculty; and (3) allowing residents to comment on the value of faculty feedback. Methods Fifty-one of 69 (74%) internal medicine (IM) residents and 20 of 25 (80%) IM core faculty participated in the study from July 1, 2020, to December 31, 2021. An iOS app was designed by authors with expertise in medical education and application development to capture entrustable professional activities (EPAs)-based feedback (eg, informed consent) based on direct observation of residents' skills in the workplace. App utilization and narrative feedback characteristics of faculty comments were examined by exporting the data from the database server. The end user satisfaction was examined using a survey instrument. Results Eighty-seven percent of assessments (117 of 134) initiated were fully completed by residents and faculty. Faculty narrative comments were noted in 97% (114 of 117) of completed assessments and 64% (75 of 117) of residents' feedback to the faculty contained narrative comments. Eighty-three percent (97 of 117) of comments were behaviorally specific and 71% (83 of 117) contained an actionable item. Eighty-six percent (18 of 21) of residents and 90% (9 of 10) of core faculty stated that this application promoted an educational interaction between them. Conclusions This app facilitates the efficient completion of EPA-based formative assessments and captures bidirectional feedback in the workplace setting.

Psychometric Properties of Entrustable Professional Activity-Based Objective Structured Clinical Examinations During Transition From Undergraduate to Graduate Medical Education: A Generalizability Study.

PURPOSE: The objective structured clinical examination (OSCE) assesses clinical competence in health sciences education. There is little research regarding the reliability and validity of using an OSCE during the transition from undergraduate to graduate medical education. The goal of this study was to measure the reliability of a unique 2-rater entrustable professional activity (EPA)-based OSCE format for transition to internship using generalizability theory for estimating reliability. METHOD: During the 2018 to 2022 academic years, 5 cohorts of interns (n = 230) at the University of Iowa Hospital and Clinics participated in a 6-station OSCE assessment delivered during orientation. A univariate and multivariate generalizability study (G study) was conducted on the scores generated by the 3 cases in the orientation OSCE that shared the 2-rater format. This analysis was supplemented with an associated decision study (D study). RESULTS: The univariate G study for the cases that used a simulated patient and a faculty rater demonstrated that this OSCE generated a moderately reliable score with 3 cases. The D study showed that increasing the OCSE to 12 cases yielded a mean score reliable enough (G = 0.76) for making high-stakes normative decisions regarding remediation and readiness to practice. The universe score correlation between 2 types of raters was 0.398. The faculty ratings displayed a larger proportion of universe (true) score variance and yielded a more reliable (G = 0.433) score compared with the standardized patient ratings (G = 0.337). CONCLUSIONS: This study provides insight into the development of an EPA-based OSCE. The univariate G study demonstrated that when using the 2 rater types, this assessment could generate a moderately reliable score with 3 cases. The multivariate G study showed that the 2 types of raters assessed different aspects of clinical skills and faculty raters were more reliable.

Pivoting during a Pandemic: Reimagining Intern Orientation EPA-Based OSCE.

OBJECTIVES: Objective structured clinical examinations (OSCEs) provide reliable and standardized means for assessing the performance of specific clinical skills. Our previous experience with entrustable professional activity-based multidisciplinary OSCEs suggests that this exercise offers just-in-time baseline information regarding critical intern skills. The coronavirus disease 2019 pandemic forced medical education programs to reimagine such educational experiences. For the safety of all of the participants, the Internal Medicine and Family Medicine residency programs pivoted from an in-person OSCE to a hybrid model (combination of in-person and virtual encounters) while maintaining the goals of the OSCEs administered in previous years. Here, we describe an innovative hybrid approach to redesigning and implementing the existing OSCE model while maximizing risk mitigation. METHODS: In total, 41 interns from Internal Medicine and Family Medicine participated in the 2020 hybrid OSCE. Five stations allowed for clinical skills assessment. Faculty completed skills checklists with global assessments and simulated patients completed communication checklists with global assessments. Interns, faculty, and simulated patients completed a post-OSCE survey. RESULTS: Informed consent, handoffs, and oral presentation were the lowest performing stations (29.2%, 53.6%, 53.6%, respectively) as assessed by faculty skill checklists. One hundred percent of interns (41/41) indicated that immediate faculty feedback was the most valuable part of the exercise, and all of the participating faculty believed that the format was efficient, allowing ample time to provide feedback and complete checklists. Eighty-nine percent of simulated patients cited they would be willing to participate again if the same assessment were to be held during the pandemic. The limitations of the study included the lack of demonstration of physical examination maneuvers by interns. CONCLUSIONS: A hybrid OSCE to assess interns' baseline skills during orientation using Zoom technology could be delivered successfully and safely during the pandemic without compromising the program's goals or satisfaction.

Longitudinal clinical reasoning theme embedded across four years of a medical school curriculum.

OBJECTIVES: The acquisition of clinical reasoning (CR) skills is essential for future healthcare providers as they advance through their education. There is growing consensus that CR skills should be longitudinally integrated into undergraduate curriculum for acquisition/application of these skills. However, only a minority of schools reported having CR focused teaching sessions, citing a lack of curricular time and faculty expertise as the largest barriers. We describe the design and implementation of this theme and report the effects of its early introduction in Phase One as measured by the Diagnostic Thinking Inventory (DTI). METHODS: The Carver College of Medicine developed and implemented a longitudinal four-year clinical reasoning theme (CRT) with a special emphasis on introducing concepts in the preclinical years (Phase One). Educational strategies used to implement the theme relied on following principles: 1) new skills are best acquired in context of application; 2) contextual learning stimulates transfer of knowledge; and 3) knowledge of pathophysiology is necessary but alone is not sufficient to develop CR skills. A patient-centered CR schema served as the framework for developing the theme. Specific focus areas, pedagogies and assessment strategies were established for each of the three phases. RESULTS: The cohort with CRT demonstrated a significant increase in total DTI score after theme implementation compared to the cohort without. CONCLUSIONS: A formal 4-year longitudinal CR theme is feasible, allowing for integration of pathophysiology, social determinants of health, and clinical skills. Early introduction of CR concepts as assessed by DTI showed improvement in student reasoning skills post-intervention.

Successful implementation of a longitudinal skill-based teaching curriculum for residents.

BACKGROUND: Despite significant teaching responsibilities and national accreditation standards, most residents do not receive adequate instruction in teaching methods. Published reports of residents-as-teachers programs vary from brief one-time exposures to curricula delivered over several months. A majority of interventions described are one or two-day workshops with no clear follow-up or reinforcement of skills. A three-year longitudinal teaching skills curriculum was implemented with these goals: 1) deliver an experiential skill-based teaching curriculum allowing all residents to acquire, practice and implement specific skills; 2) provide spaced skills instruction promoting deliberate practice/reflection; and 3) help residents gain confidence in their teaching skills. METHODS: One hundred percent of internal medicine residents (82/82) participated in the curriculum. Every 10 weeks residents attended a topic-specific experiential skills-based workshop. Each workshop followed the same pedagogy starting with debriefing/reflection on residents' deliberate practice of the previously taught skill and introduction of a new skill followed by skill practice with feedback. Every year, participants completed: 1) assessment of overall confidence in each skill and 2) retrospective pre-post self-assessment. A post-curriculum survey was completed at the end of 3 years. RESULTS: Residents reported improved confidence and self-assessed competence in their teaching skills after the first year of the curriculum which was sustained through the three-year curriculum. The curriculum was well received and valued by residents. CONCLUSIONS: A formal longitudinal, experiential skills-based teaching skills curriculum is feasible and can be delivered to all residents. For meaningful skill acquisition to occur, recurrent continuous skill-based practice with feedback and reflection is important.

Embedding a longitudinal diagnostic reasoning curriculum in a residency program using a bolus/booster approach.

Background Diagnostic reasoning skills are essential to the practice of medicine, yet longitudinal curricula to teach residents and evaluate performance in this area is lacking. We describe a longitudinal diagnostic reasoning curriculum implemented in a university-based internal medicine residency program and self-evaluation assessment of the curriculum's effectiveness. Methods A longitudinal diagnostic reasoning curriculum (bolus/booster) was developed and implemented in the fall of 2015 at the University of Iowa. R1, R2, and R3 cohorts were taught the "bolus" curriculum at the beginning of each academic year followed by a "booster" component to maintain and build upon diagnostic reasoning skills taught during the "bolus" phase. Self-administered diagnostic thinking inventory (DTI) scores were collected in the spring of pre-curriculum (baseline, 2014-2015) and post-curriculum (2016-2017). Results The overall DTI scores improved in the R1 cohort, although statistically significant differences were not seen with R2s and R3s. In the original DTI categories, R1s improved in both flexibility of thinking and structure of thinking, the R2s improved in structure of thinking and the R3s did not improve in either category. R1s showed improvement in three of the four subcategories - data acquisition, problem representation, and hypothesis generation. The R2s improved in the subcategory of problem representation. R3s showed no improvement in any of the subcategories. The R3 cohort had higher mean scores in all categories but this did not reach statistical significance. Conclusions Our program created and successfully implemented a longitudinal diagnostic reasoning curriculum. DTI scores improved after implementation of a new diagnostic reasoning curriculum, particularly in R1 cohort.

Assessing Entrustable Professional Activities Using an Orientation OSCE: Identifying the Gaps.

BACKGROUND: A residency program's intern cohort is comprised of individuals from different medical schools that place varying levels of emphasis on Core Entrustable Professional Activities for Entering Residency (CEPAERs). Program directors have expressed concerns about the preparedness of medical school graduates. Though guiding principles for implementation of the CEPAERs have been published, studies using this framework to assess interns' baseline skills during orientation are limited. OBJECTIVE: A CEPAER-based objective structured clinical examination (OSCE) was implemented with the aims to (1) assess each intern's baseline clinical skills and provide formative feedback; (2) determine an intern's readiness for resident responsibilities; (3) inform individualized education plans; and (4) address identified gaps through curricular change. METHODS: During orientation, all 33 interns from internal medicine (categorical, preliminary, and medicine-psychiatry) participated in the OSCE. Six 20-minute stations evaluated 8 EPAs. Faculty completed a global assessment, and standardized patients completed a communications checklist and global assessment. All interns completed a self-assessment of baseline skills and a post-OSCE survey. RESULTS: Stations assessing handoffs, informed consent, and subjective, objective, assessment, and plan (SOAP) note were the lowest-performing stations. Interns performed lower in skills for which they did not report previous training. Formal instruction was incorporated into didactic sessions for the lowest-performing stations. The majority of interns indicated that the assessment was useful, and immediate feedback was beneficial. CONCLUSIONS: This OSCE during orientation offers just-in-time baseline information regarding interns' critical skills and may lead to individualized feedback as well as continuous curricular improvement.

Through students' eyes: ethical and professional issues identified by third-year medical students during clerkships.

BACKGROUND: Education in ethics and professionalism should reflect the realities medical students encounter in the hospital and clinic. METHOD: We performed content analyses on Case Observation and Assessments (COAs) written by third-year medical students about ethical and professional issues encountered during their internal medicine and paediatrics clinical clerkships. RESULTS: A cohort of 141 third-year medical students wrote 272 COAs. Content analyses identified 35 subcategories of ethical and professional issues within 7 major domains: decisions regarding treatment (31.4%), communication (21.4%), professional duties (18.4%), justice (9.8%), student-specific issues (5.4%), quality of care (3.8%), and miscellaneous (9.8%). CONCLUSIONS: Students encountered a wide variety of ethical and professional issues that can be used to guide pre-clinical and clinical education. Comparison of our findings with results from similar studies suggests that the wording of an assignment (specifying "ethical" issues, "professional" issues, or both) may influence the kinds of issues students identify in their experience-based clinical narratives.

Developing future faculty: a program targeting internal medicine fellows' teaching skills.

INTRODUCTION: The increased demand for clinician-educators in academic medicine necessitates additional training in educational skills to prepare potential candidates for these positions. Although many teaching skills training programs for residents exist, there is a lack of reports in the literature evaluating similar programs during fellowship training. AIM: To describe the implementation and evaluation of a unique program aimed at enhancing educational knowledge and teaching skills for subspecialty medicine fellows and chief residents. SETTING: Fellows as Clinician-Educators (FACE) program is a 1-year program open to fellows (and chief residents) in the Department of Internal Medicine at the University of Iowa. PROGRAM DESCRIPTION: The course involves interactive monthly meetings held throughout the academic year and has provided training to 48 participants across 11 different subspecialty fellowships between 2004 and 2009. PROGRAM EVALUATION: FACE participants completed a 3-station Objective Structured Teaching Examination using standardized learners, which assessed participants' skills in giving feedback, outpatient precepting, and giving a mini-lecture. Based on reviews of station performance by 2 independent raters, fellows demonstrated statistically significant improvement on overall scores for 2 of the 3 cases. Participants self-assessed their knowledge and teaching skills prior to starting and after completing the program. Analyses of participants' retrospective preassessments and postassessments showed improved perceptions of competence after training. CONCLUSION: The FACE program is a well-received intervention that objectively demonstrates improvement in participants' teaching skills. It offers a model approach to meeting important training skills needs of subspecialty medicine fellows and chief residents in a resource-effective manner.

Medical students' use of information resources: is the digital age dawning?

PURPOSE: One of the many challenges clinicians face is applying growing medical knowledge to specific patients; however, there is an information gap between information needs and delivery. Digital information resources could potentially bridge this gap. Because most medical students are exposed to personal computers throughout their education, this study postulated that students may be more comfortable using computer-based information resources within clinical interactions. METHOD: In 2001, the authors monitored second-year medical students' use of a unique digital textbook, UpToDate, as they transitioned from preclinical to clinical years at the University of Iowa Roy J. and Lucille A. Carver College of Medicine. In 2002, at the end of their third year, students were surveyed about their preferred clinical information resources. RESULTS: Medical students rapidly adopted UpToDate as a clinical resource during their clinical clerkship as evidenced by a rapid growth in the electronic textbook's use. One hundred sixteen of a possible 154 students (75%) responded to the survey. More than 85% of respondents identified electronic sources as their primary resource (UpToDate 53%, MDConsult 33%; p <.001 when compared to paper resources). They also reported using the information resources on a daily basis and requiring less than 15 minutes to answer most of their clinical questions. CONCLUSIONS: This study clearly demonstrates that medical students embrace and use electronic information resources much more than has been reported among practicing clinicians. The current generation of students may be the leaders in a medical culture shift from paper to electronic resources.

Is that your final answer? Relationship of changed answers to overall performance on a computer-based medical school course examination.

BACKGROUND: Whether examinees benefit from the opportunity to change answers to examination questions has been discussed widely. PURPOSE: This study was undertaken to document the impact of answer changing on exam performance on a computer-based course examination in a second-year medical school course. METHODS: This study analyzed data from a 2 hour, 80-item computer delivered multiple-choice exam administered to 190 students (166 second-year medical students and 24 physician's assistant students). RESULTS: There was a small but significant net improvement in overall score when answers were changed: one student's score increased by 7 points, 93 increased by 1 to 4 points, and 38 decreased by 1 to 3 points. On average, lower-performing students benefited slightly less than higher-performing students. Students spent more time on questions for which they changed the answers and were more likely to change items that were more difficult. CONCLUSIONS: Students should not be discouraged from changing answers, especially to difficult questions that require careful consideration, although the net effect is quite small.