Assessment of Implicit Gender Bias During Evaluation of Procedural Competency Among Emergency Medicine Residents.

Importance: Gender disparities exist throughout medicine. Recent studies have highlighted an attainment gap between male and female residents in performance evaluations on Accreditation Council for Graduate Medical Education (ACGME) milestones. Because of difficulties in blinding evaluators to gender, it remains unclear whether these observed disparities are because of implicit bias or other causes. Objective: To estimate the magnitude of implicit gender bias in assessments of procedural competency in emergency medicine residents and whether the gender of the evaluator is associated with identified implicit gender bias. Design, Setting, and Participants: A cross-sectional study was performed from 2018 to 2020 in which emergency medicine residency faculty assessed procedural competency by evaluating videos of residents performing 3 procedures in a simulated environment. They were blinded to the intent of the study. Proceduralists were filmed performing each procedure from 2 different viewpoints simultaneously by 2 different cameras. One was a gender-blinded (ie, hands-only) view, and the other a wide-angled gender-evident (ie, whole-body) view. The faculty evaluators viewed videos in a random order and assessed procedural competency on a global rating scale with extensive validity evidence for the evaluation of video-recorded procedural performance. Main Outcomes and Measures: The primary outcome was to determine if there was a difference in the evaluation of procedural competency based on gender. The secondary outcome was to determine if there was a difference in the evaluations based on the gender of the evaluator. Results: Fifty-one faculty evaluators enrolled from 19 states, with 22 male participants (43.1%), 29 female participants (56.9%), and a mean (SD) age of 37 (6.4) years. Each evaluator assessed all 60 procedures: 30 gender-blinded (hands-only view) videos and 30 identical gender-evident (wide angle) videos. There were no statistically significant differences in the study evaluators' scores of the proceduralists based on their gender, and the gender of the evaluator was not associated with the difference in mean scores. Conclusions and Relevance: In this study, we did not identify a difference in the evaluation of procedural competency based upon the gender of the resident proceduralist or the gender of the faculty evaluator.

Consensus Guidelines for Digital Scholarship in Academic Promotion.

INTRODUCTION: As scholarship moves into the digital sphere, applicant and promotion and tenure (P&T) committee members lack formal guidance on evaluating the impact of digital scholarly work. The P&T process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. As dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship. METHODS: We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document at the Council of Emergency Medicine Residency Directors Academic Assembly on April 1, 2019. RESULTS: We discussed consensus guidelines for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. These recommendations included the following: demonstrate scholarship criteria; provide external evidence of impact; and include digital peer-review roles. As traditional scholarship continues to evolve within the digital realm, academic medicine should adapt how that scholarship is evaluated. P&T committees in academic medicine are at the epicenter for supporting this changing paradigm in scholarship. CONCLUSION: P&T committees can critically appraise the quality and impact of digital scholarship using specific, validated tools. Applicants for appointment and promotion should highlight and prepare their digital scholarship to specifically address quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study.

INTRODUCTION: Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine. METHODS: An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8. RESULTS: A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts. DISCUSSION: Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.

Effect of Interpolated Questions on Podcast Knowledge Acquisition and Retention: A Double-Blind, Multicenter, Randomized Controlled Trial.

STUDY OBJECTIVE: We determine whether interpolated questions in a podcast improve knowledge acquisition and retention. METHODS: This double-blinded controlled trial randomized trainees from 6 emergency medicine programs to listen to 1 of 2 versions of a podcast, produced de novo on the history of hypertension. The versions were identical except that 1 included 5 interpolated questions to highlight educational points (intervention). There were 2 postpodcast assessments, 1 within 48 hours of listening (immediate posttest) and a second 2 to 3 weeks later (retention test), consisting of the same 15 multiple-choice questions, 5 directly related to the interpolated questions and 10 unrelated. The primary outcome was the difference in intergroup mean assessment scores at each point. A sample size was calculated a priori to identify a significant interarm difference. RESULTS: One hundred thirty-seven emergency medicine trainees were randomized to listen to a podcast, 69 without interpolated questions (control) and 68 with them (intervention). Although no significant difference was detected at the immediate posttest, trainees randomized to the interpolated arm had a significantly higher mean score on the retention test, with a 5.6% absolute difference (95% confidence interval 0.2% to 10.9%). For the material covered by the interpolated questions, the intervention arm had significantly higher mean scores on both the immediate posttest and retention tests, with absolute differences of 8.7% (95% confidence interval 1.6% to 15.8%) and 10.1% (95% confidence interval 2.8% to 17.4%), respectively. There was no significant intergroup difference for noninterpolated questions. CONCLUSION: Podcasts containing interpolated questions improved knowledge retention at 2 to 3 weeks, driven by the retention of material covered by the interpolated questions. Although the gains may be modest, educational podcasters may improve knowledge retention by using interpolating questions.

The Challenging Case Conference: A Gamified Approach to Clinical Reasoning in the Video Conference Era.

The development of clinical reasoning abilities is a core competency of emergency medicine (EM) resident education and has historically been accomplished through case conferences and clinical learning. The advent of the SARS-CoV-2 pandemic has fundamentally changed these traditional learning opportunities by causing a nationwide reliance on virtual education environments and reducing the clinical diversity of cases encountered by EM trainees.We propose an innovative case conference that combines low-fidelity simulation with elements of gamification to foster the development of clinical reasoning skills and increase engagement among trainees during a virtual conference. After a team of residents submits a real clinical case that challenged their clinical reasoning abilities, a different team of residents "plays" through a gamified, simulated version of the case live on a video conference call. The case concludes with a facilitated debriefing led by a simulation-trained faculty, where both the resident teams and live virtual audience discuss the challenges of the case. Participants described how the Challenging Case Conference improved their perceptions of their clinical reasoning skills. Audience members reported increased engagement compared to traditional conferences. Participants also reported an unexpected, destigmatizing effect on the discussion of medical errors produced by this exercise. Residency programs could consider implementing a similar case conference as a component of their conference curriculum.

An international, interprofessional investigation of the self-reported podcast listening habits of emergency clinicians: A METRIQ Study.

OBJECTIVES: Podcasts are increasingly being used for medical education. A deeper understanding of usage patterns would inform both producers and researchers of medical podcasts. We aimed to determine how and why podcasts are used by emergency medicine and critical care clinicians. METHODS: An international interprofessional sample (medical students, residents, physicians, nurses, physician assistants, and paramedics) was recruited through direct contact and a multimodal social media (Twitter and Facebook) campaign. Each participant completed a survey outlining how and why they utilize medical podcasts. Recruitment materials included an infographic and study website. RESULTS: 390 participants from 33 countries and 4 professions (medicine, nursing, paramedicine, physician assistant) completed the survey. Participants most frequently listened to medical podcasts to review new literature (75.8%), learn core material (75.1%), and refresh memory (71.8%). The majority (62.6%) were aware of the ability to listen at increased speeds, but most (76.9%) listened at 1.0 x (normal) speed. All but 25 (6.4%) participants concurrently performed other tasks while listening. Driving (72.3%), exercising (39.7%), and completing chores (39.2%) were the most common. A minority of participants used active learning techniques such as pausing, rewinding, and replaying segments of the podcast. Very few listened to podcasts multiple times. CONCLUSIONS: An international cohort of emergency clinicians use medical podcasts predominantly for learning. Their listening habits (rarely employing active learning strategies and frequently performing concurrent tasks) may not support this goal. Further exploration of the impact of these activities on learning from podcasts is warranted.

Does the Medium Matter? Evaluating the Depth of Reflective Writing by Medical Students on Social Media Compared to the Traditional Private Essay Using the REFLECT Rubric.

INTRODUCTION: Social media is a novel medium to host reflective writing (RW) essays, yet its impact on depth of students' reflection is unknown. Shifting reflection on to social platforms offers opportunities for students to engage with their community, yet may leave them feeling vulnerable and less willing to reflect deeply. Using sociomateriality as a conceptual framework, we aimed to compare the depth of reflection in RW samples submitted by medical students in a traditional private essay format to those posted on a secure social media platform. METHODS: Fourth-year medical students submitted a RW essay as part of their emergency medicine clerkship, either in a private essay format (academic year [AY] 2015) or onto a closed, password-protected social media website (AY 2016). Five raters used the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) to score 122 de-identified RW samples (55 private, 67 social media). Average scores on two platforms were compared. Students were also surveyed regarding their comfort with the social media experience. RESULTS: There were no differences in average composite REFLECT scores between the private essay (14.1, 95% confidence interval [CI], 12.0-16.2) and social media (13.7 95% CI, 11.4-16.0) submission formats (t [1,120] = 0.94, p = 0.35). Of the 73% of students who responded to the survey, 72% reported feeling comfortable sharing their personal reflections with peers, and 84% felt comfortable commenting on peers' writing. CONCLUSION: Students generally felt comfortable using social media for shared reflection. The depth of reflection in RW essays was similar between the private and social media submission formats.

What's All the Chatter? A Mixed-Methods Analysis of Emergency Physicians' Tweets.

INTRODUCTION: Twitter is growing in popularity and influence among emergency physicians (EP), with over 2200 self-identified EP users. As Twitter's popularity has increased among EPs so too has its influence. While there has been debate about the value of Twitter as an effective educational delivery tool, little attention has been paid to the nature of the conversation occurring on Twitter. We aim to describe how influential EPs use Twitter by characterizing the language, purpose, frequencies, content, and degree of engagement of their tweets. METHODS: We performed a mixed-methods analysis following a combined content analysis approach. We conducted qualitative and quantitative analyses of a sample of tweets from the 61 most influential EPs on Twitter. We present descriptive tweet characteristics and noteworthy themes. RESULTS: We analyzed 1375 unique tweets from 57 unique users, representing 93% of the influential Twitter EPs. A majority of tweets (1104/1375, 80%) elicited some response in the form of retweets, likes, or replies, demonstrating community engagement. The qualitative analysis identified 15 distinct categories of tweets. CONCLUSION: Influential EPs on Twitter were engaged in largely medical conversations in which most messages generated some form of interaction. They shared resources and opinions while also building social rapport in a community of practice. This data can help EPs make informed decisions about social media engagement.

Which Emergency Medicine Milestone Sub-competencies are Identified Through Narrative Assessments?

INTRODUCTION: Evaluators use assessment data to make judgments on resident performance within the Accreditation Council for Graduate Medical Education (ACGME) milestones framework. While workplace-based narrative assessments (WBNA) offer advantages to rating scales, validity evidence for their use in assessing the milestone sub-competencies is lacking. This study aimed to determine the frequency of sub-competencies assessed through WBNAs in an emergency medicine (EM) residency program. METHODS: We performed a retrospective analysis of WBNAs of postgraduate year (PGY) 2-4 residents. A shared mental model was established by reading and discussing the milestones framework, and we created a guide for coding WBNAs to the milestone sub-competencies in an iterative process. Once inter-rater reliability was satisfactory, raters coded each WBNA to the 23 EM milestone sub-competencies. RESULTS: We analyzed 2517 WBNAs. An average of 2.04 sub-competencies were assessed per WBNA. The sub-competencies most frequently identified were multitasking, medical knowledge, practice-based performance improvement, patient-centered communication, and team management. The sub-competencies least frequently identified were pharmacotherapy, airway management, anesthesia and acute pain management, goal-directed focused ultrasound, wound management, and vascular access. Overall, the frequency with which WBNAs assessed individual sub-competencies was low, with 14 of the 23 sub-competencies being assessed in less than 5% of WBNAs. CONCLUSION: WBNAs identify few milestone sub-competencies. Faculty assessed similar sub-competencies related to interpersonal and communication skills, practice-based learning and improvement, and medical knowledge, while neglecting sub-competencies related to patient care and procedural skills. These findings can help shape faculty development programs designed to improve assessments of specific workplace behaviors and provide more robust data for the summative assessment of residents.

Blockchain Technology: A Data Framework to Improve Validity, Trust, and Accountability of Information Exchange in Health Professions Education.

Health professions educators face multiple challenges, among them the need to adapt educational methods to new technologies. In the last decades, multiple new digital platforms have appeared in the learning arena, including massive open online courses and social-media-based education. The major critique of these novel methods is the lack of the ability to ascertain the origin, validity, and accountability of the knowledge that is created, shared, and acquired. Recently, a novel technology based on secured data storage and transmission, called blockchain, has emerged as a way to generate networks where validity, trust, and accountability can be created. Conceptually, blockchain is an open, public, distributed, and secure digital registry where information transactions are secured and have a clear origin, explicit pathways, and concrete value. Health professions education based on blockchain will potentially allow improved tracking of content and the individuals who create it, quantify educational impact on multiple generations of learners, and build a relative value of educational interventions. Furthermore, institutions adopting blockchain technology would be able to provide certification and credentialing of health care professionals with no intermediaries. There is potential for blockchain to significantly change the future of health professions education and radically transform how patients, professionals, educators, and learners interact around safe, valid, and accountable information.

Does the Flipped Classroom Improve Learning in Graduate Medical Education?

BACKGROUND: The flipped classroom model for didactic education has recently gained popularity in medical education; however, there is a paucity of performance data showing its effectiveness for knowledge gain in graduate medical education. OBJECTIVE: We assessed whether a flipped classroom module improves knowledge gain compared with a standard lecture. METHODS: We conducted a randomized crossover study in 3 emergency medicine residency programs. Participants were randomized to receive a 50-minute lecture from an expert educator on one subject and a flipped classroom module on the other. The flipped classroom included a 20-minute at-home video and 30 minutes of in-class case discussion. The 2 subjects addressed were headache and acute low back pain. A pretest, immediate posttest, and 90-day retention test were given for each subject. RESULTS: Of 82 eligible residents, 73 completed both modules. For the low back pain module, mean test scores were not significantly different between the lecture and flipped classroom formats. For the headache module, there were significant differences in performance for a given test date between the flipped classroom and the lecture format. However, differences between groups were less than 1 of 10 examination items, making it difficult to assign educational importance to the differences. CONCLUSIONS: In this crossover study comparing a single flipped classroom module with a standard lecture, we found mixed statistical results for performance measured by multiple-choice questions. As the differences were small, the flipped classroom and lecture were essentially equivalent.

JGME-ALiEM Hot Topics in Medical Education: Analysis of a Multimodal Online Discussion About Team-Based Learning.

BACKGROUND: Team-based learning (TBL) is an instructional method that is being increasingly incorporated in health professions education, although use in graduate medical education (GME) has been more limited. OBJECTIVE: To curate and describe themes that emerged from a virtual journal club discussion about TBL in GME, held across multiple digital platforms, while also evaluating the use of social media in online academic discussions. METHODS: The Journal of Graduate Medical Education (JGME) and the Academic Life in Emergency Medicine blog facilitated a weeklong, open-access, virtual journal club on the 2015 JGME article "Use of Team-Based Learning Pedagogy for Internal Medicine Ambulatory Resident Teaching." Using 4 stimulus questions (hosted on a blog as a starting framework), we facilitated discussions via the blog, Twitter, and Google Hangouts on Air platforms. We evaluated 2-week web analytics and performed a thematic analysis of the discussion. RESULTS: The virtual journal club reached a large international audience as exemplified by the blog page garnering 685 page views from 241 cities in 42 countries. Our thematic analysis identified 4 domains relevant to TBL in GME: (1) the benefits and barriers to TBL; (2) the design of teams; (3) the role of assessment and peer evaluation; and (4) crowdsourced TBL resources. CONCLUSIONS: The virtual journal club provided a novel forum across multiple social media platforms, engaging authors, content experts, and the health professions education community in a discussion about the importance, impediments to implementation, available resources, and logistics of adopting TBL in GME.

Trends in National Emergency Medicine Conference Didactic Lectures Over a 6-Year Period.

INTRODUCTION: National conference didactic lectures have traditionally featured hour-long lecture-based presentations. However, there is evidence that longer lectures can lead to both decreased attention and retention of information. The authors sought to identify trends in lecture duration, lecture types, and number of speakers at four national emergency medicine (EM) conferences over a 6-year period. METHODS: The authors performed a retrospective analysis of the length, number of speakers, and format of didactic lectures at four different national EM conferences over 6 years. The authors abstracted data from the national academic assemblies for the four largest not-for-profit EM organizations in the United States: American Academy of Emergency Medicine, American College of Emergency Physicians, Council of Emergency Medicine Residency Directors, and Society for Academic Emergency Medicine. RESULTS: There was a significant yearly decrease in the mean lecture lengths for three of the four conferences. There was an increase in the percentage of rapid fire sessions over the preceding 2 years with a corresponding decrease in the percentage of general educational sessions. There was no significant difference in the mean number of speakers per lecture. DISCUSSION: An analysis of 4210 didactic lecture sessions from the annual meetings of four national EM organizations over a 6-year period showed significant decreases in mean lecture length. These findings can help to guide EM continuing medical education conference planning and research.

Ketamine as a first-line treatment for severely agitated emergency department patients.

OBJECTIVE: Emergency physicians often need to control agitated patients who present a danger to themselves and hospital personnel. Commonly used medications have limitations. Our primary objective was to compare the time to a defined reduction in agitation scores for ketamine versus benzodiazepines and haloperidol, alone or in combination. Our secondary objectives were to compare rates of medication redosing, vital sign changes, and adverse events in the different treatment groups. METHODS: We conducted a single-center, prospective, observational study examining agitation levels in acutely agitated emergency department patients between the ages of 18 and 65 who required sedation medication for acute agitation. Providers measured agitation levels on a previously validated 6-point sedation scale at 0-, 5-, 10-, and 15-min after receiving sedation. We also assessed the incidence of adverse events, repeat or rescue medication dosing, and changes in vital signs. RESULTS: 106 patients were enrolled and 98 met eligibility criteria. There was no significant difference between groups in initial agitation scores. Based on agitation scores, more patients in the ketamine group were no longer agitated than the other medication groups at 5-, 10-, and 15-min after receiving medication. Patients receiving ketamine had similar rates of redosing, changes in vital signs, and adverse events to the other groups. CONCLUSION: In highly agitated and violent emergency department patients, significantly fewer patients receiving ketamine as a first line sedating agent were agitated at 5-, 10-, and 15-min. Ketamine appears to be faster at controlling agitation than standard emergency department medications.

A Survey of Emergency Medicine Residents' Use of Educational Podcasts.

INTRODUCTION: Emergency medicine (EM) educational podcasts have become increasingly popular. Residents spend a greater percentage of their time listening to podcasts than they do using other educational materials. Despite this popularity, research into podcasting in the EM context is sparse. We aimed to determine EM residents' consumption habits, optimal podcast preferences, and motivation for listening to EM podcasts. METHODS: We created a survey and emailed it to EM residents at all levels of training at 12 residencies across the United States from September 2015 to June 2016. In addition to demographics, the 20-question voluntary survey asked questions exploring three domains: habits, attention, and motivation. We used descriptive statistics to analyze results. RESULTS: Of the 605 residents invited to participate, 356 (n= 60.3%) completed the survey. The vast majority listen to podcasts at least once a month (88.8%). Two podcasts were the most popular by a wide margin, with 77.8% and 62.1% regularly listening to Emergency Medicine: Reviews and Perspectives (EM:RAP) and the EMCrit Podcast, respectively; 84.6% reported the ideal length of a podcast was less than 30 minutes. Residents reported their motivation to listen to EM podcasts was to "Keep up with current literature" (88.5%) and "Learn EM core content" (70.2%). Of those responding, 72.2% said podcasts change their clinical practice either "somewhat" or "very much." CONCLUSION: The results of this survey study suggest most residents listen to podcasts at least once a month, prefer podcasts less than 30 minutes in length, have several motivations for choosing podcasts, and report that podcasts change their clinical practice.

Who Are the Most Influential Emergency Physicians on Twitter?

INTRODUCTION: Twitter has recently gained popularity in emergency medicine (EM). Opinion leaders on Twitter have significant influence on the conversation and content, yet little is known about these opinion leaders. We aimed to describe a methodology to identify the most influential emergency physicians (EP) on Twitter and present a current list. METHODS: We analyzed 2,234 English-language EPs on Twitter from a previously published list of Twitter accounts generated by a snowball sampling technique. Using NodeXL software, we performed a network analysis of these EPs and ranked them on three measures of influence: in-degree centrality, eigenvector centrality, and betweenness centrality. We analyzed the top 100 users in each of these three measures of influence and compiled a list of users found in the top 100 in all three measures. RESULTS: Of the 300 total users identified by one of the measures of influence, there were 142 unique users. Of the 142 unique users, 61 users were in the top 100 on all three measures of influence. We identify these 61 users as the most influential EM Twitter users. CONCLUSION: We both describe a method for identifying the most influential users and provide a list of the 61 most influential EPs on Twitter as of January 1, 2016. This application of network science to the EM Twitter community can guide future research to better understand the networked global community of EM.