Repeatability, reproducibility, and diagnostic accuracy of a commercial large language model (ChatGPT) to perform emergency department triage using the Canadian triage and acuity scale.

PURPOSE: The release of the ChatGPT prototype to the public in November 2022 drastically reduced the barrier to using artificial intelligence by allowing easy access to a large language model with only a simple web interface. One situation where ChatGPT could be useful is in triaging patients arriving to the emergency department. This study aimed to address the research problem: "can emergency physicians use ChatGPT to accurately triage patients using the Canadian Triage and Acuity Scale (CTAS)?". METHODS: Six unique prompts were developed independently by five emergency physicians. An automated script was used to query ChatGPT with each of the 6 prompts combined with 61 validated and previously published patient vignettes. Thirty repetitions of each combination were performed for a total of 10,980 simulated triages. RESULTS: In 99.6% of 10,980 queries, a CTAS score was returned. However, there was considerable variations in results. Repeatability (use of the same prompt repeatedly) was responsible for 21.0% of overall variation. Reproducibility (use of different prompts) was responsible for 4.0% of overall variation. Overall accuracy of ChatGPT to triage simulated patients was 47.5% with a 13.7% under-triage rate and a 38.7% over-triage rate. More extensively detailed text given as a prompt was associated with greater reproducibility, but minimal increase in accuracy. CONCLUSIONS: This study suggests that the current ChatGPT large language model is not sufficient for emergency physicians to triage simulated patients using the Canadian Triage and Acuity Scale due to poor repeatability and accuracy. Medical practitioners should be aware that while ChatGPT can be a valuable tool, it may lack consistency and may frequently provide false information.

ABSTRAIT: OBJECTIF: La sortie du prototype ChatGPT au public en novembre 2022 a considérablement réduit l'obstacle à l'utilisation de l'intelligence artificielle en permettant un accès facile à un grand modèle de langage avec une interface web simple. Une situation où ChatGPT pourrait être utile est de trier les patients qui arrivent au service d'urgence. Cette étude visait à résoudre le problème de la recherche : «Les médecins d'urgence peuvent-ils utiliser ChatGPT pour trier avec précision les patients à l'aide de l'Échelle canadienne de triage et d'acuité (ECTC) ?¼. MéTHODES: Six invites uniques ont été élaborées indépendamment par cinq urgentologues. Un script automatisé a été utilisé pour interroger ChatGPT avec chacune des six invites combinées à 61 vignettes de patients validées et précédemment publiées. Trente répétitions de chaque combinaison ont été réalisées pour un total de 10980 triages simulés. RéSULTATS: Dans 99.6 % des 10980 requêtes, un score CTAS a été obtenu. Cependant, il y a eu des variations considérables dans les résultats. La répétabilité (utilisation répétée de la même invite) était responsable de 21.0 % de la variation globale. La reproductibilité (utilisation de différentes invites) était responsable de 4.0 % de la variation globale. La précision globale de ChatGPT pour le triage des patients simulés était de 47.5 %, avec un taux de sous-triage de 13.7 % et un taux de triage supérieur de 38.7 %. Un texte plus détaillé donné à titre d'invite était associé à une plus grande reproductibilité, mais à une augmentation minimale de la précision. CONCLUSIONS: Cette étude suggère que le modèle actuel de ChatGPT en langage large n'est pas suffisant pour permettre aux médecins d'urgence de trier des patients simulés à l'aide de l'échelle canadienne de triage et d'acuité en raison de la faible répétabilité et de la faible précision. Les médecins doivent être conscients que, bien que ChatGPT puisse être un outil précieux, il peut manquer de cohérence et fournir fréquemment de fausses informations.

Editorial Comments: The 2023 Model Core Content of Disaster Medicine.

The recently published Model Core Content of Disaster Medicine introduces proposed curriculum elements for specialized education and training in Disaster Medicine. This editorial comments on the publishing decision for the manuscript.

Sudden-Onset Disaster Mass-Casualty Incident Response: A Modified Delphi Study on Triage, Prehospital Life Support, and Processes.

The application and provision of prehospital care in disasters and mass-casualty incident response in Europe is currently being explored for opportunities to improve practice. The objective of this translational science study was to align common principles of approach and action and to identify how technology can assist and enhance response. To achieve this objective, the application of a modified Delphi methodology study based on statements derived from key findings of a scoping review was undertaken. This resulted in 18 triage, eight life support and damage control interventions, and 23 process consensus statements. These findings will be utilized in the development of evidence-based prehospital mass-casualty incident response tools and guidelines.

A Qualitative Assessment of Studies Evaluating the Classification Accuracy of Personnel Using START in Disaster Triage: A Scoping Review.

Background: Mass casualty incidents (MCIs) can occur as a consequence of a wide variety of events and often require overwhelming prehospital and emergency support and coordinated emergency response. A variety of disaster triage systems have been developed to assist health care providers in making difficult choices with regards to prioritization of victim treatment. The simple triage and rapid treatment (START) triage system is one of the most widely used triage algorithms; however, the research literature addressing real-world or simulation studies documenting the classification accuracy of personnel using START is lacking. Aims and Objectives: To explore the existing literature related to the current state of knowledge about studies assessing the classification accuracy of the START triage system. Design: Scoping review based on Arksey and O'Malley's methodological framework and narrative synthesis based on methods described by Popay and colleagues were performed. Results: The literature search identified 1,820 citations, of which 32 studies met the inclusion criteria. Thirty were peer-reviewed articles and 28 published in the last 10 years (i.e., 2010 and onward). Primary research studies originated in 13 countries and included 3,706 participants conducting triaging assessments involving 2,950 victims. Included studies consisted of five randomized controlled trials, 17 non-randomized controlled studies, eight descriptive studies, and two mixed-method studies. Simulation techniques, mode of delivery, contextual features, and participants' required skills varied among studies. Overall, there was no consistent reporting of outcomes across studies and results were heterogeneous. Data were extracted from the included studies and categorized into two themes: (1) typology of simulations and (2) START system in MCIs simulations. Each theme contains sub-themes regarding the development of simulation employing START as a system for improving individuals' preparedness. These include types of simulation training, settings, and technologies. Other sub-themes include outcome measures and reference standards. Conclusion: This review demonstrates a variety of factors impacting the development and implementation of simulation to assess characteristics of the START system. To further improve simulation-based assessment of triage systems, we recommend the use of reporting guidelines specifically designed for health care simulation research. In particular, reporting of reference standards and test characteristics need to improve in future studies.

Residents working with Médecins Sans Frontières: training and pilot evaluation.

BACKGROUND: Well-prepared humanitarian workers are now more necessary than ever. Essential to the preparation process are: clearly defined learning objectives, curricula tailored to the nuances of humanitarian settings, simulation-based training, and evaluation. This manuscript describes a training program designed to prepare medical residents for their first field deployment with Médecins Sans Frontières and presents the results of a pilot assessment of its effectiveness. METHODS: The training was jointly developed by the Research Center in Emergency and Disaster Medicine- CRIMEDIM of the Università del Piemonte Orientale, Novara, Italy, and the humanitarian aid organization Médecins Sans Frontières- Italy (MSF-Italy); the following topics were covered: disaster medicine, public health, safety and security, infectious diseases, psychological support, communication, humanitarian law, leadership, and job-specific skills. It used a blended-learning approach consisting of a 3-month distance learning module; 1-week instructor-led coaching; and a field placement with MSF. We assessed its effectiveness using the first three levels of Kirkpatrick's training evaluation model. RESULTS: Eight residents took part in the evaluation. Four were residents in emergency medicine, 3 in anesthesia, and 1 in pediatrics; 3 of them were female and the median age was 31 years. Two residents were deployed in Pakistan, 1 in Afghanistan, 1 in the Democratic Republic of Congo, 1 in Iraq, 2 in Haiti and 1 on board of the MSF Mediterranean search & rescue ship. Mean deployment time was 3 months. The average median score for the overall course was 5 (excellent). There was a significant improvement in post-test multiple choice scores (p = 0.001) and in residents' overall performance scores (P = 0.000001). CONCLUSION: Residents were highly satisfied with the training program and their knowledge and skills improved as a result of participation. TRIAL REGISTRATION: This study was approved by the Institutional Ethics Committee (date 24-02-2016, study code UPO.2015.4.10).

A Randomized Trial of Instructor-Led Training Versus Video Lesson in Training Health Care Providers in Proper Donning and Doffing of Personal Protective Equipment.

OBJECTIVE: This study compared live instructor-led training with video-based instruction in personal protective equipment (PPE) donning and doffing. It assessed the difference in performance between (1) attending 1 instructor-led training session in donning and doffing PPE at 1 month prior to assessment, and (2) watching training videos for 1 month. METHODS: This randomized controlled trial pilot study divided 21 medical students and junior doctors into 2 groups. Control group participants attended 1 instructor-led training session. Video group participants watched training videos demonstrating the same procedures, which they could freely watch again at home. After 1 month, a doctor performed a blind evaluation of performance using checklists. RESULTS: Nineteen participants were assessed after 1 month. The mean donning score was 84.8/100 for the instructor-led group and 88/100 for the video group; mean effect size was 3.2 (95% CI: -7.5 to 9.5). The mean doffing score was 79.1/100 for the instructor-led group and 73.9/100 for the video group; mean effect size was 5.2 (95% CI: -7.6 to 18). CONCLUSION: Our study found no significant difference in donning and doffing scores between instructor-led and video lessons. Video training could be a fast and resource-efficient method of training in PPE donning and doffing in responding to the COVID-19 pandemic.

Intra-operative low-dose ketamine does not reduce the cost of post-operative pain management after surgery: a randomized controlled trial in a low-income country.

BACKGROUND: In developing countries, post-operative pain remains underestimated and undertreated due to economic constraints, lack of awareness and limited resources. In contrast, ketamine is an effective, readily available, easy to use and inexpensive drug frequently used in poor settings. OBJECTIVES: The aim of this study was to explore the overall reduction in the medication treatment cost of acute post-operative pain by adding intra-operative low-dose ketamine to traditional intravenous morphine for surgery in a low-income country. METHODS: A double blind randomized controlled trial with placebo-controlled parallel group was performed in Mulago National Hospital (Uganda). Consenting adults scheduled for elective surgery were randomized into two study arms: Group K received ketamine 0.15mg/kg bolus at induction and a continuous infusion of 0.12 mg/kg/hour till start of skin closure; Group C (control) received normal saline. Both groups received Morphine 0.1 mg/kg IV at debulking. The total medication cost was registered. NRS pain scores and other measurements such vital signs and incidence of major and minor side effects were also recorded. RESULTS: A total of 46 patients were included. Patients' baseline characteristics were comparable in both groups. No statistically significant difference was found between the groups concerning the overall medication cost of post-operative pain management. Pain scores, patients' satisfaction in the first 24 hours after surgery and hospital length of stay were similar in both groups. CONCLUSION: Our results do not support the utilization of intra-operative low dose ketamine as a cost-saving post-operative pain treatment strategy for all types of surgery in low-resource settings.

A novel simulation competition format as an effective instructional tool in post-graduate medical education.

OBJECTIVE: Medical simulation competitions are a growing reality. This study aims at exploring if a novel format of simulation competition (SIMCUP) can be an effective educational format in post-graduate education. DESIGN: We designed a 2-day event that included scientific educational lectures, an orientation to the competition, familiarization with the simulation lab, and competition time. Day 1 was devoted to preliminary rounds and was structured using an Objective Structured Clinical Examination (OSCE)-like system. On day 2, the first four teams advanced to semi-finals and then to finals, which were held using a classical SimWars style. SETTING AND SUBJECTS: A total of 14 four-participant teams participated in the event over two editions (Ed.1 in 2015 and Ed.2 in 2016). INTERVENTIONS: External referees evaluated both technical and non-technical skills for each simulated scenario. Each participant was also administered pre- and post-test questionnaires covering self-perception about the confidence in managing simulated clinical cases, educational effectiveness, satisfaction with the simulation experience, and previous simulation training. MAIN RESULTS: Overall participants found SIMCUP a useful learning experience, rating it 10 [9, 10] and 10 [7.75-10] out of 10 for Ed.1 and Ed.2, respectively. Participants reported, using a 10-point semantic differential scale ranging from "1 - strongly disagree." to "10 - strongly agree," finding both days to be educationally effective: day 1 was rated 9 [7-10] and 9 [8-10] as day 2 was rated 8 [7-10] and 8 [7-10] for Ed. 1 and Ed. 2, respectively.Participants' self-perception regarding the confidence of managing the specific scenarios significantly improved immediately after the event as measured by pre- and post-questionnaires for all stations and during both editions. CONCLUSION: This study suggests that simulation competition can serve as an effective instructional format in residency training.

Prioritization in medical school simulation curriculum development using survey tools and desirability function: a pilot experiment.

BACKGROUND: In Italy, there is no framework of procedural skills that all medical students should be able to perform autonomously at graduation. The study aims at identifying (1) a set of essential procedural skills and (2) which abilities could be potentially taught with simulation. Desirability score was calculated for each procedure to determine the most effective manner to proceed with simulation curriculum development. METHODS: A web poll was conducted at the School of Medicine in Novara, looking at the level of expected and self-perceived competency for common medical procedures. Three groups were enrolled: (1) faculty, (2) junior doctors in their first years of practice, and (3) recently graduated medical students. Level of importance of procedural skills for independent practice expressed by teachers, level of mastery self-perceived by learners (students and junior doctors) and suitability of simulation training for the given technical skills were measured. Desirability function was used to set priorities for future learning. RESULTS: The overall mean expected level of competency for the procedural skills was 7.9/9. Mean level of self reported competency was 4.7/9 for junior doctors and 4.4/9 for recently graduated students. The highest priority skills according to the desirability function were urinary catheter placement, nasogastric tube insertion, and incision and drainage of superficial abscesses. CONCLUSIONS: This study identifies those technical competencies thought by faculty to be important and assessed the junior doctors and recent graduates level of self-perceived confidence in performing these skills. The study also identifies the perceived utility of teaching these skills by simulation. The study prioritizes those skills that have a gap between expected and observed competency and are also thought to be amenable to teaching by simulation. This allows immediate priorities for simulation curriculum development in the most effective manner. This methodology may be useful to researchers in other centers to prioritize simulation training.

A Simple Graphical Method for Quantification of Disaster Management Surge Capacity Using Computer Simulation and Process-control Tools.

INTRODUCTION: Surge capacity, or the ability to manage an extraordinary volume of patients, is fundamental for hospital management of mass-casualty incidents. However, quantification of surge capacity is difficult and no universal standard for its measurement has emerged, nor has a standardized statistical method been advocated. As mass-casualty incidents are rare, simulation may represent a viable alternative to measure surge capacity. Hypothesis/Problem The objective of the current study was to develop a statistical method for the quantification of surge capacity using a combination of computer simulation and simple process-control statistical tools. Length-of-stay (LOS) and patient volume (PV) were used as metrics. The use of this method was then demonstrated on a subsequent computer simulation of an emergency department (ED) response to a mass-casualty incident. METHODS: In the derivation phase, 357 participants in five countries performed 62 computer simulations of an ED response to a mass-casualty incident. Benchmarks for ED response were derived from these simulations, including LOS and PV metrics for triage, bed assignment, physician assessment, and disposition. In the application phase, 13 students of the European Master in Disaster Medicine (EMDM) program completed the same simulation scenario, and the results were compared to the standards obtained in the derivation phase. RESULTS: Patient-volume metrics included number of patients to be triaged, assigned to rooms, assessed by a physician, and disposed. Length-of-stay metrics included median time to triage, room assignment, physician assessment, and disposition. Simple graphical methods were used to compare the application phase group to the derived benchmarks using process-control statistical tools. The group in the application phase failed to meet the indicated standard for LOS from admission to disposition decision. CONCLUSIONS: This study demonstrates how simulation software can be used to derive values for objective benchmarks of ED surge capacity using PV and LOS metrics. These objective metrics can then be applied to other simulation groups using simple graphical process-control tools to provide a numeric measure of surge capacity. Repeated use in simulations of actual EDs may represent a potential means of objectively quantifying disaster management surge capacity. It is hoped that the described statistical method, which is simple and reusable, will be useful for investigators in this field to apply to their own research.

Italian medical students and disaster medicine: awareness and formative needs.

OBJECTIVE: Over the last century, the number of disasters has increased. Many governments and scientific institutions agree that disaster medicine education should be included in the standard medical curriculum. Italian medical students' perceptions of mass casualty incidents and disasters and whether-and if so to what extent-such topics are part of their academic program were investigated. DESIGN, SETTING, AND PARTICIPANTS: A Web-based survey was disseminated to all students registered with the national medical students' association (Segretariato Italiano Studenti Medicina), a member of the International Federation of Medical Students' Associations. The survey consisted of 14 questions divided into four sections. RESULTS: Six hundred thirty-nine medical students completed the survey; 38.7 percent had never heard about disaster medicine; 90.9 percent had never attended elective academic courses on disaster medicine; 87.6 percent had never attended non-academic courses on disaster medicine; 91.4 percent would welcome the introduction of a course on disaster medicine in their core curriculum; and 94.1 percent considered a knowledge of disaster medicine important for their future career. CONCLUSIONS: Most of the students surveyed had never attended courses on disaster medicine during their medical school program. However, respondents would like to increase their knowledge in this area and would welcome the introduction of specific courses into the standard medical curriculum.

Increasing emergency medicine residents' confidence in disaster management: use of an emergency department simulator and an expedited curriculum.

INTRODUCTION: Disaster Medicine is an increasingly important part of medicine. Emergency Medicine residency programs have very high curriculum commitments, and adding Disaster Medicine training to this busy schedule can be difficult. Development of a short Disaster Medicine curriculum that is effective and enjoyable for the participants may be a valuable addition to Emergency Medicine residency training. METHODS: A simulation-based curriculum was developed. The curriculum included four group exercises in which the participants developed a disaster plan for a simulated hospital. This was followed by a disaster simulation using the Disastermed.Ca Emergency Disaster Simulator computer software Version 3.5.2 (Disastermed.Ca, Edmonton, Alberta, Canada) and the disaster plan developed by the participants. Progress was assessed by a pre- and post-test, resident evaluations, faculty evaluation of Command and Control, and markers obtained from the Disastermed.Ca software. RESULTS: Twenty-five residents agreed to partake in the training curriculum. Seventeen completed the simulation. There was no statistically significant difference in pre- and post-test scores. Residents indicated that they felt the curriculum had been useful, and judged it to be preferable to a didactic curriculum. In addition, the residents' confidence in their ability to manage a disaster increased on both a personal and and a departmental level. CONCLUSIONS: A simulation-based model of Disaster Medicine training, requiring approximately eight hours of classroom time, was judged by Emergency Medicine residents to be a valuable component of their medical training, and increased their confidence in personal and departmental disaster management capabilities.

Tornado hazard communication disparities among Spanish-speaking individuals in an English-speaking community.

BACKGROUND: The state of Oklahoma, known for destructive tornados, has a native Spanish-speaking (NSS) population of approximately 180,241, of which 50% report being able to speak English "very well" (US Census Bureau). With almost 50% of these native Spanish-speaking persons being limited English proficient (LEP), their reception of tornado hazard communications may be restricted. This study conducted in northeast Oklahoma (USA) evaluates the association between native language and receiving tornado hazard communications. METHODS: This study was a cross-sectional survey conducted among a convenience sample of NSS and native English-speaking (NES) adults at Xavier Clinic and St. Francis Trauma Emergency Center in Tulsa, OK, USA from September 2009 through December 2009. Of the 82 surveys administered, 80 were returned, with 40 NES and 40 NSS participants. A scoring system (Severe Weather Information Reception (SWIR)) was developed to quantify reception of hazard information among the study participants (1-3 points=poor reception, 4-5=adequate reception, 6-8=excellent reception). Pearson's chi-squared test was used to calculate differences between groups with Yates' continuity correction applied where appropriate, and SWIR scores were analyzed using ANOVA. P-values<.05 were considered significant. RESULTS: NSS fluency in English was 25.6%. No significant association was found between native language and those who watch television, listen to radio, have a National Oceanic and Atmospheric Administration (NOAA) All Hazards radio or telephone, or are in audible range of a tornado siren. NSS were less likely to have Internet access (P<.004), and less likely to know of local telephone warning programs (P<.03). The mean NSS SWIR score was 3.2 (95% CI, 2.8-3.7) while LEP NSS averaged 2.8 (95% CI, 2.4-3.2). The mean NES SWIR score was 4.5 (95% CI, 4.1-5.0). CONCLUSION: Results demonstrate a disparity in tornado warning reception between NSS and NES. Poor English proficiency was noted to be 75% among NSS, which is approximately 25% more than estimated by the US Census Bureau. This study demonstrates a need for emergency managers to recognize when appropriate and overcome communication disparities among limited English proficient populations.

Evaluation of medical management during a mass casualty incident exercise: an objective assessment tool to enhance direct observation.

BACKGROUND: Functional exercises represent an important link between disaster planning and disaster response. Although these exercises are widely performed, no standardized method exists for their evaluation. STUDY OBJECTIVES: To describe a simple and objective method to assess medical performance during functional exercise events. METHODS: An evaluation tool comprising three data fields (triage, clinical maneuvers, and radio usage), accompanied by direct anecdotal observational methods, was used to evaluate a large functional mass casualty incident exercise. RESULTS: Seventeen medical responders managed 112 victims of a simulated building explosion. Although 81% of the patients were assigned the appropriate triage codes, evacuation from the site did not follow in priority. Required maneuvers were performed correctly in 85.2% of airway maneuvers and 78.7% of breathing maneuvers, however, significant under-treatment occurred, possibly due to equipment shortages. Extensive use of radio communication was documented. In evaluating this tool, the structured markers were informative, but further information provided by direct observation was invaluable. CONCLUSION: A three-part tool (triage, medical maneuvers, and radio usage) can provide a method to evaluate functional mass casualty incident exercises, and is easily implemented. For the best results, it should be used in conjunction with direct observation. The evaluation tool has great potential as a reproducible and internationally recognized tool for evaluating disaster management exercises.