Effects of interacting with a large language model compared with a human coach on the clinical diagnostic process and outcomes among fourth-year medical students: study protocol for a prospective, randomised experiment using patient vignettes.

INTRODUCTION: Versatile large language models (LLMs) have the potential to augment diagnostic decision-making by assisting diagnosticians, thanks to their ability to engage in open-ended, natural conversations and their comprehensive knowledge access. Yet the novelty of LLMs in diagnostic decision-making introduces uncertainties regarding their impact. Clinicians unfamiliar with the use of LLMs in their professional context may rely on general attitudes towards LLMs more broadly, potentially hindering thoughtful use and critical evaluation of their input, leading to either over-reliance and lack of critical thinking or an unwillingness to use LLMs as diagnostic aids. To address these concerns, this study examines the influence on the diagnostic process and outcomes of interacting with an LLM compared with a human coach, and of prior training vs no training for interacting with either of these 'coaches'. Our findings aim to illuminate the potential benefits and risks of employing artificial intelligence (AI) in diagnostic decision-making. METHODS AND ANALYSIS: We are conducting a prospective, randomised experiment with N=158 fourth-year medical students from Charité Medical School, Berlin, Germany. Participants are asked to diagnose patient vignettes after being assigned to either a human coach or ChatGPT and after either training or no training (both between-subject factors). We are specifically collecting data on the effects of using either of these 'coaches' and of additional training on information search, number of hypotheses entertained, diagnostic accuracy and confidence. Statistical methods will include linear mixed effects models. Exploratory analyses of the interaction patterns and attitudes towards AI will also generate more generalisable knowledge about the role of AI in medicine. ETHICS AND DISSEMINATION: The Bern Cantonal Ethics Committee considered the study exempt from full ethical review (BASEC No: Req-2023-01396). All methods will be conducted in accordance with relevant guidelines and regulations. Participation is voluntary and informed consent will be obtained. Results will be published in peer-reviewed scientific medical journals. Authorship will be determined according to the International Committee of Medical Journal Editors guidelines.

Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting.

BACKGROUND: General practitioners (GPs) work in an ill-defined environment where diagnostic errors are prevalent. Previous research indicates that aggregating independent diagnoses can improve diagnostic accuracy in a range of settings. We examined whether aggregating independent diagnoses can also improve diagnostic accuracy for GP decision making. In addition, we investigated the potential benefit of such an approach in combination with a decision support system (DSS). METHODS: We simulated virtual groups using data sets from 2 previously published studies. In study 1, 260 GPs independently diagnosed 9 patient cases in a vignette-based study. In study 2, 30 GPs independently diagnosed 12 patient actors in a patient-facing study. In both data sets, GPs provided diagnoses in a control condition and/or DSS condition(s). Each GP's diagnosis, confidence rating, and years of experience were entered into a computer simulation. Virtual groups of varying sizes (range: 3-9) were created, and different collective intelligence rules (plurality, confidence, and seniority) were applied to determine each group's final diagnosis. Diagnostic accuracy was used as the performance measure. RESULTS: Aggregating independent diagnoses by weighing them equally (i.e., the plurality rule) substantially outperformed average individual accuracy, and this effect increased with increasing group size. Selecting diagnoses based on confidence only led to marginal improvements, while selecting based on seniority reduced accuracy. Combining the plurality rule with a DSS further boosted performance. DISCUSSION: Combining independent diagnoses may substantially improve a GP's diagnostic accuracy and subsequent patient outcomes. This approach did, however, not improve accuracy in all patient cases. Therefore, future work should focus on uncovering the conditions under which collective intelligence is most beneficial in general practice. HIGHLIGHTS: We examined whether aggregating independent diagnoses of GPs can improve diagnostic accuracy.Using data sets of 2 previously published studies, we composed virtual groups of GPs and combined their independent diagnoses using 3 collective intelligence rules (plurality, confidence, and seniority).Aggregating independent diagnoses by weighing them equally substantially outperformed average individual GP accuracy, and this effect increased with increasing group size.Combining independent diagnoses may substantially improve GP's diagnostic accuracy and subsequent patient outcomes.

The inseparability of context and clinical reasoning.

Early descriptions of clinical reasoning have described a dual process model that relies on analytical or nonanalytical approaches to develop a working diagnosis. In this classic research, clinical reasoning is portrayed as an individual-driven cognitive process based on gathering information from the patient encounter, forming mental representations that rely on previous experience and engaging developed patterns to drive working diagnoses and management plans. Indeed, approaches to patient safety, as well as teaching and assessing clinical reasoning focus on the individual clinician, often ignoring the complexity of the system surrounding the diagnostic process. More recent theories and evidence portray clinical reasoning as a dynamic collection of processes that takes place among and between persons across clinical settings. Yet, clinical reasoning, taken as both an individual and a system process, is insufficiently supported by theories of cognition based on individual clinicals and lacks the specificity needed to describe the phenomenology of clinical reasoning. In this review, we reinforce that the modern healthcare ecosystem - with its people, processes and technology - is the context in which health care encounters and clinical reasoning take place.

Exploring objective measures for assessing team performance in healthcare: an interview study.

Introduction: Effective teamwork plays a critical role in achieving high-performance outcomes in healthcare. Consequently, conducting a comprehensive assessment of team performance is essential for providing meaningful feedback during team trainings and enabling comparisons in scientific studies. However, traditional methods like self-reports or behavior observations have limitations such as susceptibility to bias or being resource consuming. To overcome these limitations and gain a more comprehensive understanding of team processes and performance, the assessment of objective measures, such as physiological parameters, can be valuable. These objective measures can complement traditional methods and provide a more holistic view of team performance. The aim of this study was to explore the potential of the use of objective measures for evaluating team performance for research and training purposes. For this, experts in the field of research and medical simulation training were interviewed to gather their opinions, ideas, and concerns regarding this novel approach. Methods: A total of 34 medical and research experts participated in this exploratory qualitative study, engaging in semi-structured interviews. During the interview, experts were asked for (a) their opinion on measuring team performance with objective measures, (b) their ideas concerning potential objective measures suitable for measuring team performance of healthcare teams, and (c) their concerns regarding the use of objective measures for evaluating team performance. During data analysis responses were categorized per question. Results: The findings from the 34 interviews revealed a predominantly positive reception of the idea of utilizing objective measures for evaluating team performance. However, the experts reported limited experience in actively incorporating objective measures into their training and research. Nevertheless, they identified various potential objective measures, including acoustical, visual, physiological, and endocrinological measures and a time layer. Concerns were raised regarding feasibility, complexity, cost, and privacy issues associated with the use of objective measures. Discussion: The study highlights the opportunities and challenges associated with employing objective measures to assess healthcare team performance. It particularly emphasizes the concerns expressed by medical simulation experts and team researchers, providing valuable insights for developers, trainers, researchers, and healthcare professionals involved in the design, planning or utilization of objective measures in team training or research.

How to enrich team research in healthcare by considering five theoretical perspectives.

The aim of this paper is to inspire team research to apply diverse and unconventional perspectives to study team dynamics and performance in healthcare settings. To illustrate that using multiple perspectives can yield valuable insights, we examine a segment of a team interaction during a heart-surgery, using five distinct interdisciplinary perspectives known from small group research: the psychodynamic, functional, conflict-power-status, temporal, and social identity perspectives. We briefly describe each theoretical perspective, discuss its application to study healthcare teams, and present possible research questions for the segment at hand using the respective perspective. We also highlight the benefits and challenges associated with employing these diverse approaches and explore how they can be integrated to analyze team processes in health care. Finally, we offer our own insights and opinions on the integration of these approaches, as well as the types of data required to conduct such analyses. We also point to further research avenues and highlight the benefits associated with employing these diverse approaches. Finally, we offer our own insights and opinions on the integration of these approaches, as well as the types of data required to conduct such analyses.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools.

BACKGROUND: Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive. METHODS: We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests. RESULTS: The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, ηp2 = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, ηp2 = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, ηp2 = 0.002. CONCLUSIONS: Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

Frequency and predictors of unspecific medical diagnoses in the emergency department: a prospective observational study.

BACKGROUND: Misdiagnosis is a major public health problem, causing increased morbidity and mortality. In the busy setting of an emergency department (ED) patients are diagnosed under difficult circumstances. As a consequence, the ED diagnosis at hospital admittance may often be a descriptive diagnosis, such as "decreased general condition". Our objective was to determine in how far patients with such an unspecific ED diagnosis differ from patients with a specific ED diagnosis and whether they experience a worse outcome. METHODS: We conducted a prospective observational study in Bern university hospital in Switzerland for all adult non-trauma patients admitted to any internal medicine ward from August 15th 2015 to December 7th 2015. Unspecific ED diagnoses were defined through the clinical classification software for ICD-10 by two outcome assessors. As outcome parameters, we assessed in-hospital mortality and length of hospital stay. RESULTS: Six hundred eighty six consecutive patients were included. Unspecific diagnoses were identified in 100 (14.6%) of all consultations. Patients receiving an unspecific diagnosis at ED discharge were significantly more often women (56.0% vs. 43.9%, p = 0.024), presented more often with a non-specific complaint (34% vs. 21%, p = 0.004), were less often demonstrating an abnormal heart rate (5.0% vs. 12.5%, p = 0.03), and less often on antibiotics (32.0% vs. 49.0%, p = 0.002). Apart from these, no studied drug intake, laboratory or clinical data including change in diagnosis was associated significantly with an unspecific diagnosis. Unspecific diagnoses were neither associated with in-hospital mortality in multivariable analysis (OR = 1.74, 95% CI: 0.60-5.04; p = 0.305) adjusted for relevant confounders nor with length of hospital stay (GMR = 0.87, 95% CI: 0.23-3.32; p = 0.840). CONCLUSIONS: Women and patients with non-specific presenting complaints and no abnormal heart rate are at risk of receiving unspecific ED diagnoses that do not allow for targeted treatment, discharge and prognosis. This study did not find an effect of such diagnoses on length of hospital stay nor in-hospital mortality.

Stereotypes of experienced health professionals in an interprofessional context: results from a cross-sectional survey in Germany.

Interprofessional teamwork often suffers from the existence of negative stereotypes. To combat their prevalence, interprofessional education (IPE) activities are being implemented worldwide. The aim of this study is to inform IPE developers of the prevalence and content of interprofessional stereotypes in the workplace in Germany and similarly structured healthcare systems. We surveyed health professionals with several years of work experience as nurse, midwife, or therapist concerning their attitudes toward their own professions and those of doctors using an established trait rating measure and a qualitative approach. Stereotypes of respondents (N = 129) were mostly related to (1) academic, medical competence (being perceived as lower than that of doctors) and (2) the traditional role relationship (strict hierarchy, dependence on doctors) that guides a lot of behavior, such as the little participation of nurses, midwives and therapists. Despite profound structural differences in the education and healthcare systems, our analyses further revealed similar topics for further IPE activities as in international research, such as the general demand to convey knowledge about the roles, skills and responsibilities of the other professions. The demand to improve the teamwork skills of all health professionals and empower them to be full and equal members of the healthcare team was also evident. Thus, a more reflective approach to stereotypes and their impact on interprofessional teamwork is indicated, particularly in workplace-based interprofessional learning activities. More generally, to counteract stereotypes a more widespread adoption of IPE in pre and post licensure health professions education and continuing professional development in Germany is needed.

What factors affect team members' evaluation of collaboration in medical teams?

Introduction: Perceived teamwork quality is associated with numerous work-related outcomes, ranging from team effectiveness to job satisfaction. This study explored what situational and stable factors affect the perceived quality of teamwork during a specific team task: when a medical team comprising a senior (supervisor) and a junior (trainee) physician diagnoses a patient. Methods: During a field study in an emergency department, multisource data describing the patients, the diagnosing physicians, and the context were collected, including physicians' ratings of their teamwork. The relationships between perceived teamwork quality and situational (e.g., workload) and stable (e.g., seniority) factors were estimated in a latent regression model using the structural equation modeling (SEM) approach. Results: Across the N = 495 patients included, SEM analyses revealed that the patient-specific case clarity and urgency influenced the perceived teamwork quality positively, whereas the work experience of the supervisor influenced the perceived teamwork quality of both supervisor and trainee negatively, albeit to different degrees. Discussion: Our findings shed light on the complex underpinnings of perceived teamwork quality, a performance-relevant factor that may influence work and organizational effectiveness in healthcare settings.

Differential diagnosis checklists reduce diagnostic error differentially: A randomised experiment.

INTRODUCTION: Wrong and missed diagnoses contribute substantially to medical error. Can a prompt to generate alternative diagnoses (prompt) or a differential diagnosis checklist (DDXC) increase diagnostic accuracy? How do these interventions affect the diagnostic process and self-monitoring? METHODS: Advanced medical students (N = 90) were randomly assigned to one of four conditions to complete six computer-based patient cases: group 1 (prompt) was instructed to write down all diagnoses they considered while acquiring diagnostic test results and to finally rank them. Groups 2 and 3 received the same instruction plus a list of 17 differential diagnoses for the chief complaint of the patient. For half of the cases, the DDXC contained the correct diagnosis (DDXC+), and for the other half, it did not (DDXC-; counterbalanced). Group 4 (control) was only instructed to indicate their final diagnosis. Mixed-effects models were used to analyse results. RESULTS: Students using a DDXC that contained the correct diagnosis had better diagnostic accuracy, mean (standard deviation), 0.75 (0.44), compared to controls without a checklist, 0.49 (0.50), P < 0.001, but those using a DDXC that did not contain the correct diagnosis did slightly worse, 0.43 (0.50), P = 0.602. The number and relevance of diagnostic tests acquired were not affected by condition, nor was self-monitoring. However, participants spent more time on a case in the DDXC-, 4:20 min (2:36), P ≤ 0.001, and DDXC+ condition, 3:52 min (2:09), than in the control condition, 2:59 min (1:44), P ≤ 0.001. DISCUSSION: Being provided a list of possible diagnoses improves diagnostic accuracy compared with a prompt to create a differential diagnosis list, if the provided list contains the correct diagnosis. However, being provided a diagnosis list without the correct diagnosis did not improve and might have slightly reduced diagnostic accuracy. Interventions neither affected information gathering nor self-monitoring.

Effects of using a cognitive aid on content and feasibility of debriefings of simulated emergencies.

Background: Adverse events in patient care are often caused by failures in teamwork. Simulation training and its debriefing can contribute to improving teamwork and thus patient care. When conducting debriefings, there are several design factors that can potentially influence learning outcomes. This study examines the use of a cognitive aid to help structure the content of debriefings and compares it with debriefings that are merely roughly structured. In addition, the feasibility of the debriefing, the satisfaction of the participants and their teamwork during the training are investigated. Methods: In a simulated night shift, seven teams of four to five medical students (n=32) took part in six cases that simulated common situations in an emergency medicine environment and received a debriefing on their teamwork after each case, either in the intervention condition with the help of the TeamTAG tool - a cognitive aid focusing on selected teamwork principles from Crisis Resource Management (CRM) - or in the control condition without it. The facilitators noted the topics of the debriefings and rated their experience of conducting them; the participants indicated their satisfaction with the debriefings, as well as their assessment of the importance of CRM principles. In addition, the quality of teamwork was assessed using the Team Emergency Assessment Measure (TEAM). Results: The analysis showed no difference in the number of teamwork principles discussed between the control and intervention conditions, but topics were repeated more frequently in the control group. The TeamTAG guideline was focused on and implemented by the tutors, who discussed the CRM principles included in the TeamTAG more consistently than in the control condition. The tutors in both conditions were satisfied with the implementation, and the use of TeamTAG facilitated time management. There were no differences in participants' satisfaction, their assessment of the importance of the teamwork principles, or the quality of teamwork between conditions. Conclusion: The use of a cognitive aid can help to direct the focus on certain topics or learning objectives and facilitate time management through pre-structuring; however, a difference in learning outcomes (in terms of the quality of teamwork) could not be identified. Besides the influence of a certain structure or script, a strong influence from the individual guiding the debriefing is likely.

Do different response formats affect how test takers approach a clinical reasoning task? An experimental study on antecedents of diagnostic accuracy using a constructed response and a selected response format.

The use of response formats in assessments of medical knowledge and clinical reasoning continues to be the focus of both research and debate. In this article, we report on an experimental study in which we address the question of how much list-type selected response formats and short-essay type constructed response formats are related to differences in how test takers approach clinical reasoning tasks. The design of this study was informed by a framework developed within cognitive psychology which stresses the importance of the interplay between two components of reasoning-self-monitoring and response inhibition-while solving a task or case. The results presented support the argument that different response formats are related to different processing behavior. Importantly, the pattern of how different factors are related to a correct response in both situations seem to be well in line with contemporary accounts of reasoning. Consequently, we argue that when designing assessments of clinical reasoning, it is crucial to tap into the different facets of this complex and important medical process.

What determines diagnostic resource consumption in emergency medicine: patients, physicians or context?

OBJECTIVES: A major cause for concern about increasing ED visits is that ED care is expensive. Recent research suggests that ED resource consumption is affected by patients' health status, varies between physicians and is context dependent. The aim of this study is to determine the relative proportion of characteristics of the patient, the physician and the context that contribute to ED resource consumption. METHODS: Data on patients, physicians and the context were obtained in a prospective observational cohort study of patients hospitalised to an internal medicine ward through the ED of the University Hospital Bern, Switzerland, between August and December 2015. Diagnostic resource consumption in the ED was modelled through a multilevel mixed effects linear regression. RESULTS: In total, 473 eligible patients seen by one of 38 physicians were included in the study. Diagnostic resource consumption heavily depends on physicians' ratings of case difficulty (p<0.001, z-standardised regression coefficient: 147.5, 95% CI 87.3 to 207.7) and-less surprising-on patients' acuity (p<0.001, 126.0, 95% CI 65.5 to 186.6). Neither the physician per se, nor their experience, the patients' chronic health status or the context seems to have a measurable impact (all p>0.05). CONCLUSIONS: Diagnostic resource consumption in the ED is heavily affected by physicians' situational confidence. Whether we should aim at altering physician confidence ultimately depends on its calibration with accuracy.

An observational study of self-monitoring in ad hoc health care teams.

BACKGROUND: Working in ad hoc teams in a health care environment is frequent but a challenging and complex undertaking. One way for teams to refine their teamwork could be through post-resuscitation reflection and debriefing. However, this would require that teams have insight into the quality of their teamwork. This study investigates (1) the accuracy of the self-monitoring of ad hoc resuscitation teams and their leaders relative to external observations of their teamwork and (2) the relationship of team self-monitoring and external observations to objective performance measures. METHODS: We conducted a quantitative observational study of real-world ad hoc interprofessional teams responding to a simulated cardiac arrest in an emergency room. Teams consisting of residents, consultants, and nurses were confronted with an unexpected, simulated, standardized cardiac arrest situation. Their teamwork was videotaped to allow for subsequent external evaluation on the team emergency assessment measure (TEAM) checklist. In addition, objective performance measures such as time to defibrillation were collected. All participants completed a demographic questionnaire prior to the simulation and a questionnaire tapping their perceptions of teamwork directly after it. RESULTS: 22 teams consisting of 115 health care professionals showed highly variable performance. All performance measures intercorrelated significantly, with the exception of team leaders' evaluations of teamwork, which were not related to any other measures. Neither team size nor cumulative experience were correlated with any measures, but teams led by younger leaders performed better than those led by older ones. CONCLUSION: Team members seem to have better insight into their team's teamwork than team leaders. As a practical consequence, the decision to debrief and the debriefing itself after a resuscitation should be informed by team members, not just leaders.

Self-monitoring accuracy does not increase throughout undergraduate medical education.

CONTEXT: Accurate self-assessment of one's performace on a moment-by-moment basis (ie, accurate self-monitoring) is vital for the self-regulation of practising physicians and indeed for the effective regulation of self-directed learning during medical education. However, little is currently known about the functioning of self-monitoring and its co-development with medical knowledge across medical education. This study is the first to simultaneously investigate a number of relevant aspects and measures that have so far been studied separately: different measures of self-monitoring for a broad area of medical knowledge across 10 different performance levels. METHODS: This study assessed the self-monitoring accuracy of medical students (n = 3145) across 10 semesters. Data collected during the administration of the formative Berlin Progress Test Medicine (PTM) were analysed. The PTM comprises 200 multiple-choice questions covering all major medical disciplines and organ systems. A self-report indicator (ie, confidence) and two behavioural indicators of self-monitoring accuracy (ie, response time and the likelihood of changing an initial answer to a correct rather than an incorrect item) were examined for their development over semesters. RESULTS: Analyses of more than 390 000 observations (of approximately 250 students per semester) showed that confidence was higher for correctly than for incorrectly answered items and that 86% of items answered with high confidence were indeed correct. Response time and the likelihood of the initial answer being changed were higher when the initial answer was incorrect than when it was correct. Contrary to expectations, no differences in self-monitoring accuracy were observed across semesters. CONCLUSIONS: Convergent evidence from different measures of self-monitoring suggests that medical students self-monitor their knowledge on a question-by-question basis well, although not perfectly, and to the same degree as has been found in studies outside medicine. Despite large differences in performance, no variations in self-monitoring across semesters (with the exception of the first semester) were observed.

Institutional strategies related to test-taking behavior in low stakes assessment.

Low stakes assessment without grading the performance of students in educational systems has received increasing attention in recent years. It is used in formative assessments to guide the learning process as well as in large-scales assessments to monitor educational programs. Yet, such assessments suffer from high variation in students' test-taking effort. We aimed to identify institutional strategies related to serious test-taking behavior in low stakes assessment to provide medical schools with practical recommendations on how test-taking effort might be increased. First, we identified strategies that were already used by medical schools to increase the serious test-taking behavior on the low stakes Berlin Progress Test (BPT). Strategies which could be assigned to self-determination theory of Ryan and Deci were chosen for analysis. We conducted the study at nine medical schools in Germany and Austria with a total of 108,140 observations in an established low stakes assessment. A generalized linear-mixed effects model was used to assess the association between institutional strategies and the odds that students will take the BPT seriously. Overall, two institutional strategies were found to be positively related to more serious test-taking behavior: discussing low test performance with the mentor and consequences for not participating. Giving choice was negatively related to more serious test-taking behavior. At medical schools that presented the BPT as evaluation, this effect was larger in comparison to medical schools that presented the BPT as assessment.