Evaluating prompt engineering on GPT-3.5's performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4.

This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Prospective Observational Study on the Prevalence and Diagnostic Value of General Practitioners' Gut Feelings for Cancer and Serious Diseases.

BACKGROUND: General practitioners (GPs) have recognized the presence of gut feelings in their diagnostic process. However, little is known about the frequency or determinants of gut feelings or the diagnostic value of gut feelings for cancer and other serious diseases. OBJECTIVE: To assess the prevalence of gut feelings in general practice, examine their determinants and impact on patient management, and measure their diagnostic value for cancer and other serious diseases. DESIGN: This prospective observational study was performed using the Gut Feelings Questionnaire (GFQ). PARTICIPANTS: Participants included 155 GPs and 1487 of their patients, from four Spanish provinces. MAIN MEASURES: Sociodemographic data from patients and GPs; the reasoning style of GPs; the characteristics of the consultation; the presence and kind of gut feeling; the patient's subsequent contacts with the health system; and new cancer and serious disease diagnoses reported at 2 and 6 months post-consultation. KEY RESULTS: GPs experienced a gut feeling during 97% of the consultations: a sense of reassurance in 75% of consultations and a sense of alarm in 22% of consultations. A sense of alarm was felt at higher frequency given an older patient, the presence of at least one cancer-associated symptom, or a non-urban setting. GPs took diagnostic action more frequently after a sense of alarm. After 2 months, the sense of alarm had a sensitivity of 59% for cancer and other serious diseases (95% CI 47-71), a specificity of 79% (95% CI 77-82), a positive predictive value of 12% (95% CI 9-16), and a negative predictive value of 98% (95% CI 86-98). CONCLUSIONS: Gut feelings are consistently present in primary care medicine, and they play a substantial role in a GP's clinical reasoning and timely diagnosis of serious disease. The sense of alarm must be taken seriously and used to support diagnostic evaluation in patients with a new reason for encounter.

Unravelling the polyphony in clinical reasoning research in medical education.

RATIONALE: Clinical reasoning lies at the heart of medical practice and has a long research tradition. Nevertheless, research is scattered across diverse academic disciplines with different research traditions in a wide range of scientific journals. This polyphony is a source of conceptual confusion. AIMS AND OBJECTIVES: We sought to explore the underlying theoretical assumptions of clinical reasoning aiming to promote a comprehensive conceptual and theoretical understanding of the subject area. In particular, we asked how clinical reasoning is defined and researched and what conceptualizations are relevant to such uses. METHODS: A scoping review of the clinical reasoning literature was undertaken. Using a "snowball" search strategy, the wider scientific literature on clinical reasoning was reviewed in order to clarify the different underlying conceptual assumptions underlying research in clinical reasoning, particularly to the field of medical education. This literature included both medical education, as well as reasoning research in other academic disciplines outside medical education, that is relevant to clinical reasoning. A total of 124 publications were included in the review. RESULTS: A detailed account of the research traditions in clinical reasoning research is presented. In reviewing this research, we identified three main conceptualisations of clinical reasoning: "reasoning as cognitive activity," "reasoning as contextually situated activity," and "reasoning as socially mediated activity." These conceptualisations reflected different theoretical understandings of clinical reasoning. Each conceptualisation was defined by its own set of epistemological assumptions, which we have identified and described. CONCLUSIONS: Our work seeks to bring into awareness implicit assumptions of the ongoing clinical reasoning research and to hopefully open much needed channels of communication between the different research communities involved in clinical reasoning research in the field.

Theories of truth and teaching clinical reasoning and problem solving.

In this paper, we will first discuss two current meta-theories dealing with different, aspects of "truth". The first metatheory conceives of truth in terms of coherence (rationality, consistency): a body of knowledge is true when it contains no inconsistencies and has at least some credibility. The second metatheory conceives of truth as correspondence, i.e., empirical accuracy. The two metatheories supplement each other, but are also incommensurable, i.e., they cannot be expressed in each other's terms, for they employ completely different criteria to establish truth (Englebretsen in Bare facts and naked truths: a new correspondence theory of truth, Routledge, London, 2005). We will discuss both the role of both metatheories in medicine, in particular in medical education in a clinical context. In line with Hammond's view (Med Decis Mak 16(3):281-287, 1996a; Human judgment and social policy: irreducible uncertainty, inevitable error, unavoidable injustice, Oxford University Press, New York, 1996b), we will extend the two metatheories to two forms of competence: coherence competence and correspondence competence, and demonstrate that distinguishing these two forms of competence increases our insights as to the best way to teach undergraduate students clinical problem solving.

Knowledge is not enough to solve the problems - The role of diagnostic knowledge in clinical reasoning activities.

BACKGROUND: Clinical reasoning is a key competence in medicine. There is a lack of knowledge, how non-experts like medical students solve clinical problems. It is known that they have difficulties applying conceptual knowledge to clinical cases, that they lack metacognitive awareness and that higher level cognitive actions correlate with diagnostic accuracy. However, the role of conceptual, strategic, conditional, and metacognitive knowledge for clinical reasoning is unknown. METHODS: Medical students (n = 21) were exposed to three different clinical cases and instructed to use the think-aloud method. The recorded sessions were transcribed and coded with regards to the four different categories of diagnostic knowledge (see above). The transcripts were coded using the frequencies and time-coding of the categories of knowledge. The relationship between the coded data and accuracy of diagnosis was investigated with inferential statistical methods. RESULTS: The use of metacognitive knowledge is correlated with application of conceptual, but not with conditional and strategic knowledge. Furthermore, conceptual and strategic knowledge application is associated with longer time on task. However, in contrast to cognitive action levels the use of different categories of diagnostic knowledge was not associated with better diagnostic accuracy. CONCLUSIONS: The longer case work and the more intense application of conceptual knowledge in individuals with high metacognitive activity may hint towards reduced premature closure as one of the major cognitive causes of errors in medicine. Additionally, for correct case solution the cognitive actions seem to be more important than the diagnostic knowledge categories.

Clinical Correlations as a Tool in Basic Science Medical Education.

Clinical correlations are tools to assist students in associating basic science concepts with a medical application or disease. There are many forms of clinical correlations and many ways to use them in the classroom. Five types of clinical correlations that may be embedded within basic science courses have been identified and described. (1) Correlated examples consist of superficial clinical information or stories accompanying basic science concepts to make the information more interesting and relevant. (2) Interactive learning and demonstrations provide hands-on experiences or the demonstration of a clinical topic. (3) Specialized workshops have an application-based focus, are more specialized than typical laboratory sessions, and range in complexity from basic to advanced. (4) Small-group activities require groups of students, guided by faculty, to solve simple problems that relate basic science information to clinical topics. (5) Course-centered problem solving is a more advanced correlation activity than the others and focuses on recognition and treatment of clinical problems to promote clinical reasoning skills. Diverse teaching activities are used in basic science medical education, and those that include clinical relevance promote interest, communication, and collaboration, enhance knowledge retention, and help develop clinical reasoning skills.

Integrating clinical communication with clinical reasoning and the broader medical curriculum.

OBJECTIVE: The objectives of this paper are to discuss the results of a workshop conducted at EACH 2012. Specifically, we will (1) examine the link between communication, clinical reasoning, and medical problem solving, (2) explore strategies for (a) integrating clinical reasoning, medical problem solving, and content from the broader curriculum into clinical communication teaching and (b) integrating communication into the broader curriculum, and (3) discuss benefits gained from such integration. METHODS: Salient features from the workshop were recorded and will be presented here, as well as a case example to illustrate important connections between clinical communication and clinical reasoning. RESULTS: Potential links between clinical communication, clinical reasoning, and medical problem solving as well as strategies to integrate clinical communication teaching and the broader curricula in human and veterinary medicine are enumerated. CONCLUSION: Participants expressed enthusiasm and keen interest in integration of clinical communication teaching and clinical reasoning during this workshop, came to the idea of the interdependence of these skills easily, and embraced the rationale immediately. PRACTICE IMPLICATIONS: Valuing the importance of communication as clinical skill and embracing the interdependence between communication and thought processes related to clinical reasoning and medical problem solving will be beneficial in teaching programs.

Problem perception, technology and effectiveness in medical practice.

RATIONALE, AIMS AND OBJECTIVES: Evidence-based medicine and clinical guidelines have been found difficult to implement in the clinical practice--mainly because lack of evidence quality and guidelines that, generally, do not account for variations in the medical cases. Variation in the medical cases enhances task uncertainty and uncertainty seems to be further enhanced through clinical guidelines. In this article, concept development is attempted, where task uncertainty is classified into a few medical problem-solving processes according to differences in medical technology and in the (initial) perception of the medical problem. Furthermore is argued the need for using different strategies in evaluating performance quality in medical health care depending on the variation in the degree of task uncertainty. METHOD: Qualitative data about medical activities related to certain diseases are used to exemplify problem-solving processes representing different types of task uncertainty. RESULTS: It is argued that the main characteristics of medical problem-solving processes vary according to differences in medical technology and perception of perceived medical problem. Four main medical problem-solving processes are defined and demonstrated through empirical examples. CONCLUSION: What may be regarded as rational behaviour is different for each type of problem-solving processes. Consequently, the processes need different organizational settings and need to be evaluated according to different criteria. Furthermore, from a practical point of view, development and education related to problem perception would seem as important as development of medical technology.