RESUMO
Large language models (LLMs) have broad potential applications in medicine, such as aiding with education, providing reassurance to patients, and supporting clinical decision-making. However, there is a notable gap in understanding their applicability and performance in the surgical domain and how their performance varies across specialties. This paper aims to evaluate the performance of LLMs in answering surgical questions relevant to clinical practice and to assess how this performance varies across different surgical specialties. We used the MedMCQA dataset, a large-scale multi-choice question-answer (MCQA) dataset consisting of clinical questions across all areas of medicine. We extracted the relevant 23,035 surgical questions and submitted them to the popular LLMs Generative Pre-trained Transformers (GPT)-3.5 and GPT-4 (OpenAI OpCo, LLC, San Francisco, CA). Generative Pre-trained Transformer is a large language model that can generate human-like text by predicting subsequent words in a sentence based on the context of the words that come before it. It is pre-trained on a diverse range of texts and can perform a variety of tasks, such as answering questions, without needing task-specific training. The question-answering accuracy of GPT was calculated and compared between the two models and across surgical specialties. Both GPT-3.5 and GPT-4 achieved accuracies of 53.3% and 64.4%, respectively, on surgical questions, showing a statistically significant difference in performance. When compared to their performance on the full MedMCQA dataset, the two models performed differently: GPT-4 performed worse on surgical questions than on the dataset as a whole, while GPT-3.5 showed the opposite pattern. Significant variations in accuracy were also observed across different surgical specialties, with strong performances in anatomy, vascular, and paediatric surgery and worse performances in orthopaedics, ENT, and neurosurgery. Large language models exhibit promising capabilities in addressing surgical questions, although the variability in their performance between specialties cannot be ignored. The lower performance of the latest GPT-4 model on surgical questions relative to questions across all medicine highlights the need for targeted improvements and continuous updates to ensure relevance and accuracy in surgical applications. Further research and continuous monitoring of LLM performance in surgical domains are crucial to fully harnessing their potential and mitigating the risks of misinformation.
RESUMO
The concept that multisensory observation and drawing can be effective for enhancing anatomy learning is supported by pedagogic research and theory, and theories of drawing. A haptico-visual observation and drawing (HVOD) process has been previously introduced to support understanding of the three-dimensional (3D) spatial form of anatomical structures. The HVOD process involves exploration of 3D anatomy with the combined use of touch and sight, and the simultaneous act of making graphite marks on paper which correspond to the anatomy under observation. Findings from a previous study suggest that HVOD can increase perceptual understanding of anatomy through memorization and recall of the 3D form of observed structures. Here, additional pedagogic and cognitive underpinnings are presented to further demonstrate how and why HVOD can be effective for anatomy learning. Delivery of a HVOD workshop is described as a detailed guide for instructors, and themes arising from a phenomenological study of educator experiences of the HVOD process are presented. Findings indicate that HVOD can provide an engaging approach for the spatial exploration of anatomy within a supportive social learning environment, but also requires modification for effective curricular integration. Consequently, based on the most effective research-informed, theoretical, and logistical elements of art-based approaches in anatomy learning, including the framework provided by the observe-reflect-draw-edit-repeat (ORDER) method, an optimized "ORDER Touch" observation and drawing process has been developed. This is with the aim of providing a widely accessible resource for supporting social learning and 3D spatial understanding of anatomy, in addition to improving specific anatomical knowledge.
