ABSTRACT
The study aimed to evaluate the performance of two Large Language Models (LLMs): ChatGPT (based on GPT-3.5) and GPT-4 with two temperature parameter values, on the Polish Medical Final Examination (MFE). The models were tested on three editions of the MFE from: Spring 2022, Autumn 2022, and Spring 2023 in two language versions-English and Polish. The accuracies of both models were compared and the relationships between the correctness of answers with the answer's metrics were investigated. The study demonstrated that GPT-4 outperformed GPT-3.5 in all three examinations regardless of the language used. GPT-4 achieved mean accuracies of 79.7% for both Polish and English versions, passing all MFE versions. GPT-3.5 had mean accuracies of 54.8% for Polish and 60.3% for English, passing none and 2 of 3 Polish versions for temperature parameter equal to 0 and 1 respectively while passing all English versions regardless of the temperature parameter value. GPT-4 score was mostly lower than the average score of a medical student. There was a statistically significant correlation between the correctness of the answers and the index of difficulty for both models. The overall accuracy of both models was still suboptimal and worse than the average for medical students. This emphasizes the need for further improvements in LLMs before they can be reliably deployed in medical settings. These findings suggest an increasing potential for the usage of LLMs in terms of medical education.
Subject(s)
Benchmarking , Education, Medical , Humans , Poland , Language , Physical ExaminationABSTRACT
It is well documented that body position significantly affects breathing indices during sleep in patients with obstructive sleep apnea. They usually worsen while changing from a non-supine to a supine position. Therefore, body position should be an accurately measured and credible parameter in all types of sleep studies. The aim of this study was to specify the accuracy of a neck-based monitoring device (Clebre, Olsztyn, Poland) mounted at the suprasternal notch, in determining a supine and non-supine sleeping position, as well as specific body positions during sleep, in comparison to polysomnography (PSG). A sleep study (PSG along with a neck-based audio-motion sensor) was performed on 89 consecutive patients. The accuracy in determining supine and non-supine positions was 96.9%±3.9% and 97.0%±3.6%, respectively. For lateral positions, the accuracy was 98.6%±2% and 97.4%±4.5% for the right and left side, respectively. The prone position was detected with an accuracy of 97.3%±5.6%. The study showed a high accuracy in detecting supine, as well as other gross positions, during sleep based on a sensor attached to the suprasternal notch, compared to the PSG study. We feel that the suprasternal notch is a promising area for placing wireless sleep study devices.
ABSTRACT
PURPOSE: Polysomnography (PSG) is considered the best objective study to diagnose and quantify sleep disorders. However, PSG involves multiple electrodes and is usually performed in a sleep laboratory that in itself may change the physiology of sleep. One of the parameters that can change during PSG is the sleep position, leading to more supine sleep. The aim of this study was to quantify the amount of supine sleep during PSG and compare it to consecutive nights of a home sleep apnea test (HSAT) in the same patients. METHODS: This prospective study evaluated 22 consecutive patients undergoing PSG followed by HSAT. Sleep position was analyzed during PSG and subsequently on 2 to 6 nights (mean 3.7 nights) at home, and the amount of supine sleep was recorded during each night. RESULTS: Of 22 patients, there were 12 men (55%). The median age was 60.0 years for women and 45.5 years for men. Median proportion of supine sleep during PSG and HSAT was 61% and 26% (p < 0.001), respectively. Four "phenotypes" were identified according to their sleep position during PSG and HSAT, with 5 patients sleeping mainly supine during all nights, 7 patients sleeping mainly non-supine during all nights, 3 patients sleeping in different positions during each night, and 7 patients sleeping supine during PSG but non-supine at home, during HSAT. CONCLUSIONS: There is a higher proportion of supine sleep during PSG compared to home sleep. We identified a subgroup of patients who slept mainly supine during PSG and mainly non-supine during HSAT. PSG may overestimate OSA severity in a specific phenotype of patients.
