RESUMO
INTRODUCTION: ChatGPT has been tested in many disciplines, but only a few have involved hearing diagnosis and none to physiology or audiology more generally. The consistency of the chatbot's responses to the same question posed multiple times has not been well investigated either. This study aimed to assess the accuracy and repeatability of ChatGPT 3.5 and 4 on test questions concerning objective measures of hearing. Of particular interest was the short-term repeatability of responses which was here tested on four separate days extended over one week. METHODS: We used 30 single-answer, multiple-choice exam questions from a one-year course on objective methods of testing hearing. The questions were posed five times to both ChatGPT 3.5 (the free version) and ChatGPT 4 (the paid version) on each of four days (two days one week and two days the following week). The accuracy of the responses was evaluated in terms of a response key. To evaluate the repeatability of the responses over time, percent agreement and Cohen's Kappa were calculated. Results: The overall accuracy of ChatGPT 3.5 was 48-49%, while that of ChatGPT 4 was 65-69%. ChatGPT 3.5 consistently failed to pass the threshold of 50% correct responses. Within a single day, the percent agreement was 76-79% for ChatGPT 3.5 and 87-88% for ChatGPT 4 (Cohen's Kappa 0.67-0.71 and 0.81-0.84 respectively). The percent agreement between responses from different days was 75-79% for ChatGPT 3.5 and 85-88% for ChatGPT 4 (Cohen's Kappa 0.65-0.69 and 0.80-0.85 respectively). CONCLUSION: ChatGPT 4 outperforms ChatGPT 3.5 both in accuracy and higher repeatability over time. However, the great variability of the responses casts doubt on possible professional applications of both versions.
RESUMO
INTRODUCTION AND OBJECTIVE: Hearing disorders among school-age children are a current concern. Continuing studies have been performed in Poland since 2008, and on 2 December 2011 the EU Council adopted Conclusions on the Early Detection and Treatment of Communication Disorders in Children, Including the Use of e-Health Tools and innovative Solutions. The discussion now focuses not only on the efficacy of hearing screening programmes in schoolchildren, but what should be its general aim and what tests it should include? This paper makes the case that it is important to include central auditory processing disorder (CAPD) tests. One such test is the dichotic digits test (DDT). The aim of the presented study was to evaluate the usefulness of the DDT in detecting central hearing disorders in school-age children. MATERIALS AND METHODS: During hearing screening programmes conducted in Poland in 2008-2010, exactly 235,664 children (7-12-years-old) were screened in 9,325 schools. Of this number, 7,642 were examined using the DDT test for CAPD. Screening programmes were conducted using the Sense Examination Platform. RESULTS: With the cut-off criterion set at the 5th percentile, results for the DDT applied in a divided attention mode were 11.4% positive for 7-year-olds and 11.3% for 12-year-olds. In the focused attention mode, the comparable result for 12-year-olds was 9.7%. There was a clear right ear advantage. In children with positive DDT results, a higher incidence of other disorders, such as dyslexia, was observed. CONCLUSIONS: A test for CAPD should be included in the hearing screening of school-age children. The results of this study form the basis for developing Polish standards in this area.
