RESUMO
Background: The deployment of OpenAI's ChatGPT-3.5 and its subsequent versions, ChatGPT-4 and ChatGPT-4 With Vision (4V; also known as "GPT-4 Turbo With Vision"), has notably influenced the medical field. Having demonstrated remarkable performance in medical examinations globally, these models show potential for educational applications. However, their effectiveness in non-English contexts, particularly in Chile's medical licensing examinations-a critical step for medical practitioners in Chile-is less explored. This gap highlights the need to evaluate ChatGPT's adaptability to diverse linguistic and cultural contexts. Objective: This study aims to evaluate the performance of ChatGPT versions 3.5, 4, and 4V in the EUNACOM (Examen Único Nacional de Conocimientos de Medicina), a major medical examination in Chile. Methods: Three official practice drills (540 questions) from the University of Chile, mirroring the EUNACOM's structure and difficulty, were used to test ChatGPT versions 3.5, 4, and 4V. The 3 ChatGPT versions were provided 3 attempts for each drill. Responses to questions during each attempt were systematically categorized and analyzed to assess their accuracy rate. Results: All versions of ChatGPT passed the EUNACOM drills. Specifically, versions 4 and 4V outperformed version 3.5, achieving average accuracy rates of 79.32% and 78.83%, respectively, compared to 57.53% for version 3.5 (P<.001). Version 4V, however, did not outperform version 4 (P=.73), despite the additional visual capabilities. We also evaluated ChatGPT's performance in different medical areas of the EUNACOM and found that versions 4 and 4V consistently outperformed version 3.5. Across the different medical areas, version 3.5 displayed the highest accuracy in psychiatry (69.84%), while versions 4 and 4V achieved the highest accuracy in surgery (90.00% and 86.11%, respectively). Versions 3.5 and 4 had the lowest performance in internal medicine (52.74% and 75.62%, respectively), while version 4V had the lowest performance in public health (74.07%). Conclusions: This study reveals ChatGPT's ability to pass the EUNACOM, with distinct proficiencies across versions 3.5, 4, and 4V. Notably, advancements in artificial intelligence (AI) have not significantly led to enhancements in performance on image-based questions. The variations in proficiency across medical fields suggest the need for more nuanced AI training. Additionally, the study underscores the importance of exploring innovative approaches to using AI to augment human cognition and enhance the learning process. Such advancements have the potential to significantly influence medical education, fostering not only knowledge acquisition but also the development of critical thinking and problem-solving skills among health care professionals.
Assuntos
Avaliação Educacional , Licenciamento em Medicina , Feminino , Humanos , Masculino , Chile , Competência Clínica/normas , Avaliação Educacional/métodos , Avaliação Educacional/normasRESUMO
Iniciado el 2001 como EMN (Examen Médico Nacional), el actual EUNACOM (Examen Único Nacional de Conocimientos de Medicina) establecido por ley desde el 2008 surge como respuesta frente al creciente número de Facultades de Medicina y al aumento de la inmigración de médicos extranjeros, con el propósito de contribuir a la medición de la calidad. Pero este es un concepto poco aclarado y se cuestiona que sea conveniente de ser evaluado mediante pruebas estandarizadas por su efecto negativo sobre la formación médica. El actual artículo aborda los problemas presentados desde su aplicación, así como los argumentos en contra del mecanismo, el efecto de experiencias similares y en la realidad chilena, las discusiones pendientes y algunas necesarias modificaciones a su normativa.
Started in 2001 as EMN, the current EUNACOM established by lay since 2008, it comes as an answer to the growing number of Medicine Faculties and the arise of immigrants doctors, with the purpose of contribute to quality measures. But this is a unclear concept and it is questioned its convenience to be evaluated trough standards tests for its negative effect over the medical education. This paper approach the presents problems since its origin, as well as the arguments against the mechanism, the similar experiences effect and in Chilean reality, the outstanding argues and some needed changes to its rules.