LLM-based automatic short answer grading in undergraduate medical education.
BMC Med Educ
; 24(1): 1060, 2024 Sep 27.
Article
em En
| MEDLINE
| ID: mdl-39334087
ABSTRACT
BACKGROUND:
Multiple choice questions are heavily used in medical education assessments, but rely on recognition instead of knowledge recall. However, grading open questions is a time-intensive task for teachers. Automatic short answer grading (ASAG) has tried to fill this gap, and with the recent advent of Large Language Models (LLM), this branch has seen a new momentum.METHODS:
We graded 2288 student answers from 12 undergraduate medical education courses in 3 languages using GPT-4 and Gemini 1.0 Pro.RESULTS:
GPT-4 proposed significantly lower grades than the human evaluator, but reached low rates of false positives. The grades of Gemini 1.0 Pro were not significantly different from the teachers'. Both LLMs reached a moderate agreement with human grades, and a high precision for GPT-4 among answers considered fully correct. A consistent grading behavior could be determined for high-quality keys. A weak correlation was found wrt. the length or language of student answers. There is a risk of bias if the LLM knows the human grade a priori.CONCLUSIONS:
LLM-based ASAG applied to medical education still requires human oversight, but time can be spared on the edge cases, allowing teachers to focus on the middle ones. For Bachelor-level medical education questions, the training knowledge of LLMs seems to be sufficient, fine-tuning is thus not necessary.Palavras-chave
Texto completo:
1
Coleções:
01-internacional
Base de dados:
MEDLINE
Assunto principal:
Educação de Graduação em Medicina
/
Avaliação Educacional
Limite:
Humans
Idioma:
En
Revista:
BMC Med Educ
/
BMC med. educ
/
BMC medical education
Assunto da revista:
EDUCACAO
Ano de publicação:
2024
Tipo de documento:
Article
País de afiliação:
Luxemburgo
País de publicação:
Reino Unido