ABSTRACT
BACKGROUND: Large language models exhibiting human-level performance in specialized tasks are emerging; examples include Generative Pretrained Transformer 3.5, which underlies the processing of ChatGPT. Rigorous trials are required to understand the capabilities of emerging technology, so that innovation can be directed to benefit patients and practitioners. OBJECTIVE: Here, we evaluated the strengths and weaknesses of ChatGPT in primary care using the Membership of the Royal College of General Practitioners Applied Knowledge Test (AKT) as a medium. METHODS: AKT questions were sourced from a web-based question bank and 2 AKT practice papers. In total, 674 unique AKT questions were inputted to ChatGPT, with the model's answers recorded and compared to correct answers provided by the Royal College of General Practitioners. Each question was inputted twice in separate ChatGPT sessions, with answers on repeated trials compared to gauge consistency. Subject difficulty was gauged by referring to examiners' reports from 2018 to 2022. Novel explanations from ChatGPT-defined as information provided that was not inputted within the question or multiple answer choices-were recorded. Performance was analyzed with respect to subject, difficulty, question source, and novel model outputs to explore ChatGPT's strengths and weaknesses. RESULTS: Average overall performance of ChatGPT was 60.17%, which is below the mean passing mark in the last 2 years (70.42%). Accuracy differed between sources (P=.04 and .06). ChatGPT's performance varied with subject category (P=.02 and .02), but variation did not correlate with difficulty (Spearman ρ=-0.241 and -0.238; P=.19 and .20). The proclivity of ChatGPT to provide novel explanations did not affect accuracy (P>.99 and .23). CONCLUSIONS: Large language models are approaching human expert-level performance, although further development is required to match the performance of qualified primary care physicians in the AKT. Validated high-performance models may serve as assistants or autonomous clinical tools to ameliorate the general practice workforce crisis.
ABSTRACT
STUDY DESIGN: Modified DELPHI Consensus Process. OBJECTIVE: To agree a single unifying term and definition. Globally, cervical myelopathy caused by degenerative changes to the spine is known by over 11 different names. This inconsistency contributes to many clinical and research challenges, including a lack of awareness. METHOD: AO Spine RECODE-DCM (Research objectives and Common Data Elements Degenerative Cervical Myelopathy). To determine the index term, a longlist of candidate terms and their rationale, was created using a literature review and interviews. This was shared with the community, to select their preferred terms (248 members (58%) including 149 (60%) surgeons, 45 (18%) other healthcare professionals and 54 (22%) People with DCM or their supporters) and finalized using a consensus meeting. To determine a definition, a medical definition framework was created using inductive thematic analysis of selected International Classification of Disease definitions. Separately, stakeholders submitted their suggested definition which also underwent inductive thematic analysis (317 members (76%), 190 (59%) surgeons, 62 (20%) other healthcare professionals and 72 (23%) persons living with DCM or their supporters). Using this definition framework, a working definition was created based on submitted content, and finalized using consensus meetings. RESULTS: Degenerative Cervical Myelopathy was selected as the unifying term, defined in short, as a progressive spinal cord injury caused by narrowing of the cervical spinal canal. CONCLUSION: A consistent term and definition can support education and research initiatives. This was selected using a structured and iterative methodology, which may serve as an exemplar for others in the future.
