Large Language Model-Based Neurosurgical Evaluation Matrix: A Novel Scoring Criteria to Assess the Efficacy of ChatGPT as an Educational Tool for Neurosurgery Board Preparation.

INTRODUCTION: Technological advancements are reshaping medical education, with digital tools becoming essential in all levels of training. Amidst this transformation, the study explores the potential of ChatGPT, an artificial intelligence model by OpenAI, in enhancing neurosurgical board education. The focus extends beyond technology adoption to its effective utilization, with ChatGPT's proficiency evaluated against practice questions from the Primary Neurosurgery Written Board Exam. METHODS: Using the Congress of Neurologic Surgeons (CNS) Self-Assessment Neurosurgery (SANS) Exam Board Review Prep questions, we conducted 3 rounds of analysis with ChatGPT. We developed a novel ChatGPT Neurosurgical Evaluation Matrix (CNEM) to assess the output quality, accuracy, concordance, and clarity of ChatGPT's answers. RESULTS: ChatGPT achieved spot-on accuracy for 66.7% of prompted questions, 59.4% of unprompted questions, and 63.9% of unprompted questions with a leading phrase. Stratified by topic, accuracy ranged from 50.0% (Vascular) to 78.8% (Neuropathology). In comparison to SANS explanations, ChatGPT output was considered better in 19.1% of questions, equal in 51.6%, and worse in 29.3%. Concordance analysis showed that 95.5% of unprompted ChatGPT outputs and 97.4% of unprompted outputs with a leading phrase were aligned. CONCLUSIONS: Our study evaluated the performance of ChatGPT in neurosurgical board education by assessing its accuracy, clarity, and concordance. The findings highlight the potential and challenges of integrating AI technologies like ChatGPT into medical and neurosurgical board education. Further research is needed to refine these tools and optimize their performance for enhanced medical education and patient care.

Defining the minimal clinically important difference in smartphone-based mobility after spine surgery: correlation of survey questionnaire to mobility data.

OBJECTIVE: Patient-reported outcome measures (PROMs) are the gold standard for assessing postoperative outcomes in spine surgery. However, PROMs are also limited by the inherent subjectivity of self-reported qualitative data. Recent literature has highlighted the utility of patient mobility data streamed from smartphone accelerometers as an objective measure of functional outcomes and complement to traditional PROMs. Still, for activity-based data to supplement existing PROMs, they must be validated against current metrics. In this study, the authors assessed the relationships and concordance between longitudinal smartphone-based mobility data and PROMs. METHODS: Patients receiving laminectomy (n = 21) or fusion (n = 10) between 2017 and 2022 were retrospectively included. Activity data (steps-per-day count) recorded in the Apple Health mobile application over a 2-year perioperative window were extracted and subsequently normalized to allow for intersubject comparison. PROMS, including the visual analog scale (VAS), Patient Reported Outcome Measurement Information System Pain Interference (PROMIS-PI), Oswestry Disability Index (ODI), and EQ-5D, collected at the preoperative and 6-week postoperative visits were retrospectively extracted from the electronic medical record. Correlations between PROMs and patient mobility were assessed and compared between patients who did and those who did not achieve the established minimal clinically important difference (MCID) for each measure. RESULTS: A total of 31 patients receiving laminectomy (n = 21) or fusion (n = 10) were included. Change between preoperative and 6-week postoperative VAS and PROMIS-PI scores demonstrated moderate (r = -0.46) and strong (r = -0.74) inverse correlations, respectively, with changes in normalized steps-per-day count. In cohorts of patients who achieved PROMIS-PI MCID postoperatively, indicating subjective improvement in pain, there was a 0.784 standard deviation increase in normalized steps per day, representing a 56.5% improvement (p = 0.027). Patients who did achieve the MCID of improvement in either PROMIS-PI or VAS after surgery were more likely to experience an earlier sustained improvement in physical activity commensurate to or greater than their preoperative baseline (p = 2.98 × 10-18) than non-MCID patients. CONCLUSIONS: This study demonstrates a strong correlation between changes in mobility data extracted from patient smartphones and changes in PROMs following spine surgery. Further elucidating this relationship will allow for more robust supplementation of existing spine outcome measure tools with analyzed objective activity data.