Evaluating Chatbot responses to patient questions in the field of glaucoma.

Objective: The aim of this study was to evaluate the accuracy, comprehensiveness, and safety of a publicly available large language model (LLM)-ChatGPT in the sub-domain of glaucoma. Design: Evaluation of diagnostic test or technology. Subjects participants and/or controls: We seek to evaluate the responses of an artificial intelligence chatbot ChatGPT (version GPT-3.5, OpenAI). Methods intervention or testing: We curated 24 clinically relevant questions in the domain of glaucoma. The questions spanned four categories: pertaining to diagnosis, treatment, surgeries, and ocular emergencies. Each question was posed to the LLM and the responses obtained were graded by an expert grader panel of three glaucoma specialists with combined experience of more than 30 years in the field. For responses which performed poorly, the LLM was further prompted to self-correct. The subsequent responses were then re-evaluated by the expert panel. Main outcome measures: Accuracy, comprehensiveness, and safety of the responses of a public domain LLM. Results: There were a total of 24 questions and three expert graders with a total number of responses of n = 72. The scores were ranked from 1 to 4, where 4 represents the best score with a complete and accurate response. The mean score of the expert panel was 3.29 with a standard deviation of 0.484. Out of the 24 question-response pairs, seven (29.2%) of them had a mean inter-grader score of 3 or less. The mean score of the original seven question-response pairs was 2.96 which rose to 3.58 after an opportunity to self-correct (z-score - 3.27, p = 0.001, Mann-Whitney U). The seven out of 24 question-response pairs which performed poorly were given a chance to self-correct. After self-correction, the proportion of responses obtaining a full score increased from 22/72 (30.6%) to 12/21 (57.1%), (p = 0.026, χ2 test). Conclusion: LLMs show great promise in the realm of glaucoma with additional capabilities of self-correction. The application of LLMs in glaucoma is still in its infancy, and still requires further research and validation.

Evaluating TESLA-G, a gamified, telegram-delivered, quizzing platform for surgical education in medical students: protocol for a pilot randomised controlled trial.

INTRODUCTION: Online multiple-choice question (MCQ) quizzes are popular in medical education due to their ease of access and ability for test-enhanced learning. However, a general lack of motivation among students often results in decreasing usage over time. We aim to address this limitation by developing Telegram Education for Surgical Learning and Application Gamified (TESLA-G), an online platform for surgical education that incorporates game elements into conventional MCQ quizzes. METHODS AND ANALYSIS: This online, pilot randomised control trial will be conducted over 2 weeks. Fifty full-time undergraduate medical students from a medical school in Singapore will be recruited and randomised into an intervention group (TESLA-G) and an active control group (non-gamified quizzing platform) with a 1:1 allocation ratio, stratified by year of study.We will evaluate TESLA-G in the area of endocrine surgery education. Our platform is designed based on Bloom's taxonomy of learning domains: questions are created in blocks of five questions per endocrine surgery topic, with each question corresponding to one level on Bloom's taxonomy. This structure promotes mastery while boosting student engagement and motivation. All questions are created by two board-certified general surgeons and one endocrinologist, and validated by the research team. The feasibility of this pilot study will be determined quantitatively by participant enrolment, participant retention and degree of completion of the quizzes. The acceptability of the intervention will be assessed quantitatively by a postintervention learner satisfaction survey consisting of a system satisfaction questionnaire and a content satisfaction questionnaire. The improvement of surgical knowledge will be assessed by comparing the scores of preintervention and postintervention knowledge tests, which consist of separately created questions on endocrine surgery. Retention of surgical knowledge will be measured using a follow-up knowledge test administered 2 weeks postintervention. Finally, qualitative feedback from participants regarding their experience will be obtained and thematically analysed. ETHICS AND DISSEMINATION: This research is approved by Singapore Nanyang Technological University (NTU) Institutional Review Boards (Reference Number: IRB-2021-732). All participants will be expected to read and sign a letter of informed consent before they are considered as recruited into the study. This study poses minimal risk to participants. Study results will be published in peer-reviewed open-access journals and presented in conference presentations. TRIAL REGISTRATION NUMBER: NCT05520671.

A data-driven approach to evaluate factors affecting resident performance in cataract surgery.

PURPOSE: To evaluate the operative duration and clinical performance of ophthalmology residents performing standard phacoemulsification cataract surgeries using information available from electronic health records (EHR). METHODS: This is a retrospective cohort study. De-identified surgical records of all standard phacoemulsifications performed in a tertiary institution between 1st January 2015 and 8th August 2018 were retrieved from the hospital EHR. The main outcome measures were improvement in operative duration with case experience, corrected distance visual acuity (CDVA) improvement, and intra-operative complication rates. RESULTS: Twelve ophthalmology residents performed a total of 1427 standard phacoemulsifications. The median operative duration was 27 min (interquartile range, 22-34 min), which improved from 31 to 24 min (before the 101st case [Group 1] versus 101st case onwards [Group 2], p < 0.001). Gradient change analysis (non-linear regression) showed significant reduction until the 100th case (p = 0.043). Older patients (0.019), worse pre-operative CDVA (0.343), and surgery performed by Group 1 (1.115) were significantly associated with operative duration above 30 min. LogMAR CDVA improved from a mean of 0.57 ± 0.52 pre-operatively to 0.10 ± 0.18 post-operatively (p < 0.001). Posterior capsule rupture (PCR) rate decreased from 4.0% [Group 1] to 2.1% [Group 2] (p = 0.096), while overall complication rate decreased from 8.9% to 3.1% (p < 0.001). CONCLUSION: The median operative duration reduced consistently with surgical experience for the first 100 cases. Older patients, poorer pre-operative VA, and surgical experience of less than 100 cases were significantly associated with an operative duration above 30 min. There was a statistically significant decrease in complication rate between Group 1 and 2.