ABSTRACT
OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance (P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.
ABSTRACT
AIMS AND OBJECTIVES: The aim was to systematically review evidence about the effectiveness of in-home community nurse-led interventions for older persons with, or at risk of, mental health disorders, to inform best practice nursing care with this focus. The primary review question was 'How effective are in-home community nurse-led interventions for older persons with or at risk of mental health disorders for improving mental health?' The outcome indices of interest were nursing actions to determine incidence or prevalence of mental health disorders, any change in a patient's attitude towards their mental health condition, any change in objective measurement of mental health, or a change in diagnostic status. BACKGROUND: The rising incidence of mental health disorders in older persons is a major concern for community nurses in developed countries. Effectively facilitating improved mental health for older persons is necessary in this era of ageing populations with increased demands on health funding. Disseminating systematically reviewed evidence for in-home community nursing that positively impacts on the mental health of older persons is crucial to ensure effective care is provided to this vulnerable patient group. RESULTS: This review reveals that there is evidence to support the superiority of applying validated screening tools for mental health disorders over relying on community nurses' opinions and non-validated tools about this matter. DESIGN: Systematic review. METHODS: Search of electronic databases. CONCLUSION: A clear need for replication and multi-centre trials of reviewed pertinent studies is identified. Relevance to clinical practice. Community nurses should consider using validated screening tools for this focus. Until such time as higher quality evidence is available about other nursing interventions, the reviewers suggest that the prime nursing action should be the identification of whether older persons receiving community nursing care might have a mental health disorder and, if so, then collaborative referral is made to appropriate services.
