ABSTRACT
Recent years have seen increased interests in applying the latest technological innovations, including artificial intelligence (AI) and machine learning (ML), to the field of education. One of the main areas of interest to researchers is the use of ML to assist teachers in assessing students' work on the one hand and to promote effective self-tutoring on the other hand. In this paper, we present a survey of the latest ML approaches to the automated evaluation of students' natural language free-text, including both short answers to questions and full essays. Existing systematic literature reviews on the subject often emphasise an exhaustive and methodical study selection process and do not provide much detail on individual studies or a technical background to the task. In contrast, we present an accessible survey of the current state-of-the-art in student free-text evaluation and target a wider audience that is not necessarily familiar with the task or with ML-based text analysis in natural language processing (NLP). We motivate and contextualise the task from an application perspective, illustrate popular feature-based and neural model architectures and present a selection of the latest work in the area. We also remark on trends and challenges in the field.
ABSTRACT
BACKGROUND: Computational linguistic methodology allows quantification of speech abnormalities in non-affective psychosis. For this patient group, incoherent speech has long been described as a symptom of formal thought disorder. Our study is an interdisciplinary attempt at developing a model of incoherence in non-affective psychosis, informed by computational linguistic methodology as well as psychiatric research, which both conceptualize incoherence as associative loosening. The primary aim of this pilot study was methodological: to validate the model against clinical data and reduce bias in automated coherence analysis. METHODS: Speech samples were obtained from patients with a diagnosis of schizophrenia or schizoaffective disorder, who were divided into two groups of n = 20 subjects each, based on different clinical ratings of positive formal thought disorder, and n = 20 healthy control subjects. RESULTS: Coherence metrics that were automatically derived from interview transcripts significantly predicted clinical ratings of thought disorder. Significant results from multinomial regression analysis revealed that group membership (controls vs. patients with vs. without formal thought disorder) could be predicted based on automated coherence analysis when bias was considered. Further improvement of the regression model was reached by including variables that psychiatric research has shown to inform clinical diagnostics of positive formal thought disorder. CONCLUSIONS: Automated coherence analysis may capture different features of incoherent speech than clinical ratings of formal thought disorder. Models of incoherence in non-affective psychosis should include automatically derived coherence metrics as well as lexical and syntactic features that influence the comprehensibility of speech.
ABSTRACT
Computer-assisted text coding can facilitate the analysis of large text collections. To evaluate the functionality of providing an analyst with a ranked list of suggestions for suitable text codes, we used a data set of discussion posts, which had been manually coded for reasons given for taking a stance on the topic of vaccination. We trained a logistic regression classifier to rank these reasons according to the probability that they would be present in the post. The approach was evaluated for its ability to include the expected reasons among the n top-ranked reasons, using an n between 1 and 6. The logistic regression-based ranking was more effective than the baseline, which ranked reasons according to their frequency in the training data. Providing such a list of possible codes, ranked by logistic regression, could therefore be a useful feature in a tool for text coding.
Subject(s)
Vaccination , Internet , Social MediaABSTRACT
Arguments used when vaccination is debated on Internet discussion forums might give us valuable insights into reasons behind vaccine hesitancy. In this study, we applied automatic topic modelling on a collection of 943 discussion posts in which vaccine was debated, and six distinct discussion topics were detected by the algorithm. When manually coding the posts ranked as most typical for these six topics, a set of semantically coherent arguments were identified for each extracted topic. This indicates that topic modelling is a useful method for automatically identifying vaccine-related discussion topics and for identifying debate posts where these topics are discussed. This functionality could facilitate manual coding of salient arguments, and thereby form an important component in a system for computer-assisted coding of vaccine-related discussions.
