Comparing human text classification performance and explainability with large language and machine learning models using eye-tracking.

To understand the alignment between reasonings of humans and artificial intelligence (AI) models, this empirical study compared the human text classification performance and explainability with a traditional machine learning (ML) model and large language model (LLM). A domain-specific noisy textual dataset of 204 injury narratives had to be classified into 6 cause-of-injury codes. The narratives varied in terms of complexity and ease of categorization based on the distinctive nature of cause-of-injury code. The user study involved 51 participants whose eye-tracking data was recorded while they performed the text classification task. While the ML model was trained on 120,000 pre-labelled injury narratives, LLM and humans did not receive any specialized training. The explainability of different approaches was compared based on the top words they used for making classification decision. These words were identified using eye-tracking for humans, explainable AI approach LIME for ML model, and prompts for LLM. The classification performance of ML model was observed to be relatively better than zero-shot LLM and non-expert humans, overall, and particularly for narratives with high complexity and difficult categorization. The top-3 predictive words used by ML and LLM for classification agreed with humans to a greater extent as compared to later predictive words.

Application of a Machine Learning-Based Decision Support Tool to Improve an Injury Surveillance System Workflow.

BACKGROUND: Emergency department (ED)-based injury surveillance systems across many countries face resourcing challenges related to manual validation and coding of data. OBJECTIVE: This study describes the evaluation of a machine learning (ML)-based decision support tool (DST) to assist injury surveillance departments in the validation, coding, and use of their data, comparing outcomes in coding time, and accuracy pre- and postimplementations. METHODS: Manually coded injury surveillance data have been used to develop, train, and iteratively refine a ML-based classifier to enable semiautomated coding of injury narrative data. This paper describes a trial implementation of the ML-based DST in the Queensland Injury Surveillance Unit (QISU) workflow using a major pediatric hospital's ED data comparing outcomes in coding time and pre- and postimplementation accuracies. RESULTS: The study found a 10% reduction in manual coding time after the DST was introduced. The Kappa statistics analysis in both DST-assisted and -unassisted data shows increase in accuracy across three data fields, that is, injury intent (85.4% unassisted vs. 94.5% assisted), external cause (88.8% unassisted vs. 91.8% assisted), and injury factor (89.3% unassisted vs. 92.9% assisted). The classifier was also used to produce a timely report monitoring injury patterns during the novel coronavirus disease 2019 (COVID-19) pandemic. Hence, it has the potential for near real-time surveillance of emerging hazards to inform public health responses. CONCLUSION: The integration of the DST into the injury surveillance workflow shows benefits as it facilitates timely reporting and acts as a DST in the manual coding process.

Applying text mining to identify relevant literature in food science: Cold denaturation as a case study.

In a research environment characterized by the five V's of big data, volume, velocity, variety, value, and veracity, the need to develop tools that quickly screen a large number of publications into relevant work is an increasing area of concern, and the data-rich food industry is no exception. Here, a combination of latent Dirichlet allocation and food keyword searches were employed to analyze and filter a dataset of 6102 publications about cold denaturation. After using the Python toolkit generated in this work, the approach yielded 22 topics that provide background and insight on the direction of research in this field, as well as identified the publications in this dataset which are most pertinent to the food industry with precision and recall of 0.419 and 0.949, respectively. Precision is related to the relevance of a paper in the filtered dataset and the recall represents papers which were not identified in the screening method. Lastly, gaps in the literature based on keyword trends are identified to improve the knowledge base of cold denaturation as it relates to the food industry. This approach is generalizable to any similarly organized dataset, and the code is available upon request. Practical Application: A common problem in research is that when you are an expert in one field, learning about another field is difficult, because you may lack the vocabulary and background needed to read cutting edge literature from a new discipline. The Python toolkit developed in this research can be applied by any researcher that is new to a field to identify what the key literature is, what topics they should familiarize themselves with, and what the current trends are in the field. Using this structure, researchers can greatly speed up how they identify new areas to research and find new projects.

Improving autocoding performance of rare categories in injury classification: Is more training data or filtering the solution?

INTRODUCTION: Classical Machine Learning (ML) models have been found to assign the external-cause-of-injury codes (E-codes) based on injury narratives with good overall accuracy but often struggle with rare categories, primarily due to lack of enough training cases and heavily skewed nature of injurdata. In this paper, we have: a) studied the effect of increasing the size of training data on the prediction performance of three classical ML models: Multinomial Naïve Bayes (MNB), Support Vector Machine (SVM) and Logistic Regression (LR), and b) studied the effect of filtering based on prediction strength of LR model when the model is trained on very-small (10,000 cases) and very-large (450,000 cases) training sets. METHOD: Data from Queensland Injury Surveillance Unit from years 2002-2012, which was categorized into 20 broad E-codes was used for this study. Eleven randomly chosen training sets of size ranging from 10,000 to 450,000 cases were used to train the ML models, and the prediction performance was analyzed on a prediction set of 50,150 cases. Filtering approach was tested on LR models trained on smallest and largest training datasets. Sensitivity was used as the performance measure for individual categories. Weighted average sensitivity (WAvg) and Unweighted average sensitivity (UAvg) were used as the measures of overall performance. Filtering approach was also tested for estimating category counts and was compared with approaches of summing prediction probabilities and counting direct predictions by ML model. RESULTS: The overall performance of all three ML models improved with increase in the size of training data. The overall sensitivities with maximum training size for LR and SVM models were similar (∼82%), and higher than MNB (76%). For all the ML models, the sensitivities of rare categories improved with increasing training data but they were considerably less than sensitivities of larger categories. With increasing training data size, LR and SVM exhibited diminishing improvement in UAvg whereas the improvement was relatively steady in case of MNB. Filtering based on prediction strength of LR model (and manual review of filtered cases) helped in improving the sensitivities of rare categories. A sizeable portion of cases still needed to be filtered even when the LR model was trained on very large training set. For estimating category counts, filtering approach provided best estimates for most E-codes and summing prediction probabilities approach provided better estimates for rare categories. CONCLUSIONS: Increasing the size of training data alone cannot solve the problem of poor classification performance on rare categories by ML models. Filtering could be an effective strategy to improve classification performance of rare categories when large training data is not available.

Bayesian decision support for coding occupational injury data.

INTRODUCTION: Studies on autocoding injury data have found that machine learning algorithms perform well for categories that occur frequently but often struggle with rare categories. Therefore, manual coding, although resource-intensive, cannot be eliminated. We propose a Bayesian decision support system to autocode a large portion of the data, filter cases for manual review, and assist human coders by presenting them top k prediction choices and a confusion matrix of predictions from Bayesian models. METHOD: We studied the prediction performance of Single-Word (SW) and Two-Word-Sequence (TW) Naïve Bayes models on a sample of data from the 2011 Survey of Occupational Injury and Illness (SOII). We used the agreement in prediction results of SW and TW models, and various prediction strength thresholds for autocoding and filtering cases for manual review. We also studied the sensitivity of the top k predictions of the SW model, TW model, and SW-TW combination, and then compared the accuracy of the manually assigned codes to SOII data with that of the proposed system. RESULTS: The accuracy of the proposed system, assuming well-trained coders reviewing a subset of only 26% of cases flagged for review, was estimated to be comparable (86.5%) to the accuracy of the original coding of the data set (range: 73%-86.8%). Overall, the TW model had higher sensitivity than the SW model, and the accuracy of the prediction results increased when the two models agreed, and for higher prediction strength thresholds. The sensitivity of the top five predictions was 93%. CONCLUSIONS: The proposed system seems promising for coding injury data as it offers comparable accuracy and less manual coding. PRACTICAL APPLICATIONS: Accurate and timely coded occupational injury data is useful for surveillance as well as prevention activities that aim to make workplaces safer.

Defect Control and n-Doping of Encapsulated Graphene by Helium-Ion-Beam Irradiation.

We study with Raman spectroscopy the influences of He(+) bombardment and the environment on beam-induced defects in graphene encapsulated in hexagonal boron nitride (h-BN). We show for the first time experimentally the autonomous behavior of the D' defect Raman peak: in contrast to the D defect peak, the D' defect peak is sensitive to the local environment. In particular, it saturates with ion dose in the encapsulated graphene. Electrical measurements reveal n-type conduction in the BN-encapsulated graphene. We conclude that unbound atoms ("interfacials") between the sp(2)-layers of graphene and h-BN promote self-healing of the beam-induced lattice damage and that nitrogen-carbon exchange leads to n-doping of graphene.