Between Always and Never: Evaluating Uncertainty in Radiology Reports Using Natural Language Processing.

The ideal radiology report reduces diagnostic uncertainty, while avoiding ambiguity whenever possible. The purpose of this study was to characterize the use of uncertainty terms in radiology reports at a single institution and compare the use of these terms across imaging modalities, anatomic sections, patient characteristics, and radiologist characteristics. We hypothesized that there would be variability among radiologists and between subspecialities within radiology regarding the use of uncertainty terms and that the length of the impression of a report would be a predictor of use of uncertainty terms. Finally, we hypothesized that use of uncertainty terms would often be interpreted by human readers as "hedging." To test these hypotheses, we applied a natural language processing (NLP) algorithm to assess and count the number of uncertainty terms within radiology reports. An algorithm was created to detect usage of a published set of uncertainty terms. All 642,569 radiology report impressions from 171 reporting radiologists were collected from 2011 through 2015. For validation, two radiologists without knowledge of the software algorithm reviewed report impressions and were asked to determine whether the report was "uncertain" or "hedging." The relationship between the presence of 1 or more uncertainty terms and the human readers' assessment was compared. There were significant differences in the proportion of reports containing uncertainty terms across patient admission status and across anatomic imaging subsections. Reports with uncertainty were significantly longer than those without, although report length was not significantly different between subspecialities or modalities. There were no significant differences in rates of uncertainty when comparing the experience of the attending radiologist. When compared with reader 1 as a gold standard, accuracy was 0.91, sensitivity was 0.92, specificity was 0.9, and precision was 0.88, with an F1-score of 0.9. When compared with reader 2, accuracy was 0.84, sensitivity was 0.88, specificity was 0.82, and precision was 0.68, with an F1-score of 0.77. Substantial variability exists among radiologists and subspecialities regarding the use of uncertainty terms, and this variability cannot be explained by years of radiologist experience or differences in proportions of specific modalities. Furthermore, detection of uncertainty terms demonstrates good test characteristics for predicting human readers' assessment of uncertainty.

Evaluating the Performance of Augmented Reality in Displaying Magnetic Resonance Imaging-Derived Three-Dimensional Holographic Models.

INTRODUCTION/BACKGROUND: Establishing accuracy and precision of magnetic resonance (MR)-derived augmented reality (AR) models is critical before clinical utilization, particularly in preoperative planning. We investigate the performance of an AR application in representing and displaying MR-derived three-dimensional holographic models. METHODS: Thirty gold standard (GS) measurements were obtained on a magnetic resonance imaging (MRI) phantom (six interfiducial distances and five configurations). Four MRI pulse sequences were obtained for each of the five configurations, and distances measured in Picture Archiving and Communication System (PACS). Digital imaging and communications in medicine files were translated into three-dimensional models and then loaded onto a novel AR platform. Measurements were also obtained with the software's AR caliper tool. Significant differences among the three groups (GS, PACS, and AR) were assessed with the Kruskal-Wallis test and nonsample median test. Accuracy analysis of GS vs. AR was performed. Precision (percent deviation) of the AR-based caliper tool was also assessed. RESULTS: No statistically significant difference existed between AR and GS measurements (P = .6208). PACS demonstrated mean squared error (MSE) of 0.29%. AR digital caliper demonstrated an MSE of 0.3%. Three-dimensional T2 CUBE AR measurements using the platform's AR caliper tool demonstrated an MSE of 8.6%. Percent deviation of AR software caliper tool ranged between 1.9% and 3.9%. DISCUSSION: AR demonstrated a high degree of accuracy in comparison to GS, comparable to PACS-based measurements. AR caliper tool demonstrated overall lower accuracy than with physical calipers, although with MSE <10% and greatest measured difference from GS measuring <5 mm. AR-based caliper demonstrated a high degree of precision. CONCLUSION: There was no statistically significant difference between GS measurements and three-dimensional AR measurements in MRI phantom models.