Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain.

RATIONALE AND OBJECTIVES: To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems. MATERIALS AND METHODS: We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N = 178,333 reports, we randomly selected N = 871 to form a reference-standard dataset, consisting of N = 413 x-ray reports and N = 458 MR reports. Using standardized criteria, four spine experts annotated the presence of 26 findings, where 71 reports were annotated by all four experts and 800 were each annotated by two experts. We calculated inter-rater agreement and finding prevalence from annotated data. We randomly split the annotated data into development (80%) and testing (20%) sets. We developed an NLP system from both rule-based and machine-learned models. We validated the system using accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). RESULTS: The multirater annotated dataset achieved inter-rater agreement of Cohen's kappa > 0.60 (substantial agreement) for 25 of 26 findings, with finding prevalence ranging from 3% to 89%. In the testing sample, rule-based and machine-learned predictions both had comparable average specificity (0.97 and 0.95, respectively). The machine-learned approach had a higher average sensitivity (0.94, compared to 0.83 for rules-based), and a higher overall AUC (0.98, compared to 0.90 for rules-based). CONCLUSIONS: Our NLP system performed well in identifying the 26 lumbar spine findings, as benchmarked by reference-standard annotation by medical experts. Machine-learned models provided substantial gains in model sensitivity with slight loss of specificity, and overall higher AUC.

Using Natural Language Processing of Free-Text Radiology Reports to Identify Type 1 Modic Endplate Changes.

Electronic medical record (EMR) systems provide easy access to radiology reports and offer great potential to support quality improvement efforts and clinical research. Harnessing the full potential of the EMR requires scalable approaches such as natural language processing (NLP) to convert text into variables used for evaluation or analysis. Our goal was to determine the feasibility of using NLP to identify patients with Type 1 Modic endplate changes using clinical reports of magnetic resonance (MR) imaging examinations of the spine. Identifying patients with Type 1 Modic change who may be eligible for clinical trials is important as these findings may be important targets for intervention. Four annotators identified all reports that contained Type 1 Modic change, using N = 458 randomly selected lumbar spine MR reports. We then implemented a rule-based NLP algorithm in Java using regular expressions. The prevalence of Type 1 Modic change in the annotated dataset was 10%. Results were recall (sensitivity) 35/50 = 0.70 (95% confidence interval (C.I.) 0.52-0.82), specificity 404/408 = 0.99 (0.97-1.0), precision (positive predictive value) 35/39 = 0.90 (0.75-0.97), negative predictive value 404/419 = 0.96 (0.94-0.98), and F1-score 0.79 (0.43-1.0). Our evaluation shows the efficacy of rule-based NLP approach for identifying patients with Type 1 Modic change if the emphasis is on identifying only relevant cases with low concern regarding false negatives. As expected, our results show that specificity is higher than recall. This is due to the inherent difficulty of eliciting all possible keywords given the enormous variability of lumbar spine reporting, which decreases recall, while availability of good negation algorithms improves specificity.

CT prediction of the Fuhrman grade of clear cell renal cell carcinoma (RCC): towards the development of computer-assisted diagnostic method.

PURPOSE: There are distinct quantifiable features characterizing renal cell carcinomas on contrast-enhanced CT examinations, such as peak tumor enhancement, tumor heterogeneity, and percent contrast washout. While qualitative visual impressions often suffice for diagnosis, quantitative metrics if developed and validated can add to the information available from standard of care diagnostic imaging. The purpose of this study is to assess the use of quantitative enhancement metrics in predicting the Fuhrman grade of clear cell RCC. MATERIALS AND METHODS: 65 multiphase CT examinations with clear cell RCCs were utilized, 44 tumors with Fuhrman grades 1 or 2 and 21 tumors with grades 3 or 4. After tumor segmentation, the following data were extracted: histogram analysis of voxel-based whole lesion attenuation in each phase, enhancement and washout using mean, median, skewness, kurtosis, standard deviation, and interquartile range. RESULTS: Statistically significant difference was observed in 4 measured parameters between grades 1-2 and grades 3-4: interquartile range of nephrographic attenuation values, standard deviation of absolute enhancement, as well as interquartile range and standard deviation of residual nephrographic enhancement. Interquartile range of nephrographic attenuation values was 292.86 HU for grades 1-2 and 241.19 HU for grades 3-4 (p value 0.02). Standard deviation of absolute enhancement was 41.26 HU for grades 1-2 and 34.66 HU for grades 3-4 (p value 0.03). Interquartile range was 297.12 HU for residual nephrographic enhancement for grades 1-2 and 235.57 HU for grades 3-4 (p value 0.02), and standard deviation of the same was 42.45 HU for grades 1-2 and 37.11 for grades 3-4 (p value 0.04). CONCLUSION: Our results indicate that absolute enhancement is more heterogeneous for lower grade tumors and that attenuation and residual enhancement in nephrographic phase is more heterogeneous for lower grade tumors. This represents an important step in devising a predictive non-invasive model to predict the nucleolar grade.

Whole lesion quantitative CT evaluation of renal cell carcinoma: differentiation of clear cell from papillary renal cell carcinoma.

PURPOSE: Clear cell renal cell carcinoma (ccRCC) is the most common subtype of renal cell cancer (RCC), followed by papillary RCC (pRCC). It is important to distinguish these two subtypes because of prognostic differences and possible changes in management, especially in cases undergoing active surveillance. The purpose of our study is to evaluate the use of voxel-based whole-lesion (WL) enhancement parameters on contrast enhanced computed tomography (CECT) to distinguish ccRCC from pRCC. MATERIALS AND METHODS: In this institutional review board-approved study, we retrospectively queried the surgical database for post nephrectomy patients who had pathology proven ccRCC or pRCC and who had preoperative multiphase CECT of the abdomen between June 2009 and June 2011. A total of 61 patients (46 with ccRCC and 15 with pRCC) who underwent robotic assisted partial nephrectomy for clinically localized disease were included in the study. Multiphase CT acquisitions were transferred to a dedicated three-dimensional workstation, and WL regions of interest were manually segmented. Voxel-based contrast enhancement values were collected from the lesion segmentation and displayed as a histogram. Mean and median enhancement and histogram distribution parameters skewness, kurtosis, standard deviation, and interquartile range were calculated for each lesion. Comparison between ccRCC and pRCC was made using each imaging parameter. For mean and median enhancement, which had a normal distribution, independent t-test was used. For histogram distribution parameters, which were not normally distributed, Wilcoxon rank sum test was used. RESULTS: ccRCC had significantly higher mean and median whole WL enhancement (p < 0.01) compared to pRCC on arterial, nephrographic, and excretory phases. ccRCC had significantly higher interquartile range and standard deviation (p < 0.01) and significantly lower skewness (p < 0.01) compared to pRCC on arterial and nephrographic phases. ccRCC had significantly lower kurtosis compared to pRCC on only the arterial phase. CONCLUSION: Our study suggests that voxel-based WL enhancement parameters can be used as a quantitative tool to differentiate ccRCC from pRCC. Differentiating between the two main types of RCC would provide the patient and the treating physicians more information to formulate the initial approach to managing the patient's renal cancer.

Quantitative detection of cirrhosis: towards the development of computer-assisted detection method.

There are distinct morphologic features of cirrhosis on CT examinations; however, such impressions may be subtle or subjective. The purpose of this study is to build a computer-aided diagnosis (CAD) method to help radiologists with this diagnosis. One hundred sixty-seven abdominal CT examinations were randomly divided into training (n = 88) and validation (n = 79) sets. Livers were analyzed for morphological markers of cirrhosis and logistic regression models were created. Using the area under curve (AUC) for model performance, the best model had 0.89 for the training set and 0.85 for the validation set. For radiology reports, sensitivity of reporting cirrhosis was 0.45 and specificity 0.99. Using the predictive model adjunctively, radiologists' sensitivity increased to 0.63 and specificity slightly decreased to 0.97. This study demonstrates that quantifying morphological features in livers may be utilized for diagnosing cirrhosis and for developing a CAD method for it.

Image coregistration: quantitative processing framework for the assessment of brain lesions.

The quantitative, multiparametric assessment of brain lesions requires coregistering different parameters derived from MRI sequences. This will be followed by analysis of the voxel values of the ROI within the sequences and calculated parametric maps, and deriving multiparametric models to classify imaging data. There is a need for an intuitive, automated quantitative processing framework that is generalized and adaptable to different clinical and research questions. As such flexible frameworks have not been previously described, we proceeded to construct a quantitative post-processing framework with commonly available software components. Matlab was chosen as the programming/integration environment, and SPM was chosen as the coregistration component. Matlab routines were created to extract and concatenate the coregistration transforms, take the coregistered MRI sequences as inputs to the process, allow specification of the ROI, and store the voxel values to the database for statistical analysis. The functionality of the framework was validated using brain tumor MRI cases. The implementation of this quantitative post-processing framework enables intuitive creation of multiple parameters for each voxel, facilitating near real-time in-depth voxel-wise analysis. Our initial empirical evaluation of the framework is an increased usage of analysis requiring post-processing and increased number of simultaneous research activities by clinicians and researchers with non-technical backgrounds. We show that common software components can be utilized to implement an intuitive real-time quantitative post-processing framework, resulting in improved scalability and increased adoption of post-processing needed to answer important diagnostic questions.

Development of a quantitative method for the diagnosis of cirrhosis.

OBJECTIVE: To develop a novel non-invasive, quantitative approach utilizing computed tomography scans to predict cirrhosis. MATERIALS AND METHODS: A total of 105 patients (54 cirrhosis and 51 normal) who had CT scans within 6 months of a liver biopsy or were identified through a Trauma registry were included in this study. Patients were randomly divided into the training set (n = 81) and the validation set (n = 24). Each liver was segmented in a semi-automated fashion from a computed tomography scan using Mimics software. The resulting liver surfaces were saved as a stereo lithography mesh into an Oracle database, and analyzed in MATLAB(®) for morphological markers of cirrhosis. RESULTS: The best predictive model for diagnosing cirrhosis consisted of liver slice-bounding box slice ratio, the dimensions of the liver bounding box, liver slice area, slice perimeter, surface volume and adjusted surface area. With this model, we calculated an area under the receiver operating characteristic curve of 0.90 for the training set, and 0.91 for the validation set. For comparison, we calculated an area under the receiver operating characteristic curve of 0.70 for our dataset when we used the lab value based aspartate aminotransferase-platelet ratio index, another reported model for predicting cirrhosis. Last, by combining the aspartate aminotransferase-platelet ratio index and our model, we obtained an area under the receiving operating characteristic of 0.95. CONCLUSION: This study shows "proof of concept" that quantitative image analysis of livers on computed tomography scans may be utilized to predict cirrhosis in the absence of a liver biopsy.