Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance.

In radiology, natural language processing (NLP) allows the extraction of valuable information from radiology reports. It can be used for various downstream tasks such as quality improvement, epidemiological research, and monitoring guideline adherence. Class imbalance, variation in dataset size, variation in report complexity, and algorithm type all influence NLP performance but have not yet been systematically and interrelatedly evaluated. In this study, we investigate these factors on the performance of four types [a fully connected neural network (Dense), a long short-term memory recurrent neural network (LSTM), a convolutional neural network (CNN), and a Bidirectional Encoder Representations from Transformers (BERT)] of deep learning-based NLP. Two datasets consisting of radiologist-annotated reports of both trauma radiographs (n = 2469) and chest radiographs and computer tomography (CT) studies (n = 2255) were split into training sets (80%) and testing sets (20%). The training data was used as a source to train all four model types in 84 experiments (Fracture-data) and 45 experiments (Chest-data) with variation in size and prevalence. The performance was evaluated on sensitivity, specificity, positive predictive value, negative predictive value, area under the curve, and F score. After the NLP of radiology reports, all four model-architectures demonstrated high performance with metrics up to > 0.90. CNN, LSTM, and Dense were outperformed by the BERT algorithm because of its stable results despite variation in training size and prevalence. Awareness of variation in prevalence is warranted because it impacts sensitivity and specificity in opposite directions.

Machine learning based natural language processing of radiology reports in orthopaedic trauma.

OBJECTIVES: To compare different Machine Learning (ML) Natural Language Processing (NLP) methods to classify radiology reports in orthopaedic trauma for the presence of injuries. Assessing NLP performance is a prerequisite for downstream tasks and therefore of importance from a clinical perspective (avoiding missed injuries, quality check, insight in diagnostic yield) as well as from a research perspective (identification of patient cohorts, annotation of radiographs). METHODS: Datasets of Dutch radiology reports of injured extremities (n = 2469, 33% fractures) and chest radiographs (n = 799, 20% pneumothorax) were collected in two different hospitals and labeled by radiologists and trauma surgeons for the presence or absence of injuries. NLP classification was applied and optimized by testing different preprocessing steps and different classifiers (Rule-based, ML, and Bidirectional Encoder Representations from Transformers (BERT)). Performance was assessed by F1-score, AUC, sensitivity, specificity and accuracy. RESULTS: The deep learning based BERT model outperforms all other classification methods which were assessed. The model achieved an F1-score of (95 ± 2)% and accuracy of (96 ± 1)% on a dataset of simple reports (n= 2469), and an F1 of (83 ± 7)% with accuracy (93 ± 2)% on a dataset of complex reports (n= 799). CONCLUSION: BERT NLP outperforms traditional ML and rule-base classifiers when applied to Dutch radiology reports in orthopaedic trauma.

Improvement of radiology reporting in a clinical cancer network: impact of an optimised multidisciplinary workflow.

PURPOSE: To assess the effectiveness of implementing a quality improvement project in a clinical cancer network directed at the response assessment of oncology patients according to RECIST-criteria. METHODS: Requests and reports of computed tomography (CT) studies from before (n = 103) and after (n = 112) implementation of interventions were compared. The interventions consisted of: a multidisciplinary working agreement with a clearly described workflow; subspecialisation of radiologists; adaptation of the Picture Archiving and Communication System (PACS); structured reporting. RESULTS: The essential information included in the requests and the reports improved significantly after implementation of the interventions. In the requests, mentioning start date increased from 2% to 49%; date of baseline CT from 7% to 64%; nadir date from 1% to 41%. In the reports, structured layout increased from 14% to 86%; mentioning target lesions from 18% to 80% and non-target lesions from 11% to 80%; measurements stored in PACS increased from 76% to 97%; labelled key images from 38% to 95%; all p values < 0.001. CONCLUSION: The combination of implementation of an optimised workflow, subspecialisation and structured reporting led to significantly better quality radiology reporting for oncology patients receiving chemotherapy. The applied multifactorial approach can be used within other radiology subspeciality areas as well. KEY POINTS: • Undeveloped subspecialisation makes adherence to RECIST guidelines difficult in general hospitals. • A clinical cancer network provides opportunities to improve healthcare. • Optimised workflow, subspecialisation and structured reporting substantially improve request and report quality. • Good interdisciplinary communication between oncologists, radiologists and others contributes to quality improvement.

Perception of radiology reporting efficacy by neurologists in general and university hospitals.

AIM: To investigate how neurologists perceive the value of the radiology report and to analyse the relation with the neurologists own expertise in radiology and the level of subspecialisation of radiologists. MATERIALS AND METHODS: A web-based survey was distributed to neurologists. The level of subspecialisation was assessed by the percentage of fellowship-trained radiologists and the percentage of radiologists that were members of the Dutch Society of Neuroradiology. RESULTS: Most neurologists interpret all computed tomography (CT) and magnetic resonance imaging (MRI) studies themselves, and their self-confidence in making correct interpretations is high. Residents gave higher scores than neurologists for "Radiologist report answers the question" (p=0.039) and for "Radiologist reports give helpful advice" (p=0.001). Neurologists from university hospitals stated more frequently that the report answered their questions than neurologists from general hospitals (p=0.008). The general appreciation for radiology reports was higher for neurologists from university hospitals than from general hospitals (8.2 versus 7.2; p=0.003). Radiologists at university hospitals have a higher level of subspecialisation than those at general hospitals. CONCLUSION: Subspecialisation of radiologists leads to higher quality of radiology reporting as perceived by neurologists. Because of their expertise in radiology, neurologists are valuable sources of feedback for radiologists. Paying attention to the clinical questions and giving advice tailored to the needs of the referring physicians are opportunities to improve radiology reporting.

Implementation and Validation of PACS Integrated Peer Review for Discrepancy Recording of Radiology Reporting.

The purpose of this work is to demonstrate the possibility of implementation of a PACS-integrated peer review system based on RADPEER™ classification providing a step-wise implementation plan utilizing features already present in the standard PACS implementation and without the requirement of additional software development. Furthermore, we show the usage and effects of the system during the first 30 months of usage. To allow fast and easy implementation into the daily workflow the key-word feature of the PACS was used. This feature allows to add a key-word to an imaging examination for easy searching in the PACS database (e.g. by entering keywords for different kinds of pathology). For peer review we implemented a keyword structure including a code for each of the existing RADPEER™ scoring language terms and a keyword with the phrase "second reading" followed by the name of the individual radiologist. The use of the short-keys to enter the codes in relation to the peer review was a simple to use solution. During the study 599 reports were peer reviewed. The active participation in this study of the radiologists varies and ranges from 3 to 327 reviews per radiologist. The number of peer review is highest in CT and CR. There are no significant technical obstacles to implement a PACS-integrated RADPEER™ -system based on key-words allowing easy integration of peer review into the daily routine without the requirement of additional software. Peer review implemented in a non-random setting based on relevant priors could already help in increasing the quality of radiological reporting and serve as continuing education among peers. Decisiveness, tact and trust are needed to promote use of the system and collaborative discussion of the results by radiologist.