Improving the Reliability of Peer Review Without a Gold Standard.

Peer review plays a crucial role in accreditation and credentialing processes as it can identify outliers and foster a peer learning approach, facilitating error analysis and knowledge sharing. However, traditional peer review methods may fall short in effectively addressing the interpretive variability among reviewing and primary reading radiologists, hindering scalability and effectiveness. Reducing this variability is key to enhancing the reliability of results and instilling confidence in the review process. In this paper, we propose a novel statistical approach called "Bayesian Inter-Reviewer Agreement Rate" (BIRAR) that integrates radiologist variability. By doing so, BIRAR aims to enhance the accuracy and consistency of peer review assessments, providing physicians involved in quality improvement and peer learning programs with valuable and reliable insights. A computer simulation was designed to assign predefined interpretive error rates to hypothetical interpreting and peer-reviewing radiologists. The Monte Carlo simulation then sampled (100 samples per experiment) the data that would be generated by peer reviews. The performances of BIRAR and four other peer review methods for measuring interpretive error rates were then evaluated, including a method that uses a gold standard diagnosis. Application of the BIRAR method resulted in 93% and 79% higher relative accuracy and 43% and 66% lower relative variability, compared to "Single/Standard" and "Majority Panel" peer review methods, respectively. Accuracy was defined by the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. Variability was defined by the 95% CI around the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. BIRAR is a practical and scalable peer review method that produces more accurate and less variable assessments of interpretive quality by accounting for variability within the group's radiologists, implicitly applying a standard derived from the level of consensus within the group across various types of interpretive findings.

Automated detection of critical findings in multi-parametric brain MRI using a system of 3D neural networks.

With the rapid growth and increasing use of brain MRI, there is an interest in automated image classification to aid human interpretation and improve workflow. We aimed to train a deep convolutional neural network and assess its performance in identifying abnormal brain MRIs and critical intracranial findings including acute infarction, acute hemorrhage and mass effect. A total of 13,215 clinical brain MRI studies were categorized to training (74%), validation (9%), internal testing (8%) and external testing (8%) datasets. Up to eight contrasts were included from each brain MRI and each image volume was reformatted to common resolution to accommodate for differences between scanners. Following reviewing the radiology reports, three neuroradiologists assigned each study to abnormal vs normal, and identified three critical findings including acute infarction, acute hemorrhage, and mass effect. A deep convolutional neural network was constructed by a combination of localization feature extraction (LFE) modules and global classifiers to identify the presence of 4 variables in brain MRIs including abnormal, acute infarction, acute hemorrhage and mass effect. Training, validation and testing sets were randomly defined on a patient basis. Training was performed on 9845 studies using balanced sampling to address class imbalance. Receiver operating characteristic (ROC) analysis was performed. The ROC analysis of our models for 1050 studies within our internal test data showed AUC/sensitivity/specificity of 0.91/83%/86% for normal versus abnormal brain MRI, 0.95/92%/88% for acute infarction, 0.90/89%/81% for acute hemorrhage, and 0.93/93%/85% for mass effect. For 1072 studies within our external test data, it showed AUC/sensitivity/specificity of 0.88/80%/80% for normal versus abnormal brain MRI, 0.97/90%/97% for acute infarction, 0.83/72%/88% for acute hemorrhage, and 0.87/79%/81% for mass effect. Our proposed deep convolutional network can accurately identify abnormal and critical intracranial findings on individual brain MRIs, while addressing the fact that some MR contrasts might not be available in individual studies.

GAMER MRI: Gated-attention mechanism ranking of multi-contrast MRI in brain pathology.

INTRODUCTION: During the last decade, a multitude of novel quantitative and semiquantitative MRI techniques have provided new information about the pathophysiology of neurological diseases. Yet, selection of the most relevant contrasts for a given pathology remains challenging. In this work, we developed and validated a method, Gated-Attention MEchanism Ranking of multi-contrast MRI in brain pathology (GAMER MRI), to rank the relative importance of MR measures in the classification of well understood ischemic stroke lesions. Subsequently, we applied this method to the classification of multiple sclerosis (MS) lesions, where the relative importance of MR measures is less understood. METHODS: GAMER MRI was developed based on the gated attention mechanism, which computes attention weights (AWs) as proxies of importance of hidden features in the classification. In the first two experiments, we used Trace-weighted (Trace), apparent diffusion coefficient (ADC), Fluid-Attenuated Inversion Recovery (FLAIR), and T1-weighted (T1w) images acquired in 904 acute/subacute ischemic stroke patients and in 6,230 healthy controls and patients with other brain pathologies to assess if GAMER MRI could produce clinically meaningful importance orders in two different classification scenarios. In the first experiment, GAMER MRI with a pretrained convolutional neural network (CNN) was used in conjunction with Trace, ADC, and FLAIR to distinguish patients with ischemic stroke from those with other pathologies and healthy controls. In the second experiment, GAMER MRI with a patch-based CNN used Trace, ADC and T1w to differentiate acute ischemic stroke lesions from healthy tissue. The last experiment explored the performance of patch-based CNN with GAMER MRI in ranking the importance of quantitative MRI measures to distinguish two groups of lesions with different pathological characteristics and unknown quantitative MR features. Specifically, GAMER MRI was applied to assess the relative importance of the myelin water fraction (MWF), quantitative susceptibility mapping (QSM), T1 relaxometry map (qT1), and neurite density index (NDI) in distinguishing 750 juxtacortical lesions from 242 periventricular lesions in 47 MS patients. Pair-wise permutation t-tests were used to evaluate the differences between the AWs obtained for each quantitative measure. RESULTS: In the first experiment, we achieved a mean test AUC of 0.881 and the obtained AWs of FLAIR and the sum of AWs of Trace and ADC were 0.11 and 0.89, respectively, as expected based on previous knowledge. In the second experiment, we achieved a mean test F1 score of 0.895 and a mean AW of Trace = 0.49, of ADC = 0.28, and of T1w = 0.23, thereby confirming the findings of the first experiment. In the third experiment, MS lesion classification achieved test balanced accuracy = 0.777, sensitivity = 0.739, and specificity = 0.814. The mean AWs of T1map, MWF, NDI, and QSM were 0.29, 0.26, 0.24, and 0.22 (p < 0.001), respectively. CONCLUSIONS: This work demonstrates that the proposed GAMER MRI might be a useful method to assess the relative importance of MRI measures in neurological diseases with focal pathology. Moreover, the obtained AWs may in fact help to choose the best combination of MR contrasts for a specific classification problem.

Extraction of airways from CT (EXACT'09).

This paper describes a framework for establishing a reference airway tree segmentation, which was used to quantitatively evaluate fifteen different airway tree extraction algorithms in a standardized manner. Because of the sheer difficulty involved in manually constructing a complete reference standard from scratch, we propose to construct the reference using results from all algorithms that are to be evaluated. We start by subdividing each segmented airway tree into its individual branch segments. Each branch segment is then visually scored by trained observers to determine whether or not it is a correctly segmented part of the airway tree. Finally, the reference airway trees are constructed by taking the union of all correctly extracted branch segments. Fifteen airway tree extraction algorithms from different research groups are evaluated on a diverse set of twenty chest computed tomography (CT) scans of subjects ranging from healthy volunteers to patients with severe pathologies, scanned at different sites, with different CT scanner brands, models, and scanning protocols. Three performance measures covering different aspects of segmentation quality were computed for all participating algorithms. Results from the evaluation showed that no single algorithm could extract more than an average of 74% of the total length of all branches in the reference standard, indicating substantial differences between the algorithms. A fusion scheme that obtained superior results is presented, demonstrating that there is complementary information provided by the different algorithms and there is still room for further improvements in airway segmentation algorithms.

Automated CT scoring of airway diseases: preliminary results.

RATIONALE AND OBJECTIVES: The aim of this study was to retrospectively evaluate an automated global scoring system for evaluating the extent and severity of disease in a known cohort of patients with documented bronchiectasis. On the basis of a combination of validated three-dimensional automated algorithms for bronchial tree extraction and quantitative airway measurements, global scoring combines the evaluation of bronchial lumen-to-artery ratios and bronchial wall-to-artery ratios, as well as the detection of mucoid-impacted airways. The result is an automatically generated global computed tomographic (CT) score designed to simplify and standardize the interpretation of scans in patients with chronic airway infections. MATERIALS AND METHODS: Twenty high-resolution CT data sets were used to evaluate an automated CT scoring method that combines algorithms for airway quantitative analysis that have been individually tested and validated. Patients with clinically documented atypical mycobacterial infections with visually assessed CT evidence of bronchiectasis varying from mild to severe were retrospectively selected. These data sets were evaluated by two independent experienced radiologists and by computer scoring, with the results compared statistically, including Spearman's rank correlation. RESULTS: Computer evaluation required 3 to 5 minutes per data set, compared to 12 to 15 minutes for manual scoring. Initial Spearman's rank tests showed positive correlations between automated and readers' global scores (r = 0.609, P = .01), extent of bronchiectasis (r = 0.69, P = .0004), and severity of bronchiectasis (r = 0.61, P = .01), while mucus plug detection showed a lesser extent of positive correlation between the scoring methods (r = 0.42, P = .07) and wall thickness a negative weak correlation (r = -0.10, P = .40). Further retrospective review of 24 lobes in which wall thickness scores showed the highest discrepancy between manual and automated methods was then performed, using electronic calipers and perpendicular cross-sections to reassess airway measurements. This resulted in an improved Spearman's rank correlation to r = 0.62 (P = .009), for a global score of r = 0.67 (P = .001). CONCLUSION: Automated computerized scoring shows considerable promise for providing a standardized, quantitative method, demonstrating overall good correlation with the results of experienced readers' evaluation of the extent and severity of bronchiectasis. It is speculated that this technique may also be applicable to a wide range of other conditions associated with chronic bronchial inflammation, as well as of potential value for monitoring response to therapy in these same populations.

Computer-aided diagnosis of the airways: beyond nodule detection.

Although to date, the major impetus for the development of computer-assisted diagnosis (CAD) has been the detection of pulmonary nodules, CAD should properly be viewed as a potential tool for assisting radiologic interpretation of the entire gamut of chest diseases, including not just enhanced detection of disease but also characterization and quantification, ideally leading to improved patient management. The use of CAD to improve visualization of the airways using advanced computer techniques, including sophisticated methods for obtaining 3-dimensional segmentation of the central airways and, in particular, the development of virtual bronchoscopy has been recently studied. In this paper, the authors review the development of a specific series of CAD applications enabling automated identification and characterization of chronically inflamed airways. The advantages to the use of computer methodologies to quantify peripheral airway disease include reproducible visualization methods to display the location, severity, and extent of airway dilatation, bronchial wall thickening, and the presence of mucoid impacted airways. Currently, a number of semiquantitative global scoring systems have been proposed to assess disease extent and severity in patients with bronchiectasis. Unfortunately, with the exception of patients with cystic fibrosis, these are rarely if ever employed, largely owing to the considerable inconvenience of measuring individual airway dimensions and computing a global score. It is apparent that for this specific purpose, CAD may be ideally suited. Automated staging allows for more complete assessment of the entire bronchial tree while providing improved standardization and eliminating an otherwise tedious and time-consuming task.

Boundary-specific cost functions for quantitative airway analysis.

Computed tomography (CT) images of the lungs provide high resolution views of the airways. Quantitative measurements such as lumen diameter and wall thickness help diagnose and localize airway diseases, assist in surgical planning, and determine progress of treatment. Automated quantitative analysis of such images is needed due to the number of airways per patient. We present an approach involving dynamic programming coupled with boundary-specific cost functions that is capable of differentiating inner and outer borders. The method allows for precise delineation of the inner lumen and outer wall. The results are demonstrated on synthetic data, evaluated on human datasets compared to human operators, and verified on phantom CT scans to sub-voxel accuracy.