Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction.

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Observer training for computer-aided detection of pulmonary nodules in chest radiography.

OBJECTIVES: To assess whether short-term feedback helps readers to increase their performance using computer-aided detection (CAD) for nodule detection in chest radiography. METHODS: The 140 CXRs (56 with a solitary CT-proven nodules and 84 negative controls) were divided into four subsets of 35; each were read in a different order by six readers. Lesion presence, location and diagnostic confidence were scored without and with CAD (IQQA-Chest, EDDA Technology) as second reader. Readers received individual feedback after each subset. Sensitivity, specificity and area under the receiver-operating characteristics curve (AUC) were calculated for readings with and without CAD with respect to change over time and impact of CAD. RESULTS: CAD stand-alone sensitivity was 59 % with 1.9 false-positives per image. Mean AUC slightly increased over time with and without CAD (0.78 vs. 0.84 with and 0.76 vs. 0.82 without CAD) but differences did not reach significance. The sensitivity increased (65 % vs. 70 % and 66 % vs. 70 %) and specificity decreased over time (79 % vs. 74 % and 80 % vs. 77 %) but no significant impact of CAD was found. CONCLUSION: Short-term feedback does not increase the ability of readers to differentiate true- from false-positive candidate lesions and to use CAD more effectively. KEY POINTS: • Computer-aided detection (CAD) is increasingly used as an adjunct for many radiological techniques. • Short-term feedback does not improve reader performance with CAD in chest radiography. • Differentiation between true- and false-positive CAD for low conspicious possible lesions proves difficult. • CAD can potentially increase reader performance for nodule detection in chest radiography.

Acute pulmonary embolism: effect of a computer-assisted detection prototype on diagnosis--an observer study.

PURPOSE: To assess the effect of a computer-assisted detection (CAD) prototype on observer performance for detection of acute pulmonary embolism (PE) with computed tomographic (CT) pulmonary angiography. MATERIALS AND METHODS: In this institutional review board-approved retrospective study, six observers with varying experience evaluated 158 PE-negative and 51 PE-positive CT pulmonary angiographic studies (mean age, 57 years; 111 women, 98 men) obtained consecutively during nights and weekends. Observers were asked to determine the presence of PE and to rank their diagnostic confidence without CAD and subsequently with CAD within a single reading session. Reading time was separately measured for both readings. Reader data were compared with an independent standard established by two readers, with a third in case of discordant results. Statistical evaluation was performed on a per-patient basis by using logistic regression for repeated measurements and Pearson correlation. RESULTS: With CAD, there was a significant increase in readers' sensitivity (P = .014) without loss of specificity (P = .853) on a per-patient basis. CAD assisted the readers in correcting an initial false-negative diagnosis in 15 cases, with the most proximal embolus at the segmental level in four cases and at the subsegmental level in 11 cases. In eight cases, readers accepted false-positive CAD candidate lesions on scans negative for PE, and in one case, a reader dismissed a true-positive finding. Reading time was extended by a mean of 22 seconds with the use of CAD. CONCLUSION: At the expense of increased reading time, CAD has the potential to increase reader sensitivity for detecting segmental and subsegmental PE without significant loss of specificity.