Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction.

Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.

Economic evaluation of an expert examiner and different ultrasound models in the diagnosis of ovarian cancer.

The Risk of Malignancy Index (RMI) is commonly used to diagnose adnexal masses. The aim of the present study was to determine the cost-effectiveness of the RMI compared with subjective assessment (SA) by an expert and the following novel ultrasound models: Cost-effectiveness and budget impact analyses were performed from a societal perspective. A decision tree was constructed, and short-term costs and effects were examined in women with adnexal masses. Sensitivity, specificity and the costs of diagnostic strategies were incorporated. Incremental cost-effectiveness ratios were expressed as costs/additional percentage of correctly diagnosed patients. Probabilistic and deterministic sensitivity analyses were performed. Effectiveness was highest for SA (90.7% [95% confidence interval = 77.3-100]), with a cost saving of 5.0% (-€398 per patient [-€1403 to 549]) compared with the RMI. The costs of SR + SA were the lowest (€7180 [6072-8436]), resulting in a cost saving of 9.0% (-€709 per patient [-€1628 to 236]) compared with the RMI, with an effectiveness of 89.6% (75.8-100). SR + SA showed the highest probability of being the most cost-effective when willingness-to-pay was <€350 per additional percentage of correctly diagnosed patients. The RMI had low cost-effectiveness probabilities (<3%) and was inferior to SA, SR + SA and LR2. Budget impact in the Netherlands compared with that of the RMI varied between a cost saving of €4.67 million for SR + SA and additional costs of €3.83 million when implementing ADNEX (cut-off: 10%). The results were robust when tested in sensitivity analyses. Although SA is the best strategy in terms of diagnostic accuracy, SR + SA might be preferred from a cost-effectiveness perspective.

Investigating the performance and cost-effectiveness of the simple ultrasound-based rules compared to the risk of malignancy index in the diagnosis of ovarian cancer (SUBSONiC-study): protocol of a prospective multicenter cohort study in the Netherlands.

BACKGROUND: Estimating the risk of malignancy is essential in the management of adnexal masses. An accurate differential diagnosis between benign and malignant masses will reduce morbidity and costs due to unnecessary operations, and will improve referral to a gynecologic oncologist for specialized cancer care, which improves outcome and overall survival. The Risk of Malignancy Index is currently the most commonly used method in clinical practice, but has a relatively low diagnostic accuracy (sensitivity 75-80% and specificity 85-90%). Recent reports show that other methods, such as simple ultrasound-based rules, subjective assessment and (Diffusion Weighted) Magnetic Resonance Imaging might be superior to the RMI in the pre-operative differentiation of adnexal masses. METHODS/DESIGN: A prospective multicenter cohort study will be performed in the south of The Netherlands. A total of 270 women diagnosed with at least one pelvic mass that is suspected to be of ovarian origin who will undergo surgery, will be enrolled. We will apply the Risk of Malignancy Index with a cut-off value of 200 and a two-step triage test consisting of simple ultrasound-based rules supplemented -if necessary- with either subjective assessment by an expert sonographer or Magnetic Resonance Imaging with diffusion weighted sequences, to characterize the adnexal masses. The histological diagnosis will be the reference standard. Diagnostic performances will be expressed as sensitivity, specificity, positive and negative predictive values and likelihood ratios. DISCUSSION: We hypothesize that this two-step triage test, including the simple ultrasound-based rules, will have better diagnostic accuracy than the Risk of Malignancy Index and therefore will improve the management of women with adnexal masses. Furthermore, we expect this two-step test to be more cost-effective. If the hypothesis is confirmed, the results of this study could have major effects on current guidelines and implementation of the triage test in daily clinical practice could be a possibility. TRIAL REGISTRATION: ClinicalTrials.gov: registration number NCT02218502.