ABSTRACT
OBJECTIVE: Artificial intelligence (AI)-based triage algorithms may improve cancer detection and expedite radiologist workflow. To this end, the performance of a commercial AI-based triage algorithm on screening mammograms was evaluated across breast densities and lesion types. METHODS: This retrospective, IRB-exempt, multicenter, multivendor study examined 1255 screening 4-view mammograms (400 positive and 855 negative studies). Images were anonymized by providing institutions and analyzed by a commercially available AI algorithm (cmTriage, CureMetrix, La Jolla, CA) that performed retrospective triage at the study level by flagging exams as "suspicious" or not. Sensitivities and specificities with confidence intervals were derived from area under the curve (AUC) calculations. RESULTS: The algorithm demonstrated an AUC of 0.95 (95% CI: 0.94-0.96) for case identification. Area under the curve held across densities (0.95) and lesion types (masses: 0.94 [95% CI: 0.92-0.96] or microcalcifications: 0.97 [95% CI: 0.96-0.99]). The algorithm has a default sensitivity of 93% (95% CI: 95.6%-90.5%) with specificity of 76.3% (95% CI: 79.2%-73.4%). To evaluate real-world performance, a sensitivity of 86.9% (95% CI: 83.6%-90.2%) was tested, as observed for practicing radiologists by the Breast Cancer Surveillance Consortium (BCSC) study. The resulting specificity was 88.5% (95% CI: 86.4%-90.7%), similar to the BCSC specificity of 88.9%, indicating performance comparable to real-world results. CONCLUSION: When tested for lesion detection, an AI-based triage software can perform at the level of practicing radiologists. Drawing attention to suspicious exams may improve reader specificity and help streamline radiologist workflow, enabling faster turnaround times and improving care.
Subject(s)
Artificial Intelligence , Mammography , Triage , Algorithms , Mammography/methods , Retrospective Studies , Triage/methodsABSTRACT
To determine whether cmAssist™, an artificial intelligence-based computer-aided detection (AI-CAD) algorithm, can be used to improve radiologists' sensitivity in breast cancer screening and detection. A blinded retrospective study was performed with a panel of seven radiologists using a cancer-enriched data set from 122 patients that included 90 false-negative mammograms obtained up to 5.8 years prior to diagnosis and 32 BIRADS 1 and 2 patients with a 2-year follow-up of negative diagnosis. The mammograms were performed between February 7, 2008 (earliest) and January 8, 2016 (latest), and were all originally interpreted as negative in conjunction with R2 ImageChecker CAD, version 10.0. In this study, the readers analyzed the 122 studies before and after review of cmAssist™, an AI-CAD software for mammography. The statistical significance of our findings was evaluated using Student's t test and bootstrap statistical analysis. There was a substantial and significant improvement in radiologist accuracy with use of cmAssist, as demonstrated in the 7.2% increase in the area-under-the-curve (AUC) of the receiver operating characteristic (ROC) curve with two-sided p value < 0.01 for the reader group. All radiologists showed a significant improvement in their cancer detection rate (CDR) with the use of cmAssist (two-sided p value = 0.030, confidence interval = 95%). The readers detected between 25 and 71% (mean 51%) of the early cancers without assistance. With cmAssist, the overall reader CDR was 41 to 76% (mean 62%). The percentage increase in CDR for the reader panel was significant, ranging from 6 to 64% (mean 27%) with the use of cmAssist. There was less than 1% increase in the readers' false-positive recalls with use of cmAssist. With the use of cmAssist TM, there was a substantial and statistically significant improvement in radiologists' accuracy and sensitivity for detection of cancers that were originally missed. The percentage increase in CDR for the radiologists in the reader panel ranged from 6 to 64% (mean 27%) with the use of cmAssist, with negligible increase in false-positive recalls.
