How to discriminate between computer-aided and computer-hindered decisions: a case study in mammography.

BACKGROUND: Computer aids can affect decisions in complex ways, potentially even making them worse; common assessment methods may miss these effects. We developed a method for estimating the quality of decisions, as well as how computer aids affect it, and applied it to computer-aided detection (CAD) of cancer, reanalyzing data from a published study where 50 professionals ("readers") interpreted 180 mammograms, both with and without computer support. METHOD: We used stepwise regression to estimate how CAD affected the probability of a reader making a correct screening decision on a patient with cancer (sensitivity), thereby taking into account the effects of the difficulty of the cancer (proportion of readers who missed it) and the reader's discriminating ability (Youden's determinant). Using regression estimates, we obtained thresholds for classifying a posteriori the cases (by difficulty) and the readers (by discriminating ability). RESULTS: Use of CAD was associated with a 0.016 increase in sensitivity (95% confidence interval [CI], 0.003-0.028) for the 44 least discriminating radiologists for 45 relatively easy, mostly CAD-detected cancers. However, for the 6 most discriminating radiologists, with CAD, sensitivity decreased by 0.145 (95% CI, 0.034-0.257) for the 15 relatively difficult cancers. CONCLUSIONS: Our exploratory analysis method reveals unexpected effects. It indicates that, despite the original study detecting no significant average effect, CAD helped the less discriminating readers but hindered the more discriminating readers. Such differential effects, although subtle, may be clinically significant and important for improving both computer algorithms and protocols for their use. They should be assessed when evaluating CAD and similar warning systems.

Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography.

RATIONALE AND OBJECTIVES: To investigate the effects of incorrect computer output on the reliability of the decisions of human users. This work followed an independent UK clinical trial that evaluated the impact of computer-aided detection(CAD) in breast screening. The aim was to use data from this trial to feed into probabilistic models (similar to those used in "reliability engineering") which would detect and assess possible ways of improving the human-CAD interaction. Some analyses required extra data; therefore, two supplementary studies were conducted. Study 1 was designed to elucidate the effects of computer failure on human performance. Study 2 was conducted to clarify unexpected findings from Study 1. MATERIALS AND METHODS: In Study 1, 20 film readers viewed 60 sets of mammograms (30 of which contained cancer) and provided "recall/no recall" decisions for each case. Computer output for each case was available to the participants. The test set was designed to contain an unusually large proportion (50%) of cancers for which CAD had generated incorrect output. In Study 2, 19 different readers viewed the same set of cases in similar conditions except that computer output was not available. RESULTS: The average sensitivity of readers in Study 1 (with CAD) was significantly lower than the average sensitivity of read-ers in Study 2 (without CAD). The difference was most marked for cancers for which CAD failed to provide correct prompting. CONCLUSION: Possible automation bias effects in CAD use deserve further study because they may degrade human decision-making for some categories of cases under certain conditions. This possibility should be taken into account in the assessment and design of CAD tools.