Evaluation of computer-aided detection and diagnosis systems.

Computer-aided detection and diagnosis (CAD) systems are increasingly being used as an aid by clinicians for detection and interpretation of diseases. Computer-aided detection systems mark regions of an image that may reveal specific abnormalities and are used to alert clinicians to these regions during image interpretation. Computer-aided diagnosis systems provide an assessment of a disease using image-based information alone or in combination with other relevant diagnostic data and are used by clinicians as a decision support in developing their diagnoses. While CAD systems are commercially available, standardized approaches for evaluating and reporting their performance have not yet been fully formalized in the literature or in a standardization effort. This deficiency has led to difficulty in the comparison of CAD devices and in understanding how the reported performance might translate into clinical practice. To address these important issues, the American Association of Physicists in Medicine (AAPM) formed the Computer Aided Detection in Diagnostic Imaging Subcommittee (CADSC), in part, to develop recommendations on approaches for assessing CAD system performance. The purpose of this paper is to convey the opinions of the AAPM CADSC members and to stimulate the development of consensus approaches and "best practices" for evaluating CAD systems. Both the assessment of a standalone CAD system and the evaluation of the impact of CAD on end-users are discussed. It is hoped that awareness of these important evaluation elements and the CADSC recommendations will lead to further development of structured guidelines for CAD performance assessment. Proper assessment of CAD system performance is expected to increase the understanding of a CAD system's effectiveness and limitations, which is expected to stimulate further research and development efforts on CAD technologies, reduce problems due to improper use, and eventually improve the utility and efficacy of CAD in clinical practice.

Quality assurance and training procedures for computer-aided detection and diagnosis systems in clinical use.

Computer-aided detection/diagnosis (CAD) is increasingly used for decision support by clinicians for detection and interpretation of diseases. However, there are no quality assurance (QA) requirements for CAD in clinical use at present. QA of CAD is important so that end users can be made aware of changes in CAD performance both due to intentional or unintentional causes. In addition, end-user training is critical to prevent improper use of CAD, which could potentially result in lower overall clinical performance. Research on QA of CAD and user training are limited to date. The purpose of this paper is to bring attention to these issues, inform the readers of the opinions of the members of the American Association of Physicists in Medicine (AAPM) CAD subcommittee, and thus stimulate further discussion in the CAD community on these topics. The recommendations in this paper are intended to be work items for AAPM task groups that will be formed to address QA and user training issues on CAD in the future. The work items may serve as a framework for the discussion and eventual design of detailed QA and training procedures for physicists and users of CAD. Some of the recommendations are considered by the subcommittee to be reasonably easy and practical and can be implemented immediately by the end users; others are considered to be "best practice" approaches, which may require significant effort, additional tools, and proper training to implement. The eventual standardization of the requirements of QA procedures for CAD will have to be determined through consensus from members of the CAD community, and user training may require support of professional societies. It is expected that high-quality CAD and proper use of CAD could allow these systems to achieve their true potential, thus benefiting both the patients and the clinicians, and may bring about more widespread clinical use of CAD for many other diseases and applications. It is hoped that the awareness of the need for appropriate CAD QA and user training will stimulate new ideas and approaches for implementing such procedures efficiently and effectively as well as funding opportunities to fulfill such critical efforts.

Investigation of optimal use of computer-aided detection systems: the role of the "machine" in decision making process.

RATIONALE AND OBJECTIVES: The aim of this study was to explore different computerized models (the "machine") as a means to achieve optimal use of computer-aided detection (CAD) systems and to investigate whether these models can play a primary role in clinical decision making and possibly replace a clinician's subjective decision for combining his or her own assessment with that provided by a CAD system. MATERIALS AND METHODS: Data previously collected from a fully crossed, multiple-reader, multiple-case observer study with and without CAD by seven observers asked to identify simulated small masses on two separate sets of 100 mammographic images (low-contrast and high-contrast sets; ie, low-contrast and high-contrast simulated masses added to random locations on normal mammograms) were used. This allowed testing two relative sensitivities between the observers and CAD. Seven models that combined detection assessments from CAD standalone, unaided read, and CAD-aided read (second read and concurrent read) were developed using the leave-one-out technique for training and testing. These models were personalized for each observer. Detection performance accuracies were analyzed using the area under a portion of the free-response receiver-operating characteristic curve (AUFC), sensitivity, and number of false-positives per image. RESULTS: For the low-contrast set, the use of computerized models resulted in significantly higher AUFCs compared to the unaided read mode for all readers, whereas the increased AUFCs between CAD-aided (second and concurrent reads; ie, decisions made by the readers) and unaided read modes were statistically significant for a majority, but not all, of the readers (four and five of the seven readers, respectively). For the high-contrast set, there were no significant trends in the AUFCs whether or not a model was used to combine the original reading modes. Similar results were observed when using sensitivity as the figure of merit. However, the average number of false-positives per image resulting from the computerized models remained the same as that obtained from the unaided read modes. CONCLUSIONS: Individual computerized models (the machine) that combine image assessments from CAD standalone, unaided read, and CAD-aided read can increase detection performance compared to the reading done by the observer. However, relative sensitivity (ie, the difference in sensitivity between CAD standalone and unaided read) was a critical factor that determined incremental improvement in decision making, whether made by the observer or using computerized models.

Investigation of reading mode and relative sensitivity as factors that influence reader performance when using computer-aided detection software.

RATIONALE AND OBJECTIVES: The aim of this study was to investigate the effects of relative sensitivity (reader without computer-aided detection [CAD] vs stand-alone CAD) and reading mode on reader performance when using CAD software. MATERIALS AND METHODS: Two sets of 100 images (low-contrast and high-contrast sets) were created by adding low-contrast or high-contrast simulated masses to random locations in 100 normal mammograms. This produced a relative sensitivity, substantially less for the low-contrast set and similar for the high-contrast set. Seven readers reviewed every image in each set and specified location and probability scores using three reading modes (without CAD, second read with CAD, and concurrent read with CAD). Reader detection accuracy was analyzed using areas under free-response receiver operating characteristic curves, sensitivity, and the number of false-positive findings per image. RESULTS: For the low-contrast set, average differences in areas under free-response receiver operating characteristic curves, sensitivity, and false-positive findings per image without CAD were 0.02, 0.12, and 0.11, respectively, compared to second read and 0.05, 0.17, and 0.09 (not statistically significant), respectively, compared to concurrent read. For the high-contrast set, average differences were 0.002 (not statistically significant), 0.04, and 0.05, respectively, compared to second read and -0.004 (not statistically significant), 0.04, and 0.08 (not statistically significant), respectively, compared to concurrent read (all differences were statistically significant except as noted). Differences were greater in the low-contrast set than the high-contrast set. Differences between second read and concurrent read were not significant. CONCLUSIONS: Relative sensitivity is a critical factor that determines incremental improvement in reader performance when using CAD and appears to be more important than reading mode. Relative sensitivity may determine the clinical usefulness of CAD in different clinical applications and for different types of users.

Radial gradient-based segmentation of mammographic microcalcifications: observer evaluation and effect on CAD performance.

Precise segmentation of microcalcifications is essential in the development of accurate mammographic computer-aided diagnosis (CAD) schemes. We have designed a radial gradient-based segmentation method for microcalcifications, and compared it to both the region-growing segmentation method currently used in our CAD scheme and to the watershed segmentation method. Two observer studies were conducted to subjectively evaluate the proposed segmentation method. The first study (A) required observers to rate the segmentation accuracy on a 100-point scale. The second observer evaluation (B) was a preference study in which observers selected their preferred method from three displayed segmentation methods. In study A, the observers gave an average accuracy rating of 88 for the radial gradient-based and 50 for the region-growing segmentation method. In study B, the two observers selected the proposed method 56% and 62% of the time. We also investigated the effect of the proposed segmentation method on the performance of computerized classification scheme in differentiating malignant from benign clustered microcalcifications. The performances of the classification scheme using a linear discriminant analysis (LDA) or a Bayesian artificial neural network classifier both showed statistically significant improvements when using the proposed segmentation method. The areas under the receiver-operating characteristic curves for case-based performance when using the LDA classifier were 0.86 with the proposed segmentation method, 0.80 with the region-growing method, and 0.83 with the watershed method.

Correlation between mammographic density and volumetric fibroglandular tissue estimated on breast MR images.

Previous studies have found that mammographic breast density is highly correlated with breast cancer risk. Therefore, mammographic breast density may be considered as an important risk factor in studies of breast cancer treatments. In this paper, we evaluated the accuracy of using mammograms for estimating breast density by analyzing the correlation between the percent mammographic dense area and the percent glandular tissue volume as estimated from MR images. A dataset of 67 cases having MR images (coronal 3-D SPGR T1-weighted pre-contrast) and corresponding 4-view mammograms was used in this study. Mammographic breast density was estimated by an experienced radiologist and an automated image analysis tool, Mammography Density ESTimator (MDEST) developed previously in our laboratory. For the estimation of the percent volume of fibroglandular tissue in breast MR images, a semiautomatic method was developed to segment the fibroglandular tissue from each slice. The tissue volume was calculated by integration over all slices containing the breast. Interobserver variation was measured for 3 different readers. It was found that the correlation between every two of the three readers for segmentation of MR volumetric fibroglandular tissue was 0.99. The correlations between the percent volumetric fibroglandular tissue on MR images and the percent dense area of the CC and MLO views segmented by an experienced radiologist were both 0.91. The correlation between the percent volumetric fibroglandular tissue on MR images and the percent dense area of the CC and MLO views segmented by MDEST was 0.91 and 0.89, respectively. The root-mean-square (rms) residual ranged from 5.4% to 6.3%. The mean bias ranged from 3% to 6%. The high correlation indicates that changes in mammographic density may be a useful indicator of changes in fibroglandular tissue volume in the breast.

Breast cancer detection: evaluation of a mass-detection algorithm for computer-aided diagnosis -- experience in 263 patients.

PURPOSE: To evaluate the performance of a computer-aided diagnosis (CAD) mass-detection algorithm in marking preoperative masses. MATERIALS AND METHODS: Digitized mammograms were processed with an adaptive enhancement filter followed by a local border refinement stage. Features were then extracted from each detected structure and used to identify potential masses. The performance of the algorithm was evaluated in independent cases obtained from 263 patients from two institutions. Each case contained one or more pathologically proved breast masses. Contralateral mammograms obtained in the same patients that did not contain a visible lesion were used to estimate the CAD marker rate for the algorithm. The tradeoff between detection sensitivity and the number of CAD marks was analyzed in this study. RESULTS: Malignant masses were detected with the computer in 87% (135 of 156), 83% (130 of 156), and 77% (120 of 156) of the malignant cases at CAD marker rates of 1.5, 1.0, and 0.5 marks per mammogram, respectively. The difference between malignant mass-detection performance in subsets of cases collected at each institution was found to be less than 1%. The detection accuracy for benign masses was lower than that for malignant masses. CONCLUSION: This mass-detection algorithm had a high sensitivity for detection of malignant masses. It may be useful as a second opinion in mammographic interpretation.

Improvement of computerized mass detection on mammograms: fusion of two-view information.

Recent clinical studies have proved that computer-aided diagnosis (CAD) systems are helpful for improving lesion detection by radiologists in mammography. However, these systems would be more useful if the false-positive rate is reduced. Current CAD systems generally detect and characterize suspicious abnormal structures in individual mammographic images. Clinical experiences by radiologists indicate that screening with two mammographic views improves the detection accuracy of abnormalities in the breast. It is expected that the fusion of information from different mammographic views will improve the performance of CAD systems. We are developing a two-view matching method that utilizes the geometric locations, and morphological and textural features to correlate objects detected in two different views using a prescreening program. First, a geometrical model is used to predict the search region for an object in a second view from its location in the first view. The distance between the object and the nipple is used to define the search area. After pairing the objects in two views, textural and morphological characteristics of the paired objects are merged and similarity measures are defined. Linear discriminant analysis is then employed to classify each object pair as a true or false mass pair. The resulting object correspondence score is combined with its one-view detection score using a fusion scheme. The fusion information was found to improve the lesion detectability and reduce the number of FPs. In a preliminary study, we used a data set of 169 pairs of cranio-caudal (CC) and mediolateral oblique (MLO) view mammograms. For the detection of malignant masses on current mammograms, the film-based detection sensitivity was found to improve from 62% with a one-view detection scheme to 73% with the new two-view scheme, at a false-positive rate of 1 FP/image. The corresponding cased-based detection sensitivity improved from 77% to 91%.