Impact of the new density reporting laws: radiologist perceptions and actual behavior.

RATIONALE AND OBJECTIVES: To assess radiologists' perceptions of how the new Breast Density Notification Act (BDNA) of Pennsylvania would affect their breast density reporting and their actual reporting patterns after implementation. MATERIALS AND METHODS: Under an institutional review board-approved protocol, we surveyed 21 radiologists about how they believe the new law affected their breast density reporting patterns and analyzed actual changes for 16 respondents before and after the law took effect. Three hundred consecutive reports were assessed for each radiologist before and after the effective date. The distributions of reported density Breast Imaging Reporting and Data System (BI-RADS) (1-4) were compared using a type III test in the context of an ordinal mixed model accounting for between-reader variability and adjusting for age (PROC GLIMMIX, SAS, version 9.3) using a two-sided .05 significance level. RESULTS: Seventeen radiologists responded to the survey; however, one retired shortly after responding. Of the 16 respondents, 56% (nine of 16) did not favor the law, 13% (two of 16) were in favor, and 31% (five of 16) were neutral. The fraction perceived that after implementation, they rated more, equally, or less frequently breasts as scattered fibroglandular densities (BI-RADS 2) versus heterogeneously dense rating (BI-RADS 3) was 50% (eight of 16), 44% (seven of 16), and 6% (one of 16), respectively. In practice, 44% (seven of 16) performed differently than their survey answers. Fourteen of 16 radiologists increased the frequency of reported BI-RADS 2 scores after BDNA implementation with seven having statistically significant (P < .05) increases after adjusting for age differences. CONCLUSIONS: Radiologists' reporting patterns changed, at least for a short duration, after the new density reporting law and for some of the radiologists in an unexpected way.

Classification of thyroid nodules using a resonance-frequency-based electrical impedance spectroscopy: a preliminary assessment.

BACKGROUND: Ultrasound and ultrasound-guided fine-needle aspiration biopsy are considered the most effective approaches for both identifying and classifying thyroid nodules. However, despite continuing improvements in scanner technology and refinements in ultrasound/cytological classification guidelines, indeterminate findings still lead to diagnostic lobectomy under general anesthesia. This study aims to investigate the feasibility of applying a modified noninvasive electrical impedance spectroscopy (EIS) approach to classifying thyroid nodules. METHOD: To increase nodule classification sensitivity, we developed a new EIS-based model that introduces an optimized inductance component, which increases the measured signal-to-noise ratio of capacitance variation in and about thyroid nodules. Our model then measures the change of resonance frequency when the positive reactance of the system inductor cancels out the negative reactance of the nodule capacitance in a multi-frequency electrical signal scan. The system is termed "resonance-frequency-based electrical impedance spectroscopy" (REIS). A portable REIS system with multiple probes was assembled and preliminarily tested in our clinical facility. From an ongoing prospective study, an initial data set of 160 REIS examinations including 27 verified cancer cases was used. From the data set, a number of EIS signal features was extracted and analyzed. A multi-feature-based Bayesian Belief Network was built to classify the detected thyroid nodules. A receiver operating characteristic data analysis method was applied to evaluate classification performance. RESULTS: The results showed that (i) the median resonance frequency measured by the probe nearest to malignant nodules was in general lower than that measured in benign cases, and (ii) the median descending slope of EIS signal sweep curves computed from cancer cases was larger than that computed from benign cases. The Bayesian Belief Network yielded a classification performance as measured by the area under the receiver operating characteristic curve of 0.794 [with a 95% confidence interval of 0.709-0.863]. CONCLUSIONS: The study demonstrates that noninvasive measurement of REIS signal features may potentially provide useful supplementary information to assist in classifying between malignant and benign thyroid nodules. Such an approach may ultimately lead to a reduction in the number of unnecessary thyroid surgeries.

Bilateral mammographic density asymmetry and breast cancer risk: a preliminary assessment.

To improve efficacy of breast cancer screening and prevention programs, it requires a risk assessment model with high discriminatory power. This study aimed to assess classification performance of using computed bilateral mammographic density asymmetry to predict risk of individual women developing breast cancer in near-term. The database includes 451 cases with multiple screening mammography examinations. The first (baseline) examinations of all case were interpreted negative. In the next sequential examinations, 187 cases developed cancer or surgically excised high-risk lesions, 155 remained negative (not-recalled), and 109 were recalled benign cases. From each of two bilateral cranio-caudal view images acquired from the baseline examination, we computed two features of average pixel value and local pixel value fluctuation. We then computed mean and difference of each feature computed from two images. When applying the computed features and other two risk factors (woman's age and subjectively rated mammographic density) to predict risk of cancer development, areas under receiver operating characteristic curves (AUC) were computed to evaluate the discriminatory/classification performance. The AUCs are 0.633±0.030, 0.535±0.036, 0.567±0.031, and 0.719±0.027 when using woman's age, subjectively rated, computed mean and asymmetry of mammographic density, to classify between two groups of cancer-verified and negative cases, respectively. When using an equal-weighted fusion method to combine woman's age and computed density asymmetry, AUC increased to 0.761±0.025 (p<0.05). The study demonstrated that bilateral mammographic density asymmetry could be a significantly stronger risk factor associated to the risk of women developing breast cancer in near-term than woman's age and assessed mean mammographic density.

Agreement of the order of overall performance levels under different reading paradigms.

RATIONALE AND OBJECTIVES: To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms. MATERIALS AND METHODS: We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations. RESULTS: All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733). CONCLUSION: The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

Binary and multi-category ratings in a laboratory observer performance study: a comparison.

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

The prevalence effect in a laboratory environment: Changing the confidence ratings.

RATIONALE AND OBJECTIVES: We sought to assess whether or not prevalence levels affected the confidence ratings of readers during the interpretation of cases in a laboratory receiver operating characteristic-type observer performance study. MATERIALS AND METHODS: We reanalyzed a previously conducted observer performance study that included 14 readers and 5 different levels of prevalence. The previous study yielded the observation that in the laboratory we could not detect a "prevalence effect" in terms of differences in areas under the receiver operating characteristic curves. The detection ratings (for presence or absence) of lung nodules, interstitial disease, and pneumothorax for the five prevalence levels were compared, and a test for trend in averaged ratings as a function of abnormality prevalence was performed within a mixed-model setting that accounts for different sources of variability and correlations induced by the study design. RESULTS: The ratings of the cases in terms of confidence that the specific abnormality in question is present tend, on average, to be larger when actual disease prevalence is lower. The rate of the increase of the average confidence ratings with the decreasing prevalence of a specific abnormality is very similar for actually positive and actually negative cases for every considered abnormality. The observed trend in the changes of the average confidence ratings as a function of prevalence levels was statistically significant (p < 0.01). CONCLUSION: Expectations of disease prevalence in the case mix during a laboratory observer performance study may systematically affect the behavior of observers in terms of their actual confidence ratings.

Reader variance in ROC studies--generalizability to reader population at high and low performance levels.

RATIONALE AND OBJECTIVES: To investigate the variability between discriminative performances of readers as a function of average performance levels during receiver operating characteristic (ROC) studies. MATERIALS AND METHODS: Four subsets of cases from previously ascertained ROC rating data by 12 observers when detecting interstitial disease and pneumothorax on posteroanterior chest films were selected for each abnormality and reanalyzed to assess changes in "reader" variance component. The subsets were selected based on a prestudy subjective assessment of the subtleness of depicted abnormality (positive cases) and the difficulty in determining its absence (negative cases). Reader variance component was estimated using a bootstrap approach for each subset and the results were used to assess a general relationship between variability and average performance level. RESULTS: The reader variance component decreased substantially (from 0.007704 to 0.000426), as expected, when the areas under the ROC curves (AUC) for detecting pneumothoraces increased from 84% to 97%. On the other hand, reader variance component increased substantially (from 0.000890 to 0.005181) when AUC for detecting interstitial disease increased from 59% to 87%. The large magnitude of and changes in the reader variance component resulted in a consistent nonmonotone relationship as a function of AUC when other related variance components were included in addition to the reader component. CONCLUSION: Among several factors affecting generalizability of ROC results to the population of readers, the reader variance component depended nonmonotonically on the average diagnostic performance and is lowest at both very high and very low levels of performance.

Screening mammography: do women prefer a higher recall rate given the possibility of earlier detection of cancer?

PURPOSE: To prospectively survey women undergoing screening mammography to assess their attitudes toward and preference for the level of recall rates given the possibility that an increase in recall rates may result in earlier detection of cancer. MATERIALS AND METHODS: This HIPAA-compliant survey was performed with an institutional review board-approved protocol. Women who arrived for their routine screening mammographic examination from November 2004 to March 2005 were informed before they consented to participate. The distribution of responses for each survey question was summarized, and proportions for the entire group and different subgroups were computed. The z score statistic was used to assess significant differences between subgroups. RESULTS: Fifteen hundred seventy anonymized questionnaires were collected; 1171 (75%) were from women between 40 and 59 years of age. Of 1528 respondents, 1486 (97%) believed that a false-positive result would not deter them from continuing with regular screening, and most would have been willing to be recalled more often for either a noninvasive (86% [1308 of 1519 respondents]) or an invasive (82% [1248 of 1515 respondents]) procedure if it might increase the chance of detecting a cancer (if present) earlier. Compared with respondents undergoing their initial screening mammographic examination, women who had undergone at least one prior screening examination reported that they were more likely to continue with screening if they had received a previous false-positive result (P = .02). Women younger than 60 years and those previously recalled were more willing to be called back more often for a noninvasive or, when indicated, an invasive procedure (P < .05). CONCLUSION: A substantial fraction of women in this study would have preferred the inconvenience of and anxiety associated with a higher recall rate if it resulted in the possibility of detecting breast cancer earlier.

The effect of image display size on observer performance an assessment of variance components.

RATIONALE AND OBJECTIVE: Our goal was to investigate the effect of the displayed image size on variance components during the performance of an observer performance study to detect masses on abdominal computed tomography (CT) examinations. MATERIALS AND METHODS: A previously performed receiver operating characteristic (ROC) study with eight observers to detect abdominal masses on 166 CT examinations was reanalyzed to assess variance components when comparing two similar modes with displayed image sizes varying by a factor of 2. Case, mode, and reader-related variance components were estimated for the group of eight observers and subsets of readers after excluding each of the participants. RESULTS: There was no significant difference in the average area under the ROC curves between the two modes using the two image sizes (P > .05). Reader and reader-by-case variability were substantially larger for the mode displaying enlarged images for the group and all subsets formed by excluding a single reader. Reader variability was affected by one observer who actually performed better with the enlarged images. CONCLUSION: Sequential viewing of enlarged CT images for the detection of abdominal masses did not improve performance and increased reader variability.

Variability in observer performance studies experimental observations.

RATIONALE AND OBJECTIVES: The aim of the study is to assess variance components in observer performance studies and the possible impact on study results and conclusions. MATERIALS AND METHODS: Two previously performed retrospective receiver operating characteristic-type observer performance studies to evaluate the performance of seven radiologists in detecting interstitial disease on conventional posteroanterior chest films and nine radiologists in detecting interstitial disease on a high-resolution workstation were reanalyzed by using the Beiden, Wagner, and Campbell nine-component model to estimate the different variance components. We estimated case-, reader-, and mode-related components of the variance for the group as a whole and after excluding (round robin) each reader. Overall variance was evaluated, and the effect of individual readers on overall study conclusions was assessed. RESULTS: Overall results and conclusions of the reanalysis agreed with the original one in that, as a group, radiologists performed significantly better when using conventional films (P < .05) in both studies. Reader variability was large compared with all other components, and in one study, it was substantially larger for the workstation reading mode. Reader variability was affected substantially by one observer in each study, and in one study, reader-by-mode variability was affected by another reader who performed better on the workstation. CONCLUSION: Estimates of variance components can shed light on the appropriateness of study design, as well as the sensitivity of results to the inclusion (or exclusion) of individual observers.

Is maximum positive predictive value a good indicator of an optimal screening mammography practice?

OBJECTIVE: Positive predictive value (PPV1) has been used as one important indicator of the quality of screening mammography programs. We show how the relationship between sensitivity and recall rate may affect the operating point at which optimal (maximum) PPV1 occurs. CONCLUSION: Optimal (maximum) PPV1 can occur at any sensitivity level and should not be used as the sole indicator for practice optimization because it does not take into account the number of cancers that would be missed at that sensitivity.

Trends in recall, biopsy, and positive biopsy rates for screening mammography in an academic practice.

PURPOSE: To retrospectively evaluate whether recall, biopsy, and positive biopsy rates for a group of radiologists who met requirements of Mammography Quality Standards Act of 1992 (MQSA) demonstrated any change over time during a 27-month period (nine consecutive calendar quarters). MATERIALS AND METHODS: Institutional review board approved study protocol, and informed consent was waived. All screening mammograms that had been interpreted by MQSA-qualified radiologists between January 1, 2001, and March 31, 2003, were reviewed. Group recall rates, biopsy rates, and detected cancer rates for nine calendar quarters were computed and attributed to performance date of original screening mammogram. Type of biopsy performed was classified as follows: stereotactic vacuum-assisted biopsy, ultrasonography (US)-guided core biopsy, US-guided fine-needle aspiration biopsy, surgical excision, and multiple biopsies. chi(2) Test for trend (two sided) and linear regression were used to assess trends over time for recall and biopsy rates, biopsy rates according to type of biopsy performed, and percentage of biopsy results positive for cancer. RESULTS: Group recall rate did not show a statistically significant trend during period studied (P = .59). Biopsy rates increased significantly from 13.02 to 20.12 per 1000 screening examinations (P < .001). A corresponding substantial decrease was seen in percentage of biopsies in which malignancy was found, although this trend was not statistically significant (P = .24). A significant increase (from 4.72 to 9.88 per 1000 screening examinations) was found in rate of stereotactic vacuum-assisted 11-gauge core biopsies performed (P < .001). CONCLUSION: Observed increase in biopsy rates reinforces the need to carefully select patients for biopsy to achieve efficient, efficacious, and cost-effective programs for early detection of breast cancers.

Prescreening mammography by technologists: a preliminary assessment.

OBJECTIVE: We assessed the ability of technologists to accurately classify screening mammograms as either showing negative findings or requiring follow-up. SUBJECTS AND METHODS: In a prospective study, 33 technologists at a central facility and five satellite breast imaging facilities recorded whether mammograms obtained during 3019 examinations showed negative findings or findings that indicated that additional procedures were required. The technologists were not specifically trained for the experiment. The technologists' interpretations were compared with radiologists' interpretations. RESULTS: Technologists and radiologists agreed in 82% of the cases (77% negative findings and 5% requiring follow-up). Of the 175 cases recommended for follow-up by only the radiologists, 17 were ultimately biopsied and two were found to be malignant. CONCLUSION: Even without undergoing additional training, technologists can perform at reasonable levels of accuracy in classifying screening mammograms. The possibility of using technologists to group cases after the technologists have undergone training is an interesting concept that should be explored further.