Can order of report prevent satisfaction of search in abdominal contrast studies?

RATIONALE AND OBJECTIVE: A previous receiver operating characteristic (ROC) study showed a systematic shift in decision thresholds for detecting plain film abnormalities on contrast examinations rather than plain radiographs. A previous eye-position study showed that this shift was based on a relative visual neglect of plain film regions on the contrast studies. We now determine whether an intervention that changes visual search can reduce this search-based satisfaction of search effect in contrast studies of the abdomen. MATERIALS AND METHODS: The authors measured detection of 23 plain film abnormalities in 44 patients who had plain film and contrast examinations. In 2 experiments, each plain-film and contrast study was examined independently in different sessions with observers providing a confidence rating of abnormality for each interpretation. There were 13 observers in the first experiment and 10 in the second experiment. The intervention required that for the contrast studies, observers first report abnormalities in the noncontrast region of the radiograph before reporting contrast findings. ROC curve areas for each observer in each treatment condition were estimated by using a proper ROC model. The analysis focused on changes in decision thresholds among the treatment conditions. RESULTS: The SOS effect on decision thresholds in abdominal contrast studies was replicated. Although reduced, the shift in decision thresholds in detecting plain film abnormalities on contrast examinations remained when observers were required to report those abnormalities before contrast findings. CONCLUSION: Reporting plain film abnormalities before reporting abnormalities demonstrated by contrast reduced somewhat the satisfaction of search effect on decision thresholds produced by a visual neglect of noncontrast regions on contrast examinations. This suggests that interventions that direct visual search do not offer protection against satisfaction of search effects that are based on faulty visual search.

An empirical comparison of discrete ratings and subjective probability ratings.

RATIONALE AND OBJECTIVES: The authors compared receiver operating characteristic (ROC) data from a five-category discrete scale with that from a 101-category subjective probability scale to determine how well the latter categories define the ROC curve. MATERIALS AND METHODS: The authors analyzed data from a pilot study performed for another purpose in which 10 radiologists provided both a five-point confidence rating and a subjective probability rating of abnormality for each interpretation. ROC operating points were plotted for a five-category scale and a 101-category scale to determine how well the observed points covered the range of false-positive probabilities. ROC curves were fitted to the subjective probability data according to the standard ROC model. RESULTS: For these data, subjective probability ratings were somewhat more effective in populating the range of false-positive probability with ROC points. For three observers, the ROC curves inappropriately crossed the chance line. For another four, prevention of such crossing seemed to depend on one or two ROC points near the upper right corner of the ROC space, points based on discriminations within the discrete category "no abnormality to report." CONCLUSION: Subjective probability rating should provide substantially better coverage of the ROC space with operating points, preventing inappropriate crossing of the chance line. Unfortunately, the protection offered by subjective probability ratings was unreliable and depended on ROC points derived from discriminations not directly related to apparent abnormality. The use of proper ROC models to fit data may offer a better solution.