Multireader sample size program for diagnostic studies: demonstration and methodology.

The software "Multireader sample size program for diagnostic studies," written by Kevin Schartz and Stephen Hillis, performs sample size computations for diagnostic reader-performance studies. The program computes the sample size needed to detect a specified difference in a reader-performance measure between two imaging modalities when using the analysis methods initially proposed by Dorfman, Berbaum, and Metz, and Obuchowski and Rockette, and later unified and improved by Hillis and colleagues. A commonly used reader-performance measure is the area under the receiver-operating-characteristic curve. The program has an easy-to-use step-by-step intuitive interface that walks the user through the entry of the needed information. It can be used with several different study designs, inference procedures, hypotheses, and input and output formats. The program is functional in Windows, OS X, and Linux. The methodology underlying the software is discussed for the most common diagnostic study design, where each reader evaluates each case using each modality.

Effect of fatigue on reading computed tomography examination of the multiply injured patient.

Our goal was to ascertain how fatigue affects performance in reading computed tomography (CT) examinations of patients with multiple injuries. CT images with multiple fractures from a previous study of satisfaction of search (SOS) were read by radiologists after a day of clinical work. Performance in this study with fatigued readers was compared to a previous study in which readers were not fatigued. Detection accuracy for obvious injuries was not affected by fatigue, but accuracy for subtle fractures was reduced ([Formula: see text]). An SOS effect on decision thresholds was evident mirroring recent studies. Without fatigue, readers spent more time interpreting and reporting findings as the number of the injuries increased. When fatigued, readers did not increase reading time as fracture number increased. Without fractures, reading time for not-fatigued and fatigued readers was the same ([Formula: see text]) but was significant ([Formula: see text]) with an added subtle fracture. The difference increased with a major injury ([Formula: see text]) and increased further with both a major injury and subtle fracture ([Formula: see text]). Fatigue and multiple abnormalities have independent effects on detection performance but do interact in determining search time.

The Impact of Fatigue on Satisfaction of Search in Chest Radiography.

RATIONALE AND OBJECTIVES: To assess the nature of the satisfaction of search (SOS) effect in chest radiography when observers are fatigued; determine if we could replicate recent findings that have documented the nature of the SOS effect to be due to a threshold shift rather than a change in diagnostic accuracy as in earlier film-based studies. MATERIALS AND METHODS: Nearing or at the end of a clinical workday, 20 radiologists read 64 chest images twice, once with and once without the addition of a simulated pulmonary nodule. Half of the images had different types of "test" abnormalities. Decision thresholds were analyzed using the center of the range of false-positive (FP) and true-positive (TP) fractions associated with each receiver operating characteristic (ROC) point for reporting test abnormalities. Detection accuracy was assessed with ROC technique and inspection time was recorded. RESULTS: The SOS effect was confirmed to be a reduction in willingness to respond (threshold shift). The center of the FP range was significantly reduced (FP = 0.10 without added nodules, FP = 0.05 with added nodules, F(1,18) = 19.85, P = 0.0003). The center of the TP range was significantly reduced (TP = 0.39 without added nodules, TP = 0.33 with added nodules, F(1,18) = 10.81, P = 0.004). CONCLUSIONS: This study suggests that fatigue does not change the nature of the SOS effect, but rather may be additive with the SOS effect. SOS reduces both TP and FP responses, whereas fatigue reduces TPs more than FPs.

Trauma in CT: The Role of Severe Injury on Satisfaction of Search Revised.

PURPOSE: The satisfaction-of-search (SOS) effect occurs when an abnormality on an image is missed because another is found. The aim of this experiment was to test whether severe distracting fractures control the magnitude of SOS on other fractures when both appear in a single CT image. METHODS: The institutional review board approved this study. The experimental (SOS) condition included 35 cervical spine CT cases, all of which contained severe cervical spine injuries. For each of these cases, a similar case was found that had no injuries. Image modification software was developed to add simulated fractures to each pair of cases, with and without a major injury. Sixteen different minor fractures were added to 16 of the 35 pairs of images. The 35 cases without native injuries constituted a control (non-SOS) condition mixed in a random order. Twenty radiologists read 35 mixed cases in each of two sessions. False-positive evaluations were collected only for cases without simulated fractures. RESULTS: An SOS effect on the detection of simulated fractures was not observed. There was a nonsignificant (P = .07) finding of poorer detection in the presence of cases with severe injuries. However, the magnitude of the effect was no greater than has been observed for less severe distracting injuries. CONCLUSIONS: The outcome agrees with the results of two previous experiments that failed to yield an SOS effect associated with detecting severe injuries, suggesting that the severity of a distracting injury does not determine whether a second injury is discovered.

The Influence of a Vocalized Checklist on Detection of Multiple Abnormalities in Chest Radiography.

RATIONALE AND OBJECTIVES: Although a checklist has been recommended for preventing satisfaction of search (SOS) errors, a previous research study did not demonstrate that benefit. However, observers in that study had to turn away from the image display to use the checklist. The current study tested a vocalized checklist to avoid this constraint. MATERIALS AND METHODS: A total of 64 chest computed radiographs, half containing various "test" abnormalities, were read twice by 20 radiologists, once with and once without the addition of a simulated pulmonary nodule. Readers used a vocalized checklist-directing search. Receiver operating characteristic (ROC) detection accuracy and decision thresholds were analyzed to study the effects of adding the nodule on detecting the test abnormalities. RESULTS: Adding nodules induced a substantial reluctance to report the other abnormalities (P < 0.001), as had been the case in the most recent study of the SOS effect in radiography. CONCLUSIONS: The vocalized checklist did not reduce nor eliminate the SOS effect on readiness to report further abnormalities. Although useful for organizing search and reporting, particularly among students, a vocalized checklist does not prevent SOS effects.

Satisfaction of Search in Chest Radiography 2015.

RATIONALE AND OBJECTIVES: Two decades have passed since the publication of laboratory studies of satisfaction of search (SOS) in chest radiography. Those studies were performed using film. The current investigation tests for SOS effects in computed radiography of the chest. METHODS: Sixty-four chest computed radiographs half demonstrating various "test" abnormalities were read twice by 20 radiologists, once with and once without the addition of a simulated pulmonary nodule. Receiver-operating characteristic detection accuracy and decision thresholds were analyzed to study the effects of adding the nodule on detecting the test abnormalities. Results of previous studies were reanalyzed using similar modern techniques. RESULTS: In the present study, adding nodules did not influence detection accuracy for the other abnormalities (P = .93), but did induce a reluctance to report them (P < .001). Adding nodules did not affect inspection time (P = .58) so the reluctance to report was not associated with reduced search. Reanalysis revealed a similar decision threshold shift that had not been recognized in the early studies of SOS in chest radiography (P < .01) in addition to reduced detection accuracy (P < .01). CONCLUSIONS: The nature of SOS in chest radiography has changed, but it is not clear why. ADVANCES IN KNOWLEDGE: SOS may be changing as a function of changes in radiology education and practice.

Demonstration of Multi- and Single-Reader Sample Size Program for Diagnostic Studies software.

The recently released software Multi- and Single-Reader Sample Size Sample Size Program for Diagnostic Studies, written by Kevin Schartz and Stephen Hillis, performs sample size computations for diagnostic reader-performance studies. The program computes the sample size needed to detect a specified difference in a reader performance measure between two modalities, when using the analysis methods initially proposed by Dorfman, Berbaum, and Metz (DBM) and Obuchowski and Rockette (OR), and later unified and improved by Hillis and colleagues. A commonly used reader performance measure is the area under the receiver-operating-characteristic curve. The program can be used with typical common reader-performance measures which can be estimated parametrically or nonparametrically. The program has an easy-to-use step-by-step intuitive interface that walks the user through the entry of the needed information. Features of the software include the following: (1) choice of several study designs; (2) choice of inputs obtained from either OR or DBM analyses; (3) choice of three different inference situations: both readers and cases random, readers fixed and cases random, and readers random and cases fixed; (4) choice of two types of hypotheses: equivalence or noninferiority; (6) choice of two output formats: power for specified case and reader sample sizes, or a listing of case-reader combinations that provide a specified power; (7) choice of single or multi-reader analyses; and (8) functionality in Windows, Mac OS, and Linux.

Satisfaction of search from detection of pulmonary nodules in computed tomography of the chest.

RATIONALE AND OBJECTIVES: We tested whether satisfaction of search (SOS) effects that occur in computed tomography (CT) examination of the chest on detection of native abnormalities are produced by the addition of simulated pulmonary nodules. MATERIALS AND METHODS: Two experiments were conducted. In the first experiment, 70 CT examinations, half that demonstrated diverse, subtle abnormalities and half that demonstrated no native lesions, were read by 18 radiology residents and fellows under two experimental conditions: presented with and without pulmonary nodules. In a second experiment, many of the examinations were replaced to include more salient native abnormalities. This set was read by 14 additional radiology residents and fellows. In both experiments, detection of the natural abnormalities was studied. Receiver operating characteristic (ROC) curve areas for each reader-treatment combination were estimated using empirical and proper ROC models. Additional analyses focused on decision thresholds and visual search time on abnormality-free CT slice ranges. Institutional review board approval and informed consent from 32 participants were obtained. RESULTS: Observers more often missed diverse native abnormalities when pulmonary nodules were added, but also made fewer false-positive responses. There was no change in ROC area, but decision criteria grew more conservative. The SOS effect on decision thresholds was accompanied by a reduction in search time on abnormality-free CT slice ranges. CONCLUSION: The SOS effect in CT examination of the chest is similar to that found in contrast examination of the abdomen, involving induced visual neglect.

Satisfaction of search for subtle skeletal fractures may not be induced by more serious skeletal injury.

PURPOSE: The aim of this experiment was to test whether radiographs of major injuries, those having serious consequences for life and limb, produce a satisfaction-of-search (SOS) effect on the detection of subtle, nondisplaced test fractures. METHODS: Institutional review board approval and informed consent from 24 participants were obtained. Seventy simulated patients with multiple trauma injuries were constructed from radiographs of 3 different anatomic areas demonstrated only skeletal injuries. Readers evaluated each patient under 2 conditions: first, in the non-SOS condition, no injuries were present in the first anatomic images, and second, in the SOS condition, the first anatomic images included major injuries requiring immediate medical intervention. The SOS effect was measured on detection accuracy using receiver operating characteristic analysis for subtle test fractures presented on examinations of the second or third anatomic areas. RESULTS: Satisfaction-of-search reduction in receiver operating characteristic experiments for detecting subtle test fractures with the addition of a major injury was not observed. CONCLUSIONS: Satisfaction of search was absent when major injuries were presented on radiographs. This finding rejects the hypothesis that SOS arises primarily from injuries requiring major intervention. Similar results have been found previously when major injuries were presented on CT but test fractures were presented on radiographs. This new finding rejects the possibility that SOS is absent because added and test fractures appear on different imaging modalities.

Do long radiology workdays affect nodule detection in dynamic CT interpretation?

PURPOSE: A previous study demonstrated decreased diagnostic accuracy for finding fractures and decreased ability to focus on skeletal radiographs after a long working day. Skeletal radiographic examinations commonly have images that are displayed statically. The aim of this study was to investigate whether diagnostic accuracy for detecting pulmonary nodules on CT of the chest displayed dynamically would be similarly affected by fatigue. METHODS: Twenty-two radiologists and 22 residents were given 2 tests searching CT chest sequences for a solitary pulmonary nodule before and after a day of clinical reading. To measure search time, 10 lung CT sequences, each containing 20 consecutive sections and a single nodule, were inspected using free search and navigation. To measure diagnostic accuracy, 100 CT sequences, each with 20 sections and half with nodules, were displayed at preset scrolling speed and duration. Accuracy was measured using receiver operating characteristic curve analysis. Visual strain was measured via dark vergence, an indicator of the ability to keep the eyes focused on the display. RESULTS: Diagnostic accuracy was reduced after a day of clinical reading (P = .0246), but search time was not affected (P > .05). After a day of reading, dark vergence was significantly larger and more variable (P = .0098), reflecting higher levels of visual strain, and subjective ratings of fatigue were also higher. CONCLUSIONS: After their usual workday, radiologists experience increased fatigue and decreased diagnostic accuracy for detecting pulmonary nodules on CT. Effects of fatigue may be mitigated by active interaction with the display.

Long radiology workdays reduce detection and accommodation accuracy.

PURPOSE: The aim of this study was to measure the diagnostic accuracy of fracture detection, visual accommodation, reading time, and subjective ratings of fatigue and visual strain before and after a day of clinical reading. METHODS: Forty attending radiologists and radiology residents viewed 60 deidentified, HIPAA-compliant bone examinations, half with fractures, once before any clinical reading (early) and once after a day of clinical reading (late). Reading time was recorded. Visual accommodation (the ability to maintain focus) was measured before and after each reading session. Subjective ratings of symptoms of fatigue and oculomotor strain were collected. The study was approved by local institutional review boards. RESULTS: Diagnostic accuracy was reduced significantly after a day of clinical reading, with average areas under the receiver operating characteristic curves of 0.885 for early reading and 0.852 for late reading (P < .05). After a day of image interpretation, visual accommodation was no more variable, though error in visual accommodation was greater (P < .01), and subjective ratings of fatigue were higher. CONCLUSIONS: After a day of clinical reading, radiologists have reduced ability to focus, increased symptoms of fatigue and oculomotor strain, and reduced ability to detect fractures. Radiologists need to be aware of the effects of fatigue on diagnostic accuracy and take steps to mitigate these effects.

Does computer-aided diagnosis for lung tumors change satisfaction of search in chest radiography?

RATIONALE AND OBJECTIVES: Computer-aided diagnosis (CAD) has been developed to ensure that the radiologist considers suspect focal opacities that may represent cancer in chest radiography. Although CAD was not developed to counteract the satisfaction of search (SOS) effect, it may be an effective intervention to do so. The objective of this study is to determine whether an idealized CAD can reduce SOS effects in chest radiography. MATERIALS AND METHODS: Fifty-seven chest radiographs, half of which demonstrated diverse, native abnormalities were read twice by 16 observers, once with and once without the addition of a simulated pulmonary nodule. Simulated CAD prompts were provided during the interpretation, which unerringly pointed to the added simulated nodule. Area under the ROC curve for detecting the native abnormalities was estimated for each observer in each treatment condition. In addition to testing for the SOS effect in the presence of CAD prompts, results were compared to those of a previous SOS study. RESULTS: Significantly more nodules were reported in the SOS with CAD experiment than in the original SOS experiment (49 versus 43, P < .01). An SOS effect was found even when CAD prompts were provided; ROC areas for detecting native abnormalities were reduced with added nodules [0.68 versus 0.65, P (one-tailed) < .05]. Comparison of the current experiment with CAD and the previous SOS experiments failed to show a significant difference of the magnitude of the SOS effect (P = .52). The threshold for reporting was more conservative with CAD prompts than in SOS studies (P = .052). CONCLUSION: Our results indicate that the CAD prompts, even those that always point to their target lesion without false-positive error, fail to counteract SOS in chest radiography. The stricter decision thresholds with CAD prompts may indicate less visual search for native abnormalities.

Satisfaction of search in multitrauma patients: severity of detected fractures.

RATIONALE AND OBJECTIVES: Satisfaction of search (SOS) occurs when an abnormality is missed because another abnormality has been detected. This research studied whether the severity of a detected fracture determines whether subsequent fractures are overlooked. MATERIALS AND METHODS: Each of 70 simulated multitrauma patients presented examinations of three anatomic areas. Readers evaluated each patient under two experimental conditions: when the images of the first anatomic area included a fracture (the SOS condition), and when it did not (the control condition). The SOS effect was measured on detection accuracy for subtle test fractures presented on examinations of the second and third anatomic areas. In an experiment with 12 radiology readers, the initial SOS radiographs showed nondisplaced fractures of extremities, fractures associated with low morbidity. In another experiment with 12 different radiology readers, the initial examination, usually a computed tomography scan, showed cervical and pelvic fractures of the type associated with high morbidity. Because of their more direct role in patient care, the experiment using high morbidity SOS fractures was repeated with 17 orthopedic readers. RESULTS: Detection of subtle test fractures was substantially reduced when fractures of low morbidity were added (P < .01). No similar SOS effect was observed in either experiment in which added fractures were associated with high morbidity. CONCLUSIONS: The satisfaction of search effect in skeletal radiology was replicated, essentially doubling the evidence for SOS in musculoskeletal radiology, and providing an essential contrast to the absence of SOS from high-morbidity fractures.

Can a checklist reduce SOS errors in chest radiography?

RATIONALE AND OBJECTIVES: A previous study demonstrated unexpected protection from satisfaction of search (SOS) effects when observers verbalized the focus of their attention during visual search and interpretation of chest radiographs. We suggested that protection from SOS might have occurred if each observer developed an informal checklist to help generate the verbal descriptions. The objective of this study is to determine whether a formal checklist reduces SOS effects in chest radiology. MATERIALS AND METHODS: Fifty-seven chest radiographs, half of which demonstrated diverse, native abnormalities were read twice by 20 observers, once with and once without the addition of a simulated pulmonary nodule. Area under the receiver operating characteristic (ROC) curve for detecting the native abnormalities was estimated for each observer in each treatment condition using the contaminated binormal ROC model. Radiologists in the current experiment used a checklist during the interpretation, rather than describing their visual search. Results were compared with those of the verbalization study, which used the same set of radiographs. RESULTS: Although no SOS effect was found when the checklist was used, ROC performance was, on average, much poorer with the checklist than when ongoing search was reported verbally (0.68 versus 0.75, F(1, 37) = 17.26, P < .001). CONCLUSIONS: Our results indicate that the recommendation to use a self-prompting checklist to counteract SOS is not warranted. The relative superiority of verbalizing search over using an imposed checklist may be based on the consistency of each of these interventions with the observer's internal strategy for searching radiographs.

A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data.

There are several different statistical methods for analysing multireader ROC studies, with the Dorfman-Berbaum-Metz (DBM) method being the most frequently used. Another method is the corrected F method proposed by Obuchowski and Rockette (OR). The DBM and OR procedures at first appear quite different: DBM is a three-way ANOVA analysis of pseudovalues while OR is a two-way ANOVA analysis of accuracy estimates with correlated errors. We show that the original DBM and OR F statistics for testing the null hypothesis of equal treatments have the same form and will typically have similar values; however, differences in the denominator degrees of freedom will result in differences in p-values even when the F statistics are identical. We show how the methods can be generalized to include variations in the accuracy measure, covariance method, and degrees of freedom. Identical results are obtained when the methods agree with respect to all three of these procedure parameters; hence for a particular choice of procedure parameters the choice of method appears to depend mainly on software preference and availability. The methods are compared using data from a factorial study with two modalities, five readers, and 114 patients.