Measuring performance in the interpretation of chest radiographs: a pilot study.

AIM: To develop a system to assess the image interpretation performance of radiologists in identifying signs of malignancy on chest radiographs. MATERIALS AND METHODS: A test set of 30 chest radiographs was chosen by an experienced radiologist consisting of 11 normal and 19 abnormal cases. The malignant cases all had biopsy-proven pathology; the normal and benign cases all had at least 2 years of imaging follow-up. Fourteen radiologists with a range of experiences were recruited. Participants individually read the test set displayed on a standard reporting workstation, with their findings entered directly into a laptop running specially designed reporting software. For each case, relevant clinical information was given and the reader was asked to mark any perceived abnormality and rate their level of suspicion on a five-point scale (normal, benign, indeterminate, suspicious, or malignant). On completion, participants were given instant feedback with performance parameters including sensitivity and specificity automatically calculated. An opportunity was then given to review the cases together with an expert opinion and pathology. The time each participant took to complete the test was recorded. RESULTS: Six consultant radiologists who took part showed significantly better performance as determined by receiver operating characteristic (ROC) analysis compared to eight specialist registrars (area under the ROC curve [AUC]=0.9297 and 0.7648 respectively, p=0.003). There was a significant correlation with years of experience in the interpretation of chest radiographs and performance on the test set (r=0.573, p=0.032). Consultant radiologists completed the test significantly more quickly that the specialist registrars: mean time 19.65 minutes compared to 26.51 minutes (p=0.033). CONCLUSION: It is possible to use a test set to measure individual differences in the interpretation of chest radiographs. This has the potential to be a useful tool in performance testing.

Use of prior mammograms in the transition to digital mammography: a performance and cost analysis.

Breast screening in Europe is gradually changing from film to digital imaging and reporting of cases. In the transition period prior mammograms (from the preceding screening round) are films thereby potentially causing difficulties in comparison to current digital mammograms. To examine this breast screening performance was measured at a digital mammography workstation with prior mammograms displayed in different formats, and the associated costs calculated. 160 selected difficult cases (41% malignant) were read by eight UK qualified mammography readers in three conditions: with film prior mammograms; with digitised prior mammograms; or without prior mammograms. Lesion location and probability of malignancy were recorded, alongside a decision of whether to recall each case for further tests. JAFROC analysis showed a difference between conditions (p=.006); performance with prior mammograms in either film or digitised formats was superior to that without prior mammograms (p<.05). There was no difference in the performance when the prior mammograms were presented in film or digitised form. The number of benign or normal cases recalled was 26% higher without prior mammograms than with digitised or film prior mammograms (p<.05). This would correspond to an increase in recall rate at the study hospital from 4.3% to 5.5% with no associated increase in cancer detection rate. The cost of this increase was estimated to be £11,581 (€13,666) per 10,000 women screened, which is higher than the cost of digitised (£11,114/€13,115), or film display (£6451/€7612) of the prior mammograms. It is recommended that, where available, prior mammograms are used in the transition to digital breast screening.

Should previous mammograms be digitised in the transition to digital mammography?

Breast screening specificity is improved if previous mammograms are available, which presents a challenge when converting to digital mammography. Two display options were investigated: mounting previous film mammograms on a multiviewer adjacent to the workstation, or digitising them for soft copy display. Eight qualified screen readers were videotaped undertaking routine screen reading for two 45-min sessions in each scenario. Analysis of gross eye and head movements showed that when digitised, previous mammograms were examined a greater number of times per case (p = 0.03), due to a combination of being used in 19% more cases (p = 0.04) and where used, looked at a greater number of times (28% increase, p = 0.04). Digitising previous mammograms reduced both the average time taken per case by 18% (p = 0.04) and the participants' perceptions of workload (p < 0.05). Digitising previous analogue mammograms may be advantageous, in particular in increasing their level of use.

Breast screening: PERFORMS identifies key mammographic training needs.

The UK Breast Screening Programme has recently expanded the age range for invitation in the prevalent round to 70 years. In contrast, fewer radiologists now choose to specialise in the area of breast cancer screening. In response to this depletion in film-reading personnel, an increasing number of radiographers have been trained as advanced practitioners in order to film-read alongside the current radiologists. As part of the quality assurance programme for the National Health Service Breast Screening Programme (NHSBSP), each film-reader can participate in a voluntary self-assessment scheme (Personal Performance in Mammographic Screening, PERFORMS) which consists of a number of recent challenging breast screening cases that are amassed nationally and distributed bi-annually. The scheme produces anonymous data on any areas of difficulties that individual participants have; these data can then be aggregated over groups of participants or over specific types of screening cases. In this paper, the areas of difficulty experienced by groups of advanced practitioners and radiologists on the PERFORMS cases were investigated to determine whether there were occupational group differences in reading skills in terms of case classification and feature type. Identifying if such problematic areas exist would be the first step to provide training sets specially tailored to the needs of particular occupational groups. As a bench mark for which cases could be problematic, the types of cases that a panel of experienced radiologists deemed as difficult was first examined in order to compare the performance of both film-reading groups against this panel standard. Secondly, any differences in performance error and case characteristics (classification, difficulty level and feature type) between radiologists and advanced practitioners were examined. The decisions of 15 experienced "panel" radiologists and approximately 400 film readers (including radiologists and advanced practitioners) were compared on 180 cases, over a number of years. This study employed a matched design which controlled for any differences between radiologists and advanced practitioners in terms of real-life factors, such as volume of cases read per week and years of radiological experience. The results elucidate the type of cases most appropriate for advanced mammographic training. No significant differences were found between the advanced practitioners and radiologists on these self-assessment screening cases, indicating that dedicated occupational group training is not required.

Computer aids to mammographic diagnosis.

The improvement of mammographic specificity was investigated by means of identifying specific radiological features. Data are presented on the first 500 patients studied who had previously undergone mammography followed by biopsy. The presence of specific mammographic features on each radiograph, first determined by retrospective examination, was entered into a computer database. Subsequent discriminant function analysis demonstrated the importance of a small number of features whose presence could be used in an algorithm to predict diagnostic outcome. Using this algorithm, this feature-identification approach correctly identified 87.6% of benign and 79% of malignant cases. Specificity was improved to 88% as compared with the original radiological diagnosis of 49%. It is argued that this approach is very promising and a computer-assisted diagnosis based on these findings is described.

Eye movement strategies involved in face perception.

Recordings were made of the eye fixations of three subjects in two tasks involving black-and-white photographs of faces. In the first task, subjects matched a test face with a previously viewed target face; in the second task, subjects compared two simultaneously presented faces. The eye movements were recorded with a corneal reflection technique. Each subject showed an individual fixation strategy for the tasks; in particular each subject had one or more preferred facial feautres which were viewed foveally in both tasks. The subjects also showed some tendency to use a regular sequential pattern of eye movements. However, the sequences used differed from one task to the other. Although some aspects of the results support the scanpath hypothesis of Noton, it is suggested that an alternative interpretation is possible.