Performance of a Breast Cancer Detection AI Algorithm Using the Personal Performance in Mammographic Screening Scheme.

Background The Personal Performance in Mammographic Screening (PERFORMS) scheme is used to assess reader performance. Whether this scheme can assess the performance of artificial intelligence (AI) algorithms is unknown. Purpose To compare the performance of human readers and a commercially available AI algorithm interpreting PERFORMS test sets. Materials and Methods In this retrospective study, two PERFORMS test sets, each consisting of 60 challenging cases, were evaluated by human readers between May 2018 and March 2021 and were evaluated by an AI algorithm in 2022. AI considered each breast separately, assigning a suspicion of malignancy score to features detected. Performance was assessed using the highest score per breast. Performance metrics, including sensitivity, specificity, and area under the receiver operating characteristic curve (AUC), were calculated for AI and humans. The study was powered to detect a medium-sized effect (odds ratio, 3.5 or 0.29) for sensitivity. Results A total of 552 human readers interpreted both PERFORMS test sets, consisting of 161 normal breasts, 70 malignant breasts, and nine benign breasts. No difference was observed at the breast level between the AUC for AI and the AUC for human readers (0.93% and 0.88%, respectively; P = .15). When using the developer's suggested recall score threshold, no difference was observed for AI versus human reader sensitivity (84% and 90%, respectively; P = .34), but the specificity of AI was higher (89%) than that of the human readers (76%, P = .003). However, it was not possible to demonstrate equivalence due to the size of the test sets. When using recall thresholds to match mean human reader performance (90% sensitivity, 76% specificity), AI showed no differences inperformance, with a sensitivity of 91% (P =. 73) and a specificity of 77% (P = .85). Conclusion Diagnostic performance of AI was comparable with that of the average human reader when evaluating cases from two enriched test sets from the PERFORMS scheme. © RSNA, 2023 See also the editorial by Philpotts in this issue.

Measuring reader fatigue in the interpretation of screening digital breast tomosynthesis (DBT).

OBJECTIVES: The interpretation of digital breast tomosynthesis (DBT) screening examinations is a complex task for an already overstretched workforce which has the potential to increase pressure on readers leading to fatigue and patient safety issues. Studies in non-medical and medical settings have suggested that changes in blink characteristics can reflect fatigue. The purpose of this study is to investigate the use of blink characteristics as an objective marker of fatigue in readers interpreting DBT breast screening examinations. METHODS: Twenty-six DBT readers involved in the UK PROSPECTS trial interpreted a test set of 40 DBT cases while being observed by an eye tracking device from November 2019 to February 2021. Raw data from the eye tracker were collected and automated processing software was used to produce eye blinking characteristics data which were analysed using multiple linear regression statistical models. RESULTS: Of the 26 DBT readers recruited, eye tracking data from 23 participants were analysed due to missing data rendering 3 participants' data uninterpretable. The mean reading time per DBT case was 2.81 min. There was a statistically significant increase in blinking duration of 0.38 ms/case as the reading session progressed (p < 0.0001). This was the result of a significant decrease in the number of ultra-short blinks lasting ≤50 ms (p = 0.0005) and a significant increase in longer blinks lasting 51-100 ms (p = 0.008). CONCLUSION: Changes in blinking characteristics could serve as objective measures of reader fatigue and may prove useful in the development of DBT reading protocols. ADVANCES IN KNOWLEDGE: Blink characteristics can be used as an objective measure of fatigue; however there is limited evidence of their use in radiological settings. Our study suggests that changes in blink duration and frequency could be used to monitor fatigue in DBT reading sessions.

Performance of Radiologists and Radiographers in Double Reading Mammograms: The UK National Health Service Breast Screening Program.

Background Double reading can be used in screening mammography, but it is labor intensive. There is limited evidence on whether trained radiographers (ie, technologists) may be used to provide double reading. Purpose To compare the performance of radiologists and radiographers double reading screening mammograms, considering reader experience level. Materials and Methods In this retrospective study, performance and experience data were obtained for radiologists and radiographer readers of all screening mammograms in England from April 2015 to March 2016. Cancer detection rate (CDR), recall rate (RR), and positive predictive value (PPV) of recall based on biopsy-proven findings were calculated for first readers. Performance metrics were analyzed according to reader professional group and years of reading experience using the analysis of variance test. P values less than .05 were considered to indicate statistically significant difference. Results During the study period, 401 readers (224 radiologists and 177 radiographers) double read 1 404 395 screening digital mammograms. There was no difference in CDR between radiologist and radiographer readers (mean, 7.84 vs 7.53 per 1000 examinations, respectively; P = .08) and no difference for readers with more than 10 years of experience compared with 5 years or fewer years of experience, regardless of professional group (mean, 7.75 vs 7.71 per 1000 examinations respectively, P = .87). No difference in the mean RR was observed between radiologists and radiographer readers (5.0% vs 5.2%, respectively, P = .63). A lower RR was seen for readers with more than 10 years of experience compared with 5 years or fewer, regardless of professional group (mean, 4.8% vs 5.8%, respectively; P = .001). No variation in PPV was observed between them (P = .42), with PPV values of 17.1% for radiologists versus 16.1% for radiographers. A higher PPV was seen for readers with more than 10 years of experience compared with 5 years or less, regardless of professional group (mean, 17.5% and 14.9%, respectively; P = .02). Conclusion No difference in performance was observed between radiographers and radiologists reading screening mammograms in a program that used double reading. Published under a CC BY 4.0 license Online supplemental material is available for this article. See also the editorial by Hooley and Durand in this issue.

The relationship between missed breast cancers on mammography in a test-set based assessment scheme and real-life performance in a National Breast Screening Programme.

PURPOSE: This retrospective study determined whether a test-set based assessment scheme (PERFORMS) used in a national breast screening programme could be used to predict real-life performance by investigating if the number of cancers missed by mammography readers in real-life related to the number of cancers missed in the PERFORMS test-set and whether real-life reading volumes affected performance. METHOD: Data was obtained from consenting readers in the screening programme in England (NHSBSP) where double reading is standard. The rate of cancers missed by individual first readers but correctly identified by second readers was compared with the number of cancers missed in the PERFORMS test-set over a 3-year period. NHSBSP readers are required to interpret at least 1500 cases per year as a first reader, so results were compared between readers who exceeded this target and those that did not. Parametric and non-parametric correlations were calculated. RESULTS: Amongst the 536 readers, there was a highly significant positive correlation between the real-life and PERFORMS test-set missed cancer metrics (Pearson Correlation = 0.228, n = 536, p < .0001, Spearman's rho = 0.265, n = 536, p < .0001). There was no significant difference in rates of missed cancers between the 452 readers who exceeded the 1500 first read per year target and those who did not (t(94.2) = -1.87, p = .0643, r = 0.19). CONCLUSIONS: The use of a test-set based assessment scheme accurately reflects real-life mammography reading performance, indicating that it can be a useful tool in identifying poor reader performance.

The influence of 'Tall Man' lettering on errors of visual perception in the recognition of written drug names.

Visual errors in the perception of written drug names can reflect orthographic similarity amongst certain names. Drug names are typically printed in lowercase text. 'Tall Man' lettering, the capitalisation of the portions that differ amongst orthographically similar drug names, is employed in the field of medication labelling and prescribing to reduce medication errors by highlighting the area most likely to prevent confusion. The influence of textual format on visual drug name perception was tested amongst healthcare professionals (n = 133) using the Reicher-Wheeler task. Relative to lowercase text, Tall Man lettering improved accuracy in drug name perception. However, an equivalent improvement in accuracy was obtained using entirely uppercase text. Thus, character size may be a key determinant of perceptual accuracy for Tall Man lettering. Specific considerations for the manner in which Tall Man lettering might be best formatted and implemented in practice to reduce medication errors are discussed. STATEMENT OF RELEVANCE: Tall Man lettering aims to prevent medication errors by reducing visual confusions amongst orthographically similar drug names. It was found that, compared to lowercase text, Tall Man lettering improved accuracy in drug name perception. Character size appeared to be a key determinant of perceptual accuracy for Tall Man lettering.

Perception of words and non-words in the upper and lower visual fields.

The findings of previous investigations into word perception in the upper and the lower visual field (VF) are variable and may have incurred non-perceptual biases caused by the asymmetric distribution of information within a word, an advantage for saccadic eye-movements to targets in the upper VF and the possibility that stimuli were not projected to the correct retinal locations. The present study used the Reicher-Wheeler task and an eye-tracker to show that, using stringent methodology, a right over left VF advantage is observed for word recognition, but that no differences were found between the upper and the lower VF for either word or non-word recognition. The results are discussed in terms of the neuroanatomy and perceptual abilities of the upper and the lower VF and implications for other studies of letter-string perception in the upper and the lower VF are presented.