Generalization of Deep Learning in Digital Pathology: Experience in Breast Cancer Metastasis Detection.

Poor generalizability is a major barrier to clinical implementation of artificial intelligence in digital pathology. The aim of this study was to test the generalizability of a pretrained deep learning model to a new diagnostic setting and to a small change in surgical indication. A deep learning model for breast cancer metastases detection in sentinel lymph nodes, trained on CAMELYON multicenter data, was used as a base model, and achieved an AUC of 0.969 (95% CI 0.926-0.998) and FROC of 0.838 (95% CI 0.757-0.913) on CAMELYON16 test data. On local sentinel node data, the base model performance dropped to AUC 0.929 (95% CI 0.800-0.998) and FROC 0.744 (95% CI 0.566-0.912). On data with a change in surgical indication (axillary dissections) the base model performance indicated an even larger drop with a FROC of 0.503 (95%CI 0.201-0.911). The model was retrained with addition of local data, resulting in about a 4% increase for both AUC and FROC for sentinel nodes, and an increase of 11% in AUC and 49% in FROC for axillary nodes. Pathologist qualitative evaluation of the retrained model´s output showed no missed positive slides. False positives, false negatives and one previously undetected micro-metastasis were observed. The study highlights the generalization challenge even when using a multicenter trained model, and that a small change in indication can considerably impact the model´s performance.

The human-in-the-loop: an evaluation of pathologists' interaction with artificial intelligence in clinical practice.

AIMS: One of the major drivers of the adoption of digital pathology in clinical practice is the possibility of introducing digital image analysis (DIA) to assist with diagnostic tasks. This offers potential increases in accuracy, reproducibility, and efficiency. Whereas stand-alone DIA has great potential benefit for research, little is known about the effect of DIA assistance in clinical use. The aim of this study was to investigate the clinical use characteristics of a DIA application for Ki67 proliferation assessment. Specifically, the human-in-the-loop interplay between DIA and pathologists was studied. METHODS AND RESULTS: We retrospectively investigated breast cancer Ki67 areas assessed with human-in-the-loop DIA and compared them with visual and automatic approaches. The results, expressed as standard deviation of the error in the Ki67 index, showed that visual estimation ('eyeballing') (14.9 percentage points) performed significantly worse (P < 0.05) than DIA alone (7.2 percentage points) and DIA with human-in-the-loop corrections (6.9 percentage points). At the overall level, no improvement resulting from the addition of human-in-the-loop corrections to the automatic DIA results could be seen. For individual cases, however, human-in-the-loop corrections could address major DIA errors in terms of poor thresholding of faint staining and incorrect tumour-stroma separation. CONCLUSION: The findings indicate that the primary value of human-in-the-loop corrections is to address major weaknesses of a DIA application, rather than fine-tuning the DIA quantifications.

Proactive Construction of an Annotated Imaging Database for Artificial Intelligence Training.

Artificial intelligence (AI) holds much promise for enabling highly desired imaging diagnostics improvements. One of the most limiting bottlenecks for the development of useful clinical-grade AI models is the lack of training data. One aspect is the large amount of cases needed and another is the necessity of high-quality ground truth annotation. The aim of the project was to establish and describe the construction of a database with substantial amounts of detail-annotated oncology imaging data from pathology and radiology. A specific objective was to be proactive, that is, to support undefined subsequent AI training across a wide range of tasks, such as detection, quantification, segmentation, and classification, which puts particular focus on the quality and generality of the annotations. The main outcome of this project was the database as such, with a collection of labeled image data from breast, ovary, skin, colon, skeleton, and liver. In addition, this effort also served as an exploration of best practices for further scalability of high-quality image collections, and a main contribution of the study was generic lessons learned regarding how to successfully organize efforts to construct medical imaging databases for AI training, summarized as eight guiding principles covering team, process, and execution aspects.

Educational level and management and outcomes in non-small cell lung cancer. A nationwide population-based study.

OBJECTIVES: We examined associations between educational level and clinical presentation, patterns of management and mortality in patients with non-small cell lung cancer (NSCLC) in Sweden, a country with a National Health Care System. MATERIALS AND METHODS: We identified 39,671 patients with a NSCLC diagnosis 2002-2016 in Lung Cancer Data Base Sweden (LCBaSe), a population-based research database. In analyses adjusted for comorbidity and other prognostic factors, odds Ratios (OR) and hazard Ratios (HR) were estimated to examine associations between patients' educational level and aspects of management and mortality. RESULTS: Stage at diagnosis and waiting times did not differ between educational groups. In multivariable analysis, the likelihood to undergo PET/CT and assessment in a multidisciplinary team setting were higher in patients with high compared to low education (aOR 1.14; CI 1.05-1.23 and aOR 1.22; CI 1.14-1.32, respectively). In patients with early stage IA-IIB disease, the likelihood to undergo stereotactic radiotherapy was elevated in patients with high education (aOR 1.40; CI 1.03-1.91). Both all-cause (aHR 0.86; CI 0.77-0.92) and cause-specific mortality (aHR 0.83; CI 0.74-0.92) was lower in patients with high compared to low education in early stage disease (IA-IIB). In higher stage NSCLC no differences were observed. Patterns were similar in separate assessments stratified by sex and histopathology. CONCLUSIONS: While stage at diagnosis and waiting times did not differ between educational groups, we found socioeconomic differences in diagnostic intensity, multidisciplinary team assessment, stereotactic radiotherapy and mortality in patients with NSCLC. These findings may in part reflect social gradients in implementation and use of novel diagnostic and treatment modalities. Our findings underscore the need for improved adherence to national guidelines.

Development and Evaluation of a Novel Point-of-Use Quality Assurance Tool for Digital Pathology.

CONTEXT.­: Flexible working at diverse or remote sites is a major advantage when reporting using digital pathology, but currently there is no method to validate the clinical diagnostic setting within digital microscopy. OBJECTIVE.­: To develop a preliminary Point-of-Use Quality Assurance (POUQA) tool designed specifically to validate the diagnostic setting for digital microscopy. DESIGN.­: We based the POUQA tool on the red, green, and blue (RGB) values of hematoxylin-eosin. The tool used 144 hematoxylin-eosin-colored, 5×5-cm patches with a superimposed random letter with subtly lighter RGB values from the background color, with differing levels of difficulty. We performed an initial evaluation across 3 phases within 2 pathology departments: 1 in the United Kingdom and 1 in Sweden. RESULTS.­: In total, 53 experiments were conducted across all phases resulting in 7632 test images viewed in all. Results indicated that the display, the user's visual system, and the environment each independently impacted performance. Performance was improved with reduction in natural light and through use of medical-grade displays. CONCLUSIONS.­: The use of a POUQA tool for digital microscopy is essential to afford flexible working while ensuring patient safety. The color-contrast test provides a standardized method of comparing diagnostic settings for digital microscopy. With further planned development, the color-contrast test may be used to create a "Verified Login" for diagnostic setting validation.