Predicting clinical endpoints and visual changes with quality-weighted tissue-based renal histological features.

Two common obstacles limiting the performance of data-driven algorithms in digital histopathology classification tasks are the lack of expert annotations and the narrow diversity of datasets. Multi-instance learning (MIL) can address the former challenge for the analysis of whole slide images (WSI), but performance is often inferior to full supervision. We show that the inclusion of weak annotations can significantly enhance the effectiveness of MIL while keeping the approach scalable. An analysis framework was developed to process periodic acid-Schiff (PAS) and Sirius Red (SR) slides of renal biopsies. The workflow segments tissues into coarse tissue classes. Handcrafted and deep features were extracted from these tissues and combined using a soft attention model to predict several slide-level labels: delayed graft function (DGF), acute tubular injury (ATI), and Remuzzi grade components. A tissue segmentation quality metric was also developed to reduce the adverse impact of poorly segmented instances. The soft attention model was trained using 5-fold cross-validation on a mixed dataset and tested on the QUOD dataset containing n = 373 PAS and n = 195 SR biopsies. The average ROC-AUC over different prediction tasks was found to be 0.598 ± 0.011 , significantly higher than using only ResNet50 ( 0.545 ± 0.012 ), only handcrafted features ( 0.542 ± 0.011 ), and the baseline ( 0.532 ± 0.012 ) of state-of-the-art performance. In conjunction with soft attention, weighting tissues by segmentation quality has led to further improvement ( A U C = 0.618 ± 0.010 ) . Using an intuitive visualisation scheme, we show that our approach may also be used to support clinical decision making as it allows pinpointing individual tissues relevant to the predictions.

Adjuvant COX inhibition augments STING signaling and cytolytic T cell infiltration in irradiated 4T1 tumors.

Immune therapy is the new frontier of cancer treatment. Therapeutic radiation is a known inducer of immune response and can be limited by immunosuppressive mediators including cyclooxygenase-2 (COX2) that is highly expressed in aggressive triple negative breast cancer (TNBC). A clinical cohort of TNBC tumors revealed poor radiation therapeutic efficacy in tumors expressing high COX2. Herein, we show that radiation combined with adjuvant NSAID (indomethacin) treatment provides a powerful combination to reduce both primary tumor growth and lung metastasis in aggressive 4T1 TNBC tumors, which occurs in part through increased antitumor immune response. Spatial immunological changes including augmented lymphoid infiltration into the tumor epithelium and locally increased cGAS/STING1 and type I IFN gene expression were observed in radiation-indomethacin-treated 4T1 tumors. Thus, radiation and adjuvant NSAID treatment shifts "immune desert phenotypes" toward antitumor M1/TH1 immune mediators in these immunologically challenging tumors. Importantly, radiation-indomethacin combination treatment improved local control of the primary lesion, reduced metastatic burden, and increased median survival when compared with radiation treatment alone. These results show that clinically available NSAIDs can improve radiation therapeutic efficacy through increased antitumor immune response and augmented local generation of cGAS/STING1 and type I IFNs.

SSL-CPCD: Self-supervised learning with composite pretext-class discrimination for improved generalisability in endoscopic image analysis.

Data-driven methods have shown tremendous progress in medical image analysis. In this context, deep learning-based supervised methods are widely popular. However, they require a large amount of training data and face issues in generalisability to unseen datasets that hinder clinical translation. Endoscopic imaging data is characterised by large inter- and intra-patient variability that makes these models more challenging to learn representative features for downstream tasks. Thus, despite the publicly available datasets and datasets that can be generated within hospitals, most supervised models still underperform. While self-supervised learning has addressed this problem to some extent in natural scene data, there is a considerable performance gap in the medical image domain. In this paper, we propose to explore patch-level instance-group discrimination and penalisation of inter-class variation using additive angular margin within the cosine similarity metrics. Our novel approach enables models to learn to cluster similar representations, thereby improving their ability to provide better separation between different classes. Our results demonstrate significant improvement on all metrics over the state-of-the-art (SOTA) methods on the test set from the same and diverse datasets. We evaluated our approach for classification, detection, and segmentation. SSL-CPCD attains notable Top 1 accuracy of 79.77% in ulcerative colitis classification, an 88.62% mean average precision (mAP) for detection, and an 82.32% dice similarity coefficient for segmentation tasks. These represent improvements of over 4%, 2%, and 3%, respectively, compared to the baseline architectures. We demonstrate that our method generalises better than all SOTA methods to unseen datasets, reporting over 7% improvement.

Artificial Intelligence-Based Quality Assessment of Histopathology Whole-Slide Images within a Clinical Workflow: Assessment of 'PathProfiler' in a Diagnostic Pathology Setting.

Digital pathology continues to gain momentum, with the promise of artificial intelligence to aid diagnosis and for assessment of features which may impact prognosis and clinical management. Successful adoption of these technologies depends upon the quality of digitised whole-slide images (WSI); however, current quality control largely depends upon manual assessment, which is inefficient and subjective. We previously developed PathProfiler, an automated image quality assessment tool, and in this feasibility study we investigate its potential for incorporation into a diagnostic clinical pathology setting in real-time. A total of 1254 genitourinary WSI were analysed by PathProfiler. PathProfiler was developed and trained on prostate tissue and, of the prostate biopsy WSI, representing 46% of the WSI analysed, 4.5% were flagged as potentially being of suboptimal quality for diagnosis. All had concordant subjective issues, mainly focus-related, 54% severe enough to warrant remedial action which resulted in improved image quality. PathProfiler was less reliable in assessment of non-prostate surgical resection-type cases, on which it had not been trained. PathProfiler shows potential for incorporation into a digitised clinical pathology workflow, with opportunity for image quality improvement. Whilst its reliability in the current form appears greatest for assessment of prostate specimens, other specimen types, particularly biopsies, also showed benefit.

Image-based consensus molecular subtyping in rectal cancer biopsies and response to neoadjuvant chemoradiotherapy.

The development of deep learning (DL) models to predict the consensus molecular subtypes (CMS) from histopathology images (imCMS) is a promising and cost-effective strategy to support patient stratification. Here, we investigate whether imCMS calls generated from whole slide histopathology images (WSIs) of rectal cancer (RC) pre-treatment biopsies are associated with pathological complete response (pCR) to neoadjuvant long course chemoradiotherapy (LCRT) with single agent fluoropyrimidine. DL models were trained to classify WSIs of colorectal cancers stained with hematoxylin and eosin into one of the four CMS classes using a multi-centric dataset of resection and biopsy specimens (n = 1057 WSIs) with paired transcriptional data. Classifiers were tested on a held out RC biopsy cohort (ARISTOTLE) and correlated with pCR to LCRT in an independent dataset merging two RC cohorts (ARISTOTLE, n = 114 and SALZBURG, n = 55 patients). DL models predicted CMS with high classification performance in multiple comparative analyses. In the independent cohorts (ARISTOTLE, SALZBURG), cases with WSIs classified as imCMS1 had a significantly higher likelihood of achieving pCR (OR = 2.69, 95% CI 1.01-7.17, p = 0.048). Conversely, imCMS4 was associated with lack of pCR (OR = 0.25, 95% CI 0.07-0.88, p = 0.031). Classification maps demonstrated pathologist-interpretable associations with high stromal content in imCMS4 cases, associated with poor outcome. No significant association was found in imCMS2 or imCMS3. imCMS classification of pre-treatment biopsies is a fast and inexpensive solution to identify patient groups that could benefit from neoadjuvant LCRT. The significant associations between imCMS1/imCMS4 with pCR suggest the existence of predictive morphological features that could enhance standard pathological assessment.

Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, appearance, and location makes the detection of polyps challenging. Moreover, colonoscopy surveillance and removal of polyps are highly operator-dependent procedures and occur in a highly complex organ topology. There exists a high missed detection rate and incomplete removal of colonic polyps. To assist in clinical procedures and reduce missed rates, automated methods for detecting and segmenting polyps using machine learning have been achieved in past years. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets from different centres, populations, modalities, and acquisition systems. To test this hypothesis rigorously, we, together with expert gastroenterologists, curated a multi-centre and multi-population dataset acquired from six different colonoscopy systems and challenged the computational expert teams to develop robust automated detection and segmentation methods in a crowd-sourcing Endoscopic computer vision challenge. This work put forward rigorous generalisability tests and assesses the usability of devised deep learning methods in dynamic and actual clinical colonoscopy procedures. We analyse the results of four top performing teams for the detection task and five top performing teams for the segmentation task. Our analyses demonstrate that the top-ranking teams concentrated mainly on accuracy over the real-time performance required for clinical applicability. We further dissect the devised methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets and routine clinical procedures.

Quantitative interpretation of bone marrow biopsies in MPN-What's the point in a molecular age?

The diagnosis of myeloproliferative neoplasms (MPN) requires the integration of clinical, morphological, genetic and immunophenotypic findings. Recently, there has been a transformation in our understanding of the cellular and molecular mechanisms underlying disease initiation and progression in MPN. This has been accompanied by the widespread application of high-resolution quantitative molecular techniques. By contrast, microscopic interpretation of bone marrow biopsies by haematologists/haematopathologists remains subjective and qualitative. However, advances in tissue image analysis and artificial intelligence (AI) promise to transform haematopathology. Pioneering studies in bone marrow image analysis offer to refine our understanding of the boundaries between reactive samples and MPN subtypes and better capture the morphological correlates of high-risk disease. They also demonstrate potential to improve the evaluation of current and novel therapeutics for MPN and other blood cancers. With increased therapeutic targeting of diverse molecular, cellular and extra-cellular components of the marrow, these approaches can address the unmet need for improved objective and quantitative measures of disease modification in the context of clinical trials. This review focuses on the state-of-the-art in image analysis/AI of bone marrow tissue, with an emphasis on its potential to complement and inform future clinical studies and research in MPN.

A multi-centre polyp detection and segmentation dataset for generalisability assessment.

Polyps in the colon are widely known cancer precursors identified by colonoscopy. Whilst most polyps are benign, the polyp's number, size and surface structure are linked to the risk of colon cancer. Several methods have been developed to automate polyp detection and segmentation. However, the main issue is that they are not tested rigorously on a large multicentre purpose-built dataset, one reason being the lack of a comprehensive public dataset. As a result, the developed methods may not generalise to different population datasets. To this extent, we have curated a dataset from six unique centres incorporating more than 300 patients. The dataset includes both single frame and sequence data with 3762 annotated polyp labels with precise delineation of polyp boundaries verified by six senior gastroenterologists. To our knowledge, this is the most comprehensive detection and pixel-level segmentation dataset (referred to as PolypGen) curated by a team of computational scientists and expert gastroenterologists. The paper provides insight into data construction and annotation strategies, quality assurance, and technical validation.

FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation.

The increase of available large clinical and experimental datasets has contributed to a substantial amount of important contributions in the area of biomedical image analysis. Image segmentation, which is crucial for any quantitative analysis, has especially attracted attention. Recent hardware advancement has led to the success of deep learning approaches. However, although deep learning models are being trained on large datasets, existing methods do not use the information from different learning epochs effectively. In this work, we leverage the information of each training epoch to prune the prediction maps of the subsequent epochs. We propose a novel architecture called feedback attention network (FANet) that unifies the previous epoch mask with the feature map of the current training epoch. The previous epoch mask is then used to provide hard attention to the learned feature maps at different convolutional layers. The network also allows rectifying the predictions in an iterative fashion during the test time. We show that our proposed feedback attention model provides a substantial improvement on most segmentation metrics tested on seven publicly available biomedical imaging datasets demonstrating the effectiveness of FANet. The source code is available at https://github.com/nikhilroxtomar/FANet.

Continuous Indexing of Fibrosis (CIF): improving the assessment and classification of MPN patients.

The grading of fibrosis in myeloproliferative neoplasms (MPN) is an important component of disease classification, prognostication and monitoring. However, current fibrosis grading systems are only semi-quantitative and fail to fully capture sample heterogeneity. To improve the quantitation of reticulin fibrosis, we developed a machine learning approach using bone marrow trephine (BMT) samples (n = 107) from patients diagnosed with MPN or a reactive marrow. The resulting Continuous Indexing of Fibrosis (CIF) enhances the detection and monitoring of fibrosis within BMTs, and aids MPN subtyping. When combined with megakaryocyte feature analysis, CIF discriminates between the frequently challenging differential diagnosis of essential thrombocythemia (ET) and pre-fibrotic myelofibrosis with high predictive accuracy [area under the curve = 0.94]. CIF also shows promise in the identification of MPN patients at risk of disease progression; analysis of samples from 35 patients diagnosed with ET and enrolled in the Primary Thrombocythemia-1 trial identified features predictive of post-ET myelofibrosis (area under the curve = 0.77). In addition to these clinical applications, automated analysis of fibrosis has clear potential to further refine disease classification boundaries and inform future studies of the micro-environmental factors driving disease initiation and progression in MPN and other stem cell disorders.

Impact of the transition to digital pathology in a clinical setting on histopathologists in training: experiences and perceived challenges within a UK training region.

AIMS: With increasing utility of digital pathology (DP), it is important to consider the experiences of histopathologists in training, particularly in view of the varied access to DP across a training region and the consequent need to remain competent in reporting on glass slides (GS), which is also relevant for the Fellowship of the Royal College of Pathologists part 2 examination. Understanding the impact of DP on training is limited but could aid development of guidance to support the transition. We sought to investigate the perceptions of histopathologists in training around the introduction of DP for clinical diagnosis within a training region, and the potential training benefits and challenges. METHODS: An anonymous online survey was circulated to 24 histopathologists in training within a UK training region, including a hospital which has been fully digitised since summer 2020. RESULTS: 19 of 24 histopathologists in training responded (79%). The results indicate that DP offers many benefits to training, including ease of access to cases to enhance individual learning and teaching in general. Utilisation of DP for diagnosis appears variable; almost half of the (10 of 19) respondents with DP experience using it only for ancillary purposes such as measurements, reporting varying levels of confidence in using DP clinically. For those yet to undergo the transition, there was a perceived anxiety regarding digital reporting despite experience with DP in other contexts. CONCLUSIONS: The survey evidences the need for provision of training and support for histopathologists in training during the transition to DP, and for consideration of their need to maintain competence and confidence with GS reporting.

Annotation-free learning of a spatio-temporal manifold of the cell life cycle.

The cell cycle is a complex biological phenomenon, which plays an important role in many cell biological processes and disease states. Machine learning is emerging to be a pivotal technique for the study of the cell cycle, resulting in a number of available tools and models for the analysis of the cell cycle. Most, however, heavily rely on expert annotations, prior knowledge of mechanisms, and imaging with several fluorescent markers to train their models. Many are also limited to processing only the spatial information in the cell images. In this work, we describe a different approach based on representation learning to construct a manifold of the cell life cycle. We trained our model such that the representations are learned without exhaustive annotations nor assumptions. Moreover, our model uses microscopy images derived from a single fluorescence channel and utilizes both the spatial and temporal information in these images. We show that even with fewer channels and self-supervision, information relevant to cell cycle analysis such as staging and estimation of cycle duration can still be extracted, which demonstrates the potential of our approach to aid future cell cycle studies and in discovery cell biology to probe and understand novel dynamic systems.

Multi-Scale Graphical Representation of Cell Environment.

We present a multi-scale graphical network that can capture the relevant representations of individual cell morphology, topological structure of cell communities in a tissue image, as well as whole slide level attributes. This helps to effectively merge the disease relevant cell morphology to the overall topological context within the sample, within one unified deep framework. From the explainability point of view, instead of empirical design, the graphs are designed with biomedical considerations in mind in order to have translational validity. We also provide a clinically interpretable visualisation of the cells and their micro- and macro-environment by leveraging label noise reduction. We demonstrate the efficacy of our methodology on myeloproliferative neoplasms (MPN), a haematopoietic stem cell disorder as an exemplar test case. The proposed method achieves an encouraging performance in the robust separation of different MPN subtypes in this exciting new dataset as part of this work.

A Graph Based Neural Network Approach to Immune Profiling of Multiplexed Tissue Samples.

Multiplexed immunofluorescence provides an un-precedented opportunity for studying specific cell-to-cell and cell microenvironment interactions. We employ graph neural networks to combine features obtained from tissue morphology with measurements of protein expression to profile the tumour microenvironment associated with different tumour stages. Our framework presents a new approach to analysing and processing these complex multi-dimensional datasets that overcomes some of the key challenges in analysing these data and opens up the opportunity to abstract biologically meaningful interactions.

Multi-class motion-based semantic segmentation for ureteroscopy and laser lithotripsy.

Ureteroscopy with laser lithotripsy has evolved as the most commonly used technique for the treatment of kidney stones. Automated segmentation of kidney stones and the laser fiber is an essential initial step to performing any automated quantitative analysis, particularly stone-size estimation, that can be used by the surgeon to decide if the stone requires further fragmentation. However, factors such as turbid fluid inside the cavity, specularities, motion blur due to kidney movements and camera motion, bleeding, and stone debris impact the quality of vision within the kidney, leading to extended operative times. To the best of our knowledge, this is the first attempt made towards multi-class segmentation in ureteroscopy and laser lithotripsy data. We propose an end-to-end convolution neural network (CNN) based learning framework for the segmentation of stones and laser fiber. The proposed approach utilizes two sub-networks: (I) HybResUNet, a hybrid version of residual U-Net, that uses residual connections in the encoder path of the U-Net to improve semantic predictions, and (II) a DVFNet that generates deformation vector field (DVF) predictions by leveraging motion differences between the adjacent video frames which is then used to prune the prediction maps. We also present ablation studies that combine different dilated convolutions, recurrent and residual connections, atrous spatial pyramid pooling, and attention gate models. Further, we propose a compound loss function that significantly boosts the segmentation performance in our data. We have also provided an ablation study to determine the optimal data augmentation strategy for our dataset. Our qualitative and quantitative results illustrate that our proposed method outperforms state-of-the-art methods such as UNet and DeepLabv3+ showing a DSC improvement of 4.15% and 13.34%, respectively, in our in vivo test dataset. We further show that our proposed model outperforms state-of-the-art methods on an unseen out-of-sample clinical dataset with a DSC improvement of 9.61%, 11%, and 5.24% over UNet, HybResUNet, and DeepLabv3+, respectively in the case of the stone class and an improvement of 31.79%, 22.15%, and 10.42% over UNet, HybResUNet, and DeepLabv3+, respectively, in case of the laser class.

RFID analysis of the complexity of cellular pathology workflow-An opportunity for digital pathology.

Digital pathology (DP) offers potential for time efficiency gains over an analog workflow however, to date, evidence supporting this claim is relatively lacking. Studies available concentrate on specific workflow points such as diagnostic reporting time, rather than overall efficiencies in slide logistics that might be expected. This is in part a result of the complexity and variation in analog working, and the challenge therefore in capturing this. We have utilized RFID technology to conduct a novel study capturing the movement of diagnostic cases within the analog pathway in a large teaching hospital setting, thus providing benchmark data for potential efficiency gains with DP. This technology overcomes the need to manually record data items and has facilitated the capture of both the physical journey of a case and the time associated with relevant components of the analog pathway predicted to be redundant in the digital setting. RFID tracking of 1,173 surgical pathology cases and over 30 staff in an analog cellular pathology workflow illustrates the complexity of the physical movement of slides within the department, which impacts on case traceability within the system. Detailed analysis of over 400 case journeys highlights redundant periods created by batching of slides at workflow points, including potentially 2-3 h for a case to become available for reporting after release from the lab, and variable lag-times prior to collection for reporting, and provides an illustration of patterns of lab and pathologist working within the analog setting. This study supports the challenge in evidencing efficiency gains to be anticipated with DP in the context of the variation and complexity of the analog pathway, but also evidences the efficiency gains that may be expected through a greater understanding of patterns of working and movement of cases. Such data may benefit other departments building a business case for DP.

Automated quality assessment of large digitised histology cohorts by artificial intelligence.

Research using whole slide images (WSIs) of histopathology slides has increased exponentially over recent years. Glass slides from retrospective cohorts, some with patient follow-up data are digitised for the development and validation of artificial intelligence (AI) tools. Such resources, therefore, become very important, with the need to ensure that their quality is of the standard necessary for downstream AI development. However, manual quality control of large cohorts of WSIs by visual assessment is unfeasible, and whilst quality control AI algorithms exist, these focus on bespoke aspects of image quality, e.g. focus, or use traditional machine-learning methods, which are unable to classify the range of potential image artefacts that should be considered. In this study, we have trained and validated a multi-task deep neural network to automate the process of quality control of a large retrospective cohort of prostate cases from which glass slides have been scanned several years after production, to determine both the usability of the images at the diagnostic level (considered in this study to be the minimal standard for research) and the common image artefacts present. Using a two-layer approach, quality overlays of WSIs were generated from a quality assessment (QA) undertaken at patch-level at [Formula: see text] magnification. From these quality overlays the slide-level quality scores were predicted and then compared to those generated by three specialist urological pathologists, with a Pearson correlation of 0.89 for overall 'usability' (at a diagnostic level), and 0.87 and 0.82 for focus and H&E staining quality scores respectively. To demonstrate its wider potential utility, we subsequently applied our QA pipeline to the TCGA prostate cancer cohort and to a colorectal cancer cohort, for comparison. Our model, designated as PathProfiler, indicates comparable predicted usability of images from the cohorts assessed (86-90% of WSIs predicted to be usable), and perhaps more significantly is able to predict WSIs that could benefit from an intervention such as re-scanning or re-staining for quality improvement. We have shown in this study that AI can be used to automate the process of quality control of large retrospective WSI cohorts to maximise their utility for research.

Learning Cellular Phenotypes through Supervision.

Image-based cell phenotyping is an important and open problem in computational pathology. The two principal challenges are: 1) making the cell cluster properties insensitive to experimental settings (like seed point and feature selection) and 2) ensuring that the phenotypes emerging are biologically relevant and support clinical reporting. To gauge robustness, we first compare the consistency of the phenotypes using self-supervised and supervised features. Through case classification, we analyse the relevance of the self-supervised and supervised feature sets with respect to the clinical diagnosis. In addition, we demonstrate how we can add model explainability through Shapley values to identify more disease relevant cellular phenotypes and measure their importance in context of the disease. Here, myeloproliferative neoplasms, a haematopoietic stem cell disorder, where one particular cell type is of diagnostic relevance is used as an exemplar. The experiments conducted on a set of bone marrow trephines demonstrate an improvement of 7.4 % in accuracy for case classification using cellular phenotypes derived from the supervised scenario.