End-to-end deep learning method for predicting hormonal treatment response in women with atypical endometrial hyperplasia or endometrial cancer.

Purpose: Endometrial cancer (EC) is the most common gynecologic malignancy in the United States, and atypical endometrial hyperplasia (AEH) is considered a high-risk precursor to EC. Hormone therapies and hysterectomy are practical treatment options for AEH and early-stage EC. Some patients prefer hormone therapies for reasons such as fertility preservation or being poor surgical candidates. However, accurate prediction of an individual patient's response to hormonal treatment would allow for personalized and potentially improved recommendations for these conditions. This study aims to explore the feasibility of using deep learning models on whole slide images (WSI) of endometrial tissue samples to predict the patient's response to hormonal treatment. Approach: We curated a clinical WSI dataset of 112 patients from two clinical sites. An expert pathologist annotated these images by outlining AEH/EC regions. We developed an end-to-end machine learning model with mixed supervision. The model is based on image patches extracted from pathologist-annotated AEH/EC regions. Either an unsupervised deep learning architecture (Autoencoder or ResNet50), or non-deep learning (radiomics feature extraction) is used to embed the images into a low-dimensional space, followed by fully connected layers for binary prediction, which was trained with binary responder/non-responder labels established by pathologists. We used stratified sampling to partition the dataset into a development set and a test set for internal validation of the performance of our models. Results: The autoencoder model yielded an AUROC of 0.80 with 95% CI [0.63, 0.95] on the independent test set for the task of predicting a patient with AEH/EC as a responder vs non-responder to hormonal treatment. Conclusions: These findings demonstrate the potential of using mixed supervised machine learning models on WSIs for predicting the response to hormonal treatment in AEH/EC patients.

Weakly Supervised Deep Learning for Predicting the Response to Hormonal Treatment of Women with Atypical Endometrial Hyperplasia: A Feasibility Study.

Endometrial cancer (EC) is the most common gynecologic malignancy in the US and complex atypical hyperplasia (CAH) is considered a high-risk precursor to EC. Treatment options for CAH and early-stage EC include hormone therapies and hysterectomy with the former preferred by certain patients, e.g., for fertility preservation or poor surgical candidates. Accurate prediction of response to hormonal treatment would allow for personalized and potentially improved recommendations for the treatment of these conditions. In this study, we investigate the feasibility of utilizing weakly supervised deep learning models on whole slide images of endometrial tissue samples for the prediction of patient response to hormonal treatment. We curated a clinical whole-slide-image (WSI) dataset of 112 patients from two clinical sites. We developed an end-to-end machine learning model using WSIs of endometrial specimens for the prediction of hormonal treatment response among women with CAH/EC. The model takes patches extracted from pathologist-annotated CAH/EC regions as input and utilizes an unsupervised deep learning architecture (Autoencoder or ResNet50) to embed the images into a low-dimensional space, followed by fully connected layers for binary prediction. Our autoencoder model yielded an AUC of 0.79 with 95% CI [0.61, 0.98] on a hold-out test set in the task of predicting a patient with CAH/EC as a responder vs non-responder to hormonal treatment. Our results, demonstrate the potential for using weakly supervised machine learning models on WSIs for predicting response to hormonal treatment of CAH/EC patients.

Histopathology images predict multi-omics aberrations and prognoses in colorectal cancer patients.

Histopathologic assessment is indispensable for diagnosing colorectal cancer (CRC). However, manual evaluation of the diseased tissues under the microscope cannot reliably inform patient prognosis or genomic variations crucial for treatment selections. To address these challenges, we develop the Multi-omics Multi-cohort Assessment (MOMA) platform, an explainable machine learning approach, to systematically identify and interpret the relationship between patients' histologic patterns, multi-omics, and clinical profiles in three large patient cohorts (n = 1888). MOMA successfully predicts the overall survival, disease-free survival (log-rank test P-value<0.05), and copy number alterations of CRC patients. In addition, our approaches identify interpretable pathology patterns predictive of gene expression profiles, microsatellite instability status, and clinically actionable genetic alterations. We show that MOMA models are generalizable to multiple patient populations with different demographic compositions and pathology images collected from distinctive digitization methods. Our machine learning approaches provide clinically actionable predictions that could inform treatments for colorectal cancer patients.

Prognostic role of inflammatory diets in colorectal cancer overall and in strata of tumor-infiltrating lymphocyte levels.

BACKGROUND: Certain dietary patterns can elicit systemic and intestinal inflammatory responses, which may influence adaptive anti-tumor immune responses and tumor behavior. We hypothesized that pro-inflammatory diets might be associated with higher colorectal cancer mortality and that the association might be stronger for tumors with lower immune responses. METHODS: We calculated an empirical dietary inflammatory pattern (EDIP) score in 2829 patients among 3988 incident rectal and colon carcinoma cases in the Nurses' Health Study and Health Professionals Follow-up Study. Using Cox proportional hazards regression analyses, we examined the prognostic association of EDIP scores and whether it might be modified by histopathologic immune reaction (in 1192 patients with available data). RESULTS: Higher EDIP scores after colorectal cancer diagnosis were associated with worse survival, with multivariable-adjusted hazard ratios (HRs) for the highest versus lowest tertile of 1.41 (95% confidence interval [CI]: 1.13-1.77; Ptrend = 0.003) for 5-year colorectal cancer-specific mortality and 1.44 (95% CI, 1.19-1.74; Ptrend = 0.0004) for 5-year all-cause mortality. The association of post-diagnosis EDIP scores with 5-year colorectal cancer-specific mortality differed by degrees of tumor-infiltrating lymphocytes (TIL; Pinteraction = .002) but not by three other lymphocytic reaction patterns. The multivariable-adjusted, 5-year colorectal cancer-specific mortality HRs for the highest versus lowest EDIP tertile were 1.59 (95% CI: 1.01-2.53) in TIL-absent/low cases and 0.48 (95% CI: 0.16-1.48) in TIL-intermediate/high cases. CONCLUSIONS: Pro-inflammatory diets after colorectal cancer diagnosis were associated with increased mortality, particularly in patients with absent or low TIL.

Correction of topological errors in automated traces of neurites.

Our understanding of synaptic connectivity in the brain relies on the ability to accurately trace sparsely labeled neurons from 3D optical microscopy stacks of images. A variety of automated algorithms and software tools have been developed for this task. These algorithms can capture the general layout of neurites with high fidelity, but the resulting traces often contain topological errors such as broken and incorrectly merged branches. Even a small number of isolated topological errors can drastically alter the connectivity, and therefore, their detection and correction are paramount for connectomics studies. Here, we describe an automated trace proofreading approach that utilizes machine learning to correct trace topology. Multiple stacks of neuron images were traced by two users to create a labeled dataset and assess the baseline of inter-user variability. All traces were then disconnected at branch points and a deep neural network was trained to detect the correct way of reconnecting the branches. Custom morphological features were generated for each cluster of branch points, in a way that is dependent on a merging scenario but invariant to translations, rotations, and reflections of the cluster in the imaging plane. The features and image volume centered at the branch point were used for training a neural network that concatenates these input streams and outputs the confidence measure for different branch merging scenarios. The designed method significantly reduces the number of topological errors in automated traces and comes close to the accuracy achieved by expert users which is the gold standard in the field.

Accurate registration of in vivo time-lapse images.

In vivo imaging experiments often require automated detection and tracking of changes in the specimen. These tasks can be hindered by variations in the position and orientation of the specimen relative to the microscope, as well as by linear and nonlinear tissue deformations. We propose a feature-based registration method, coupled with optimal transformations, designed to address these problems in 3D time-lapse microscopy images. Features are detected as local regions of maximum intensity in source and target image stacks, and their bipartite intensity dissimilarity matrix is used as an input to the Hungarian algorithm to establish initial correspondences. A random sampling refinement method is employed to eliminate outliers, and the resulting set of corresponding features is used to determine an optimal translation, rigid, affine, or B-spline transformation for the registration of the source and target images. Accuracy of the proposed algorithm was tested on fluorescently labeled axons imaged over a 68-day period with a two-photon laser scanning microscope. To that end, multiple axons in individual stacks of images were traced semi-manually and optimized in 3D, and the distances between the corresponding traces were measured before and after the registration. The results show that there is a progressive improvement in the registration accuracy with increasing complexity of the transformations. In particular, sub-micrometer accuracy (2-3 voxels) was achieved with the regularized affine and B-spline transformations.

Artificial neural network filters for enhancing 3D optical microscopy images of neurites.

The ability to extract accurate morphology of labeled neurons from microscopy images is crucial for mapping brain connectivity and for understanding changes in connectivity that underlie learning and neurological disorders. There are, however, two problems, specific to optical microscopy imaging of neurons, which make accurate neuron tracing exceedingly challenging: (i) neurites can appear broken due to inhomogeneous labeling and (ii) neurites can appear fused in 3D due to limited resolution. Here, we propose and evaluate several artificial neural network (ANN) architectures and conventional image enhancement filters with the aim of alleviating both problems. We developed four image quality metrics to evaluate the effects of the proposed filters: normalized intensity in the cross-over regions between neurites, effective radius of neurites, coefficient of variation of intensity along neurites, and local background to neurite intensity ratio. Our results show that ANN-based filters, trained on optimized semi-manual traces of neurites, can significantly outperform conventional filters. In particular, U-Net based filtering can virtually eliminate background intensity, while also reducing the effective radius of neurites to nearly 1 voxel. In addition, this filter significantly decreases intensity in the cross-over regions between neurites and reduces fluctuations of intensity on neurites' centerlines. These results suggest that including an ANN-based filtering step, which does not require substantial extra time or computing power, can be beneficial for automated neuron tracing projects.

Geometric feature descriptor and dissimilarity-based registration of remotely sensed imagery.

Image registration of remotely sensed imagery is challenging, as complex deformations are common. Different deformations, such as affine and homogenous transformation, combined with multimodal data capturing can emerge in the data acquisition process. These effects, when combined, tend to compromise the performance of the currently available registration methods. A new image transform, known as geometric mean projection transform, is introduced in this work. As it is deformation invariant, it can be employed as a feature descriptor, whereby it analyzes the functions of all vertical and horizontal signals in local areas of the image. Moreover, an invariant feature correspondence method is proposed as a point matching algorithm, which incorporates new descriptor's dissimilarity metric. Considering the image as a signal, the proposed approach utilizes a square Eigenvector correlation (SEC) based on the Eigenvector properties. In our experiments on standard test images sourced from "Featurespace" and "IKONOS" datasets, the proposed method achieved higher average accuracy relative to that obtained from other state of the art image registration techniques. The accuracy of the proposed method was assessed using six standard evaluation metrics. Furthermore, statistical analyses, including t-test and Friedman test, demonstrate that the method developed as a part of this study is superior to the existing methods.

Invariant Feature Matching for Image Registration Application Based on New Dissimilarity of Spatial Features.

An invariant feature matching method is proposed as a spatially invariant feature matching approach. Deformation effects, such as affine and homography, change the local information within the image and can result in ambiguous local information pertaining to image points. New method based on dissimilarity values, which measures the dissimilarity of the features through the path based on Eigenvector properties, is proposed. Evidence shows that existing matching techniques using similarity metrics--such as normalized cross-correlation, squared sum of intensity differences and correlation coefficient--are insufficient for achieving adequate results under different image deformations. Thus, new descriptor's similarity metrics based on normalized Eigenvector correlation and signal directional differences, which are robust under local variation of the image information, are proposed to establish an efficient feature matching technique. The method proposed in this study measures the dissimilarity in the signal frequency along the path between two features. Moreover, these dissimilarity values are accumulated in a 2D dissimilarity space, allowing accurate corresponding features to be extracted based on the cumulative space using a voting strategy. This method can be used in image registration applications, as it overcomes the limitations of the existing approaches. The output results demonstrate that the proposed technique outperforms the other methods when evaluated using a standard dataset, in terms of precision-recall and corner correspondence.

Double Line Image Rotation.

This paper proposes a fast algorithm for rotating images while preserving their quality. The new approach rotates images based on vertical or horizontal lines in the original image and their rotated equation in the target image. The proposed method is a one-pass method that determines a based-line equation in the target image and extracts all corresponding pixels on the base-line. Floating-point multiplications are performed to calculate the base-line in the target image, and other line coordinates are calculated using integer addition or subtraction and logical justifications from the base-line pixel coordinates in the target image. To avoid a heterogeneous distance between rotated pixels in the target image, each line rotates to two adjacent lines. The proposed method yields good performance in terms of speed and quality according to the results of an analysis of the computation speed and accuracy.

Contour-based corner detection and classification by using mean projection transform.

Image corner detection is a fundamental task in computer vision. Many applications require reliable detectors to accurately detect corner points, commonly achieved by using image contour information. The curvature definition is sensitive to local variation and edge aliasing, and available smoothing methods are not sufficient to address these problems properly. Hence, we propose Mean Projection Transform (MPT) as a corner classifier and parabolic fit approximation to form a robust detector. The first step is to extract corner candidates using MPT based on the integral properties of the local contours in both the horizontal and vertical directions. Then, an approximation of the parabolic fit is calculated to localize the candidate corner points. The proposed method presents fewer false-positive (FP) and false-negative (FN) points compared with recent standard corner detection techniques, especially in comparison with curvature scale space (CSS) methods. Moreover, a new evaluation metric, called accuracy of repeatability (AR), is introduced. AR combines repeatability and the localization error (Le) for finding the probability of correct detection in the target image. The output results exhibit better repeatability, localization, and AR for the detected points compared with the criteria in original and transformed images.