A study of criteria for grading follicular lymphoma using a cell type classifier from pathology images based on complementary-label learning.

We propose a criterion for grading follicular lymphoma that is consistent with the intuitive evaluation, which is conducted by experienced pathologists. A criterion for grading follicular lymphoma is defined by the World Health Organization (WHO) based on the number of centroblasts and centrocytes within the field of view. However, the WHO criterion is not often used in clinical practice because it is impractical for pathologists to visually identify the cell type of each cell and count the number of centroblasts and centrocytes. Hence, based on the widespread use of digital pathology, we make it practical to identify and count the cell type by using image processing and then construct a criterion for grading based on the number of cells. Here, the problem is that labeling the cell type is not easy even for experienced pathologists. To alleviate this problem, we build a new dataset for cell type classification, which contains the pathologists' confusion records during labeling, and we construct the cell type classifier using complementary-label learning from this dataset. Then we propose a criterion based on the composition ratio of cell types that is consistent with the pathologists' grading. Our experiments demonstrate that the classifier can accurately identify cell types and the proposed criterion is more consistent with the pathologists' grading than the current WHO criterion.

Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma.

In this study, we present a deep-learning-based multimodal classification method for lymphoma diagnosis in digital pathology, which utilizes a whole slide image (WSI) as the primary image data and flow cytometry (FCM) data as auxiliary information. In pathological diagnosis of malignant lymphoma, FCM serves as valuable auxiliary information during the diagnosis process, offering useful insights into predicting the major class (superclass) of subtypes. By incorporating both images and FCM data into the classification process, we can develop a method that mimics the diagnostic process of pathologists, enhancing the explainability. In order to incorporate the hierarchical structure between superclasses and their subclasses, the proposed method utilizes a network structure that effectively combines the mixture of experts (MoE) and multiple instance learning (MIL) techniques, where MIL is widely recognized for its effectiveness in handling WSIs in digital pathology. The MoE network in the proposed method consists of a gating network for superclass classification and multiple expert networks for (sub)class classification, specialized for each superclass. To evaluate the effectiveness of our method, we conducted experiments involving a six-class classification task using 600 lymphoma cases. The proposed method achieved a classification accuracy of 72.3%, surpassing the 69.5% obtained through the straightforward combination of FCM and images, as well as the 70.2% achieved by the method using only images. Moreover, the combination of multiple weights in the MoE and MIL allows for the visualization of specific cellular and tumor regions, resulting in a highly explanatory model that cannot be attained with conventional methods. It is anticipated that by targeting a larger number of classes and increasing the number of expert networks, the proposed method could be effectively applied to the real problem of lymphoma diagnosis.

The Detection of Neoplastic Cells Using Objective Cytomorphologic Parameters in Malignant Lymphoma.

Pathologic evaluation is the most crucial method for diagnosing malignant lymphomas. However, there are no established diagnostic criteria for evaluating pathologic morphology. We manually circled cell nuclei in the lesions of 10 patients with diffuse large B-cell lymphoma (DLBCL), follicular lymphoma, and reactive lymphadenitis. Seventeen parameters related to nuclear shape, color, and other characteristics were measured. We attempted to compare the statistical differences between these subtypes and extract distinctive disease-specific populations on the basis of these parameters. Statistically significant differences were observed between the different types of lymphoma for many of the 17 parameters. Through t-distributed stochastic neighbor embedding analysis, we extracted a cluster of cells that showed distinctive features of DLBCL and were not found in follicular lymphoma or reactive lymphadenitis. We created a decision tree to identify the characteristics of the cells within that cluster. Based on a 5-fold cross-validation study, the average sensitivity, specificity, and accuracy obtained were 84.1%, 98.4%, and 97.3%, respectively. A similar result was achieved using a validation experiment. Important parameters that indicate the features of DLBCL include Area, ConcaveCount, MaxGray, and ModeGray. By quantifying pathologic morphology, it was possible to objectively represent the cell morphology specific to each lymphoma subtype using quantitative indicators. The quantified morphologic information has the potential to serve as a reproducible and flexible diagnostic tool.

Improved artificial intelligence discrimination of minor histological populations by supplementing with color-adjusted images.

Despite the dedicated research of artificial intelligence (AI) for pathological images, the construction of AI applicable to histopathological tissue subtypes, is limited by insufficient dataset collection owing to disease infrequency. Here, we present a solution involving the addition of supplemental tissue array (TA) images that are adjusted to the tonality of the main data using a cycle-consistent generative adversarial network (CycleGAN) to the training data for rare tissue types. F1 scores of rare tissue types that constitute < 1.2% of the training data were significantly increased by improving recall values after adding color-adjusted TA images constituting < 0.65% of total training patches. The detector also enabled the equivalent discrimination of clinical images from two distinct hospitals and the capability was more increased following color-correction of test data before AI identification (F1 score from 45.2 ± 27.1 to 77.1 ± 10.3, p < 0.01). These methods also classified intraoperative frozen sections, while excessive supplementation paradoxically decreased F1 scores. These results identify strategies for building an AI that preserves the imbalance between training data with large differences in actual disease frequencies, which is important for constructing AI for practical histopathological classification.

Transformer-based personalized attention mechanism for medical images with clinical records.

In medical image diagnosis, identifying the attention region, i.e., the region of interest for which the diagnosis is made, is an important task. Various methods have been developed to automatically identify target regions from given medical images. However, in actual medical practice, the diagnosis is made based on both the images and various clinical records. Consequently, pathologists examine medical images with prior knowledge of the patients and the attention regions may change depending on the clinical records. In this study, we propose a method, called the Personalized Attention Mechanism (PersAM) method, by which the attention regions in medical images according to the clinical records. The primary idea underlying the PersAM method is the encoding of the relationships between medical images and clinical records using a variant of the Transformer architecture. To demonstrate the effectiveness of the PersAM method, we applied it to a large-scale digital pathology problem involving identifying the subtypes of 842 malignant lymphoma patients based on their gigapixel whole-slide images and clinical records.

Case-based similar image retrieval for weakly annotated large histopathological images of malignant lymphoma using deep metric learning.

In the present study, we propose a novel case-based similar image retrieval (SIR) method for hematoxylin and eosin (H&E) stained histopathological images of malignant lymphoma. When a whole slide image (WSI) is used as an input query, it is desirable to be able to retrieve similar cases by focusing on image patches in pathologically important regions such as tumor cells. To address this problem, we employ attention-based multiple instance learning, which enables us to focus on tumor-specific regions when the similarity between cases is computed. Moreover, we employ contrastive distance metric learning to incorporate immunohistochemical (IHC) staining patterns as useful supervised information for defining appropriate similarity between heterogeneous malignant lymphoma cases. In the experiment with 249 malignant lymphoma patients, we confirmed that the proposed method exhibited higher evaluation measures than the baseline case-based SIR methods. Furthermore, the subjective evaluation by pathologists revealed that our similarity measure using IHC staining patterns is appropriate for representing the similarity of H&E stained tissue images for malignant lymphoma.

Subtype classification of malignant lymphoma using immunohistochemical staining pattern.

PURPOSE: For the image classification problem, the construction of appropriate training data is important for improving the generalization ability of the classifier in particular when the size of the training data is small. We propose a method that quantitatively evaluates the typicality of a hematoxylin-and-eosin (H&E)-stained tissue slide from a set of immunohistochemical (IHC) stains and applies the typicality to instance selection for the construction of classifiers that predict the subtype of malignant lymphoma to improve the generalization ability. METHODS: We define the typicality of the H&E-stained tissue slides by the ratio of the probability density of the IHC staining patterns on low-dimensional embedded space. Employing a multiple-instance-learning-based convolutional neural network for the construction of the subtype classifier without the annotations indicating cancerous regions in whole slide images, we select the training data by referring to the evaluated typicality to improve the generalization ability. We demonstrate the effectiveness of the instance selection based on the proposed typicality in a three-class subtype classification of 262 malignant lymphoma cases. RESULTS: In the experiment, we confirmed that the subtypes of typical instances could be predicted more accurately than those of atypical instances. Furthermore, it was confirmed that instance selection for the training data based on the proposed typicality improved the generalization ability of the classifier, wherein the classification accuracy was improved from 0.664 to 0.683 compared with the baseline method when the training data was constructed focusing on typical instances. CONCLUSION: The experimental results showed that the typicality of the H&E-stained tissue slides computed from IHC staining patterns is useful as a criterion for instance selection to enhance the generalization ability, and this typicality could be employed for instance selection under some practical limitations.

Manifold Modeling in Embedded Space: An Interpretable Alternative to Deep Image Prior.

Deep image prior (DIP), which uses a deep convolutional network (ConvNet) structure as an image prior, has attracted wide attention in computer vision and machine learning. DIP empirically shows the effectiveness of the ConvNet structures for various image restoration applications. However, why the DIP works so well is still unknown. In addition, the reason why the convolution operation is useful in image reconstruction, or image enhancement is not very clear. This study tackles this ambiguity of ConvNet/DIP by proposing an interpretable approach that divides the convolution into "delay embedding" and "transformation" (i.e., encoder-decoder). Our approach is a simple, but essential, image/tensor modeling method that is closely related to self-similarity. The proposed method is called manifold modeling in embedded space (MMES) since it is implemented using a denoising autoencoder in combination with a multiway delay-embedding transform. In spite of its simplicity, MMES can obtain quite similar results to DIP on image/tensor completion, super-resolution, deconvolution, and denoising. In addition, MMES is proven to be competitive with DIP, as shown in our experiments. These results can also facilitate interpretation/characterization of DIP from the perspective of a "low-dimensional patch-manifold prior."

Real-time estimation of electric fields induced by transcranial magnetic stimulation with deep neural networks.

BACKGROUND: Transcranial magnetic stimulation (TMS) plays an important role in treatment of mental and neurological illnesses, and neurosurgery. However, it is difficult to target specific brain regions accurately because the complex anatomy of the brain substantially affects the shape and strength of the electric fields induced by the TMS coil. A volume conductor model can be used for determining the accurate electric fields; however, the construction of subject-specific anatomical head structures is time-consuming. OBJECTIVE: The aim of this study is to propose a method to estimate electric fields induced by TMS from only T1 magnetic resonance (MR) images, without constructing a subject-specific anatomical model. METHODS: Very large sets of electric fields in the brain of subject-specific anatomical models, which are constructed from T1 and T2 MR images, are computed by a volume conductor model. The relation between electric field distribution and T1 MR images is used for machine learning. Deep neural network (DNN) models are applied for the first time to electric field estimation. RESULTS: By determining the relationships between the T1 MR images and electric fields by DNN models, the process of electric field estimation is markedly accelerated (to 0.03 s) due to the absence of a requirement for anatomical head structure reconstruction and volume conductor computation. Validation shows promising estimation accuracy, and rapid computations of the DNN model are apt for practical applications. CONCLUSION: The study showed that the DNN model can estimate the electric fields from only T1 MR images and requires low computation time, suggesting the possibility of using machine learning for real-time electric field estimation in navigated TMS.

Robust 3D image reconstruction of pancreatic cancer tumors from histopathological images with different stains and its quantitative performance evaluation.

PURPOSE: Histopathological imaging is widely used for the analysis and diagnosis of multiple diseases. Several methods have been proposed for the 3D reconstruction of pathological images, captured from thin sections of a given specimen, which get nonlinearly deformed due to the preparation process. The majority of the available methods for registering such images use the degree of matching of adjacent images as the criteria for registration, which can result in unnatural deformations of the anatomical structures. Moreover, most methods assume that the same staining is used for all images, when in fact multiple staining is usually applied in order to enhance different structures in the images. METHODS: This paper proposes a non-rigid 3D reconstruction method based on the assumption that internal structures on the original tissue must be smooth and continuous. Landmarks are detected along anatomical structures using template matching based on normalized cross-correlation (NCC), forming jagged shape trajectories that traverse several slices. The registration process smooths out these trajectories and deforms the images accordingly. Artifacts are automatically handled by using the confidence of the NCC in order to reject unreliable landmarks. RESULTS: The proposed method was applied to a large series of histological sections from the pancreas of a KPC mouse. Some portions were dyed primarily with HE stain, while others were dyed alternately with HE, CK19, MT and Ki67 stains. A new evaluation method is proposed to quantitatively evaluate the smoothness and isotropy of the obtained reconstructions, both for single and multiple staining. CONCLUSIONS: The experimental results show that the proposed method produces smooth and nearly isotropic 3D reconstructions of pathological images with either single or multiple stains. From these reconstructions, microanatomical structures enhanced by different stains can be simultaneously observed.

Super-Resolution of Magnetic Resonance Images via Convex Optimization with Local and Global Prior Regularization and Spectrum Fitting.

Given a low-resolution image, there are many challenges to obtain a super-resolved, high-resolution image. Many of those approaches try to simultaneously upsample and deblur an image in signal domain. However, the nature of the super-resolution is to restore high-frequency components in frequency domain rather than upsampling in signal domain. In that sense, there is a close relationship between super-resolution of an image and extrapolation of the spectrum. In this study, we propose a novel framework for super-resolution, where the high-frequency components are theoretically restored with respect to the frequency fidelities. This framework helps to introduce multiple simultaneous regularizers in both signal and frequency domains. Furthermore, we propose a new super-resolution model where frequency fidelity, low-rank (LR) prior, low total variation (TV) prior, and boundary prior are considered at once. The proposed method is formulated as a convex optimization problem which can be solved by the alternating direction method of multipliers. The proposed method is the generalized form of the multiple super-resolution methods such as TV super-resolution, LR and TV super-resolution, and the Gerchberg method. Experimental results show the utility of the proposed method comparing with some existing methods using both simulational and practical images.

A study on graphical model structure for representing statistical shape model of point distribution model.

In this article, the authors demonstrate that you can improve the performance of the registration of a point distribution model (PDM) by accurately estimating the structure of an undirected graphical model that represents the statistical shape model (SSM) of a target surface. Many existing methods for constructing SSMs determine the structure of the graphical model without analyzing the conditional dependencies among the points in PDM, though an edge in the PDM should link two nodes if and only if they are conditionally dependent. In this study, the authors employed four popular methods for estimating the structure of graphical model and obtained four different SSMs from an identical set of training surfaces. The registration performances of the SSMs were experimentally compared, and the results showed that the graphical lasso, which could estimate more accurate structure of the graphical model by avoiding the overfitting to the training data, outperformed the other methods.

Quantitative evaluation of myocardial function by a volume-normalized map generated from relative blood flow.

Our study aimed to quantitatively evaluate blood flow in the left ventricle (LV) of apical hypertrophic cardiomyopathy (APH) by combining wall thickness obtained from cardiac magnetic resonance imaging (MRI) and myocardial perfusion from single-photon emission computed tomography (SPECT). In this study, we considered paired MRI and myocardial perfusion SPECT from ten patients with APH and ten normals. Myocardial walls were detected using a level set method, and blood flow per unit myocardial volume was calculated using 3D surface-based registration between the MRI and SPECT images. We defined relative blood flow based on the maximum in the whole myocardial region. Accuracies of wall detection and registration were around 2.50 mm and 2.95 mm, respectively. We finally created a bull's-eye map to evaluate wall thickness, blood flow (cardiac perfusion) and blood flow per unit myocardial volume. In patients with APH, their wall thicknesses were over 10 mm. Decreased blood flow per unit myocardial volume was detected in the cardiac apex by calculation using wall thickness from MRI and blood flow from SPECT. The relative unit blood flow of the APH group was 1/7 times that of the normals in the apex. This normalization by myocardial volume distinguishes cases of APH whose SPECT images resemble the distributions of normal cases.