Multi-channel feature extraction for virtual histological staining of photon absorption remote sensing images.

Accurate and fast histological staining is crucial in histopathology, impacting diagnostic precision and reliability. Traditional staining methods are time-consuming and subjective, causing delays in diagnosis. Digital pathology plays a vital role in advancing and optimizing histology processes to improve efficiency and reduce turnaround times. This study introduces a novel deep learning-based framework for virtual histological staining using photon absorption remote sensing (PARS) images. By extracting features from PARS time-resolved signals using a variant of the K-means method, valuable multi-modal information is captured. The proposed multi-channel cycleGAN model expands on the traditional cycleGAN framework, allowing the inclusion of additional features. Experimental results reveal that specific combinations of features outperform the conventional channels by improving the labeling of tissue structures prior to model training. Applied to human skin and mouse brain tissue, the results underscore the significance of choosing the optimal combination of features, as it reveals a substantial visual and quantitative concurrence between the virtually stained and the gold standard chemically stained hematoxylin and eosin images, surpassing the performance of other feature combinations. Accurate virtual staining is valuable for reliable diagnostic information, aiding pathologists in disease classification, grading, and treatment planning. This study aims to advance label-free histological imaging and opens doors for intraoperative microscopy applications.

Deep ROC Analysis and AUC as Balanced Average Accuracy, for Improved Classifier Selection, Audit and Explanation.

Optimal performance is desired for decision-making in any field with binary classifiers and diagnostic tests, however common performance measures lack depth in information. The area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve are too general because they evaluate all decision thresholds including unrealistic ones. Conversely, accuracy, sensitivity, specificity, positive predictive value and the F1 score are too specific-they are measured at a single threshold that is optimal for some instances, but not others, which is not equitable. In between both approaches, we propose deep ROC analysis to measure performance in multiple groups of predicted risk (like calibration), or groups of true positive rate or false positive rate. In each group, we measure the group AUC (properly), normalized group AUC, and averages of: sensitivity, specificity, positive and negative predictive value, and likelihood ratio positive and negative. The measurements can be compared between groups, to whole measures, to point measures and between models. We also provide a new interpretation of AUC in whole or part, as balanced average accuracy, relevant to individuals instead of pairs. We evaluate models in three case studies using our method and Python toolkit and confirm its utility.

Deep Learning for Instance Retrieval: A Survey.

In recent years a vast amount of visual content has been generated and shared from many fields, such as social media platforms, medical imaging, and robotics. This abundance of content creation and sharing has introduced new challenges, particularly that of searching databases for similar content - Content Based Image Retrieval (CBIR) - a long-established research area in which improved efficiency and accuracy are needed for real-time retrieval. Artificial intelligence has made progress in CBIR and has significantly facilitated the process of instance search. In this survey we review recent instance retrieval works that are developed based on deep learning algorithms and techniques, with the survey organized by deep feature extraction, feature embedding and aggregation methods, and network fine-tuning strategies. Our survey considers a wide variety of recent methods, whereby we identify milestone work, reveal connections among various methods and present the commonly used benchmarks, evaluation results, common challenges, and propose promising future directions.

Time-domain feature extraction for target specificity in photoacoustic remote sensing microscopy.

Photoacoustic remote sensing (PARS) microscopy is an emerging label-free optical absorption imaging modality. PARS operates by capturing nanosecond-scale optical fluctuations produced by photoacoustic pressures. These time-domain (TD) variations are usually projected by amplitude to determine optical absorption magnitude. However, valuable details on a target's material properties (e.g., density, speed of sound) are contained within the TD signals. This work uses a novel, to the best of our knowledge, clustering method to learn TD features, based on signal shape, which relate to underlying material traits. A modified K-means method is used to cluster TD data, capturing representative signal features. These features are then used to form virtual colorizations which may highlight tissues based on their underlying material properties. Applied in fresh resected murine brain tissue, colorized visualizations highlight distinct regions of tissue. This may potentially facilitate differentiation of tissue constituents (e.g., myelinated and unmyelinated axons, cell nuclei) in a single acquisition.

Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS).

Histopathological visualizations are a pillar of modern medicine and biological research. Surgical oncology relies exclusively on post-operative histology to determine definitive surgical success and guide adjuvant treatments. The current histology workflow is based on bright-field microscopic assessment of histochemical stained tissues and has some major limitations. For example, the preparation of stained specimens for brightfield assessment requires lengthy sample processing, delaying interventions for days or even weeks. Therefore, there is a pressing need for improved histopathology methods. In this paper, we present a deep-learning-based approach for virtual label-free histochemical staining of total-absorption photoacoustic remote sensing (TA-PARS) images of unstained tissue. TA-PARS provides an array of directly measured label-free contrasts such as scattering and total absorption (radiative and non-radiative), ideal for developing H&E colorizations without the need to infer arbitrary tissue structures. We use a Pix2Pix generative adversarial network to develop visualizations analogous to H&E staining from label-free TA-PARS images. Thin sections of human skin tissue were first virtually stained with the TA-PARS, then were chemically stained with H&E producing a one-to-one comparison between the virtual and chemical staining. The one-to-one matched virtually- and chemically- stained images exhibit high concordance validating the digital colorization of the TA-PARS images against the gold standard H&E. TA-PARS images were reviewed by four dermatologic pathologists who confirmed they are of diagnostic quality, and that resolution, contrast, and color permitted interpretation as if they were H&E. The presented approach paves the way for the development of TA-PARS slide-free histological imaging, which promises to dramatically reduce the time from specimen resection to histological imaging.

Deep learning methods for inverse problems.

In this paper we investigate a variety of deep learning strategies for solving inverse problems. We classify existing deep learning solutions for inverse problems into three categories of Direct Mapping, Data Consistency Optimizer, and Deep Regularizer. We choose a sample of each inverse problem type, so as to compare the robustness of the three categories, and report a statistical analysis of their differences. We perform extensive experiments on the classic problem of linear regression and three well-known inverse problems in computer vision, namely image denoising, 3D human face inverse rendering, and object tracking, in presence of noise and outliers, are selected as representative prototypes for each class of inverse problems. The overall results and the statistical analyses show that the solution categories have a robustness behaviour dependent on the type of inverse problem domain, and specifically dependent on whether or not the problem includes measurement outliers. Based on our experimental results, we conclude by proposing the most robust solution category for each inverse problem class.

In-vivo functional and structural retinal imaging using multiwavelength photoacoustic remote sensing microscopy.

Many important eye diseases as well as systemic disorders manifest themselves in the retina. Retinal imaging technologies are rapidly growing and can provide ever-increasing amounts of information about the structure, function, and molecular composition of retinal tissue in-vivo. Photoacoustic remote sensing (PARS) is a novel imaging modality based on all-optical detection of photoacoustic signals, which makes it suitable for a wide range of medical applications. In this study, PARS is applied for in-vivo imaging of the retina and estimating oxygen saturation in the retinal vasculature. To our knowledge, this is the first time that a non-contact photoacoustic imaging technique is applied for in-vivo imaging of the retina. Here, optical coherence tomography is also used as a well-established retinal imaging technique to navigate the PARS imaging beams and demonstrate the capabilities of the optical imaging setup. The system is applied for in-vivo imaging of both microanatomy and the microvasculature of the retina. The developed system has the potential to advance the understanding of the ocular environment and to help in monitoring of ophthalmic diseases.

Functional photoacoustic remote sensing microscopy using a stabilized temperature-regulated stimulated Raman scattering light source.

Stimulated Raman scattering (SRS) has been widely used in functional photoacoustic microscopy to generate multiwavelength light and target multiple chromophores inside tissues. Despite offering a simple, cost-effective technique with a high pulse repetition rate; it suffers from pulse-to-pulse intensity fluctuations and power drift that can affect image quality. Here, we propose a new technique to improve the temporal stability of the pulsed SRS multiwavelength source. We achieve this by lowering the temperature of the SRS medium. The results suggest that a decrease in temperature causes an improvement of temporal stability of the output, considerable rise in the intensity of the SRS peaks, and significant increase of SRS cross section. The application of the method is shown for in vivo functional imaging of capillary networks in a chicken embryo chorioallantois membrane using photoacoustic remote sensing microscopy.

Improving maximal safe brain tumor resection with photoacoustic remote sensing microscopy.

Malignant brain tumors are among the deadliest neoplasms with the lowest survival rates of any cancer type. In considering surgical tumor resection, suboptimal extent of resection is linked to poor clinical outcomes and lower overall survival rates. Currently available tools for intraoperative histopathological assessment require an average of 20 min processing and are of limited diagnostic quality for guiding surgeries. Consequently, there is an unaddressed need for a rapid imaging technique to guide maximal resection of brain tumors. Working towards this goal, presented here is an all optical non-contact label-free reflection mode photoacoustic remote sensing (PARS) microscope. By using a tunable excitation laser, PARS takes advantage of the endogenous optical absorption peaks of DNA and cytoplasm to achieve virtual contrast analogous to standard hematoxylin and eosin (H&E) staining. In conjunction, a fast 266 nm excitation is used to generate large grossing scans and rapidly assess small fields in real-time with hematoxylin-like contrast. Images obtained using this technique show comparable quality and contrast to the current standard for histopathological assessment of brain tissues. Using the proposed method, rapid, high-throughput, histological-like imaging was achieved in unstained brain tissues, indicating PARS' utility for intraoperative guidance to improve extent of surgical resection.

A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms.

BACKGROUND: In classification and diagnostic testing, the receiver-operator characteristic (ROC) plot and the area under the ROC curve (AUC) describe how an adjustable threshold causes changes in two types of error: false positives and false negatives. Only part of the ROC curve and AUC are informative however when they are used with imbalanced data. Hence, alternatives to the AUC have been proposed, such as the partial AUC and the area under the precision-recall curve. However, these alternatives cannot be as fully interpreted as the AUC, in part because they ignore some information about actual negatives. METHODS: We derive and propose a new concordant partial AUC and a new partial c statistic for ROC data-as foundational measures and methods to help understand and explain parts of the ROC plot and AUC. Our partial measures are continuous and discrete versions of the same measure, are derived from the AUC and c statistic respectively, are validated as equal to each other, and validated as equal in summation to whole measures where expected. Our partial measures are tested for validity on a classic ROC example from Fawcett, a variation thereof, and two real-life benchmark data sets in breast cancer: the Wisconsin and Ljubljana data sets. Interpretation of an example is then provided. RESULTS: Results show the expected equalities between our new partial measures and the existing whole measures. The example interpretation illustrates the need for our newly derived partial measures. CONCLUSIONS: The concordant partial area under the ROC curve was proposed and unlike previous partial measure alternatives, it maintains the characteristics of the AUC. The first partial c statistic for ROC plots was also proposed as an unbiased interpretation for part of an ROC curve. The expected equalities among and between our newly derived partial measures and their existing full measure counterparts are confirmed. These measures may be used with any data set but this paper focuses on imbalanced data with low prevalence. FUTURE WORK: Future work with our proposed measures may: demonstrate their value for imbalanced data with high prevalence, compare them to other measures not based on areas; and combine them with other ROC measures and techniques.

Texture Classification in Extreme Scale Variations Using GANet.

Research in texture recognition often concentrates on recognizing textures with intraclass variations, such as illumination, rotation, viewpoint, and small-scale changes. In contrast, in real-world applications, a change in scale can have a dramatic impact on texture appearance to the point of changing completely from one texture category to another. As a result, texture variations due to changes in scale are among the hardest to handle. In this paper, we conduct the first study of classifying textures with extreme variations in scale. To address this issue, we first propose and then reduce scale proposals on the basis of dominant texture patterns. Motivated by the challenges posed by this problem, we propose a new GANet network where we use a genetic algorithm to change the filters in the hidden layers during network training in order to promote the learning of more informative semantic texture patterns. Finally, we adopt a Fisher vector pooling of a convolutional neural network filter bank feature encoder for global texture representation. Because extreme scale variations are not necessarily present in most standard texture databases, to support the proposed extreme-scale aspects of texture understanding, we are developing a new dataset, the extreme scale variation textures (ESVaT), to test the performance of our framework. It is demonstrated that the proposed framework significantly outperforms the gold-standard texture features by more than 10% on ESVaT. We also test the performance of our proposed approach on the KTHTIPS2b and OS datasets and a further dataset synthetically derived from Forrest, showing the superior performance compared with the state-of-the-art.

Median Robust Extended Local Binary Pattern for Texture Classification.

Local binary patterns (LBP) are considered among the most computationally efficient high-performance texture features. However, the LBP method is very sensitive to image noise and is unable to capture macrostructure information. To best address these disadvantages, in this paper, we introduce a novel descriptor for texture classification, the median robust extended LBP (MRELBP). Different from the traditional LBP and many LBP variants, MRELBP compares regional image medians rather than raw image intensities. A multiscale LBP type descriptor is computed by efficiently comparing image medians over a novel sampling scheme, which can capture both microstructure and macrostructure texture information. A comprehensive evaluation on benchmark data sets reveals MRELBP's high performance-robust to gray scale variations, rotation changes and noise-but at a low computational cost. MRELBP produces the best classification scores of 99.82%, 99.38%, and 99.77% on three popular Outex test suites. More importantly, MRELBP is shown to be highly robust to image noise, including Gaussian noise, Gaussian blur, salt-and-pepper noise, and random pixel corruption.

Sorted self-similarity for multi-modal image registration.

In medical image analysis, registration of multimodal images has been challenging due to the complex intensity relationship between images. Classical multi-modal registration approaches evaluate the degree of the alignment by measuring the statistical dependency of the intensity values between images to be aligned. Employing statistical similarity measures, such as mutual information, is not promising in those cases with complex and spatially dependent intensity relations. A new similarity measure is proposed based on the assessing the similarity of pixels within an image, based on the idea that similar structures in an image are more probable to undergo similar intensity transformations. The most significant pixel similarity values are considered to transmit the most significant self-similarity information. The proposed method is employed in a framework to register different modalities of real brain scans and the performance of the method is compared to the conventional multi-modal registration approach. Quantitative evaluation of the method demonstrates the better registration accuracy in both rigid and non-rigid deformations.

Homotopic non-local regularized reconstruction from sparse positron emission tomography measurements.

BACKGROUND: Positron emission tomography scanners collect measurements of a patient's in vivo radiotracer distribution. The system detects pairs of gamma rays emitted indirectly by a positron-emitting radionuclide (tracer), which is introduced into the body on a biologically active molecule, and the tomograms must be reconstructed from projections. The reconstruction of tomograms from the acquired PET data is an inverse problem that requires regularization. The use of tightly packed discrete detector rings, although improves signal-to-noise ratio, are often associated with high costs of positron emission tomography systems. Thus a sparse reconstruction, which would be capable of overcoming the noise effect while allowing for a reduced number of detectors, would have a great deal to offer. METHODS: In this study, we introduce and investigate the potential of a homotopic non-local regularization reconstruction framework for effectively reconstructing positron emission tomograms from such sparse measurements. RESULTS: Results obtained using the proposed approach are compared with traditional filtered back-projection as well as expectation maximization reconstruction with total variation regularization. CONCLUSIONS: A new reconstruction method was developed for the purpose of improving the quality of positron emission tomography reconstruction from sparse measurements. We illustrate that promising reconstruction performance can be achieved for the proposed approach even at low sampling fractions, which allows for the use of significantly fewer detectors and have the potential to reduce scanner costs.

Apparent Ultra-High b-Value Diffusion-Weighted Image Reconstruction via Hidden Conditional Random Fields.

A promising, recently explored, alternative to ultra-high b-value diffusion weighted imaging (UHB-DWI) is apparent ultra-high b-value diffusion-weighted image reconstruction (AUHB-DWR), where a computational model is used to assist in the reconstruction of apparent DW images at ultra-high b -values. Firstly, we present a novel approach to AUHB-DWR that aims to improve image quality. We formulate the reconstruction of an apparent DW image as a hidden conditional random field (HCRF) in which tissue model diffusion parameters act as hidden states in this random field. The second contribution of this paper is a new generation of fully connected conditional random fields, called the hidden stochastically fully connected conditional random fields (HSFCRF) that allows for efficient inference with significantly reduced computational complexity via stochastic clique structures. The proposed AUHB-DWR algorithms, HCRF and HSFCRF, are evaluated quantitatively in nine different patient cases using Fisher's criteria, probability of error, and coefficient of variation metrics to validate its effectiveness for the purpose of improving intensity delineation between expert identified suspected cancerous and healthy tissue within the prostate gland. The proposed methods are also examined using a prostate phantom, where the apparent ultra-high b-value DW images reconstructed using the tested AUHB-DWR methods are compared with real captured UHB-DWI. The results illustrate that the proposed AUHB-DWR methods has improved reconstruction quality and improved intensity delineation compared with existing AUHB-DWR approaches.

Noise-compensated homotopic non-local regularized reconstruction for rapid retinal optical coherence tomography image acquisitions.

BACKGROUND: Optical coherence tomography (OCT) is a minimally invasive imaging technique, which utilizes the spatial and temporal coherence properties of optical waves backscattered from biological material. Recent advances in tunable lasers and infrared camera technologies have enabled an increase in the OCT imaging speed by a factor of more than 100, which is important for retinal imaging where we wish to study fast physiological processes in the biological tissue. However, the high scanning rate causes proportional decrease of the detector exposure time, resulting in a reduction of the system signal-to-noise ratio (SNR). One approach to improving the image quality of OCT tomograms acquired at high speed is to compensate for the noise component in the images without compromising the sharpness of the image details. METHODS: In this study, we propose a novel reconstruction method for rapid OCT image acquisitions, based on a noise-compensated homotopic modified James-Stein non-local regularized optimization strategy. The performance of the algorithm was tested on a series of high resolution OCT images of the human retina acquired at different imaging rates. RESULTS: Quantitative analysis was used to evaluate the performance of the algorithm using two state-of-art denoising strategies. Results demonstrate significant SNR improvements when using our proposed approach when compared to other approaches. CONCLUSIONS: A new reconstruction method based on a noise-compensated homotopic modified James-Stein non-local regularized optimization strategy was developed for the purpose of improving the quality of rapid OCT image acquisitions. Preliminary results show the proposed method shows considerable promise as a tool to improve the visualization and analysis of biological material using OCT.

BRINT: binary rotation invariant and noise tolerant texture classification.

In this paper, we propose a simple, efficient, yet robust multiresolution approach to texture classification-binary rotation invariant and noise tolerant (BRINT). The proposed approach is very fast to build, very compact while remaining robust to illumination variations, rotation changes, and noise. We develop a novel and simple strategy to compute a local binary descriptor based on the conventional local binary pattern (LBP) approach, preserving the advantageous characteristics of uniform LBP. Points are sampled in a circular neighborhood, but keeping the number of bins in a single-scale LBP histogram constant and small, such that arbitrarily large circular neighborhoods can be sampled and compactly encoded over a number of scales. There is no necessity to learn a texton dictionary, as in methods based on clustering, and no tuning of parameters is required to deal with different data sets. Extensive experimental results on representative texture databases show that the proposed BRINT not only demonstrates superior performance to a number of recent state-of-the-art LBP variants under normal conditions, but also performs significantly and consistently better in presence of noise due to its high distinctiveness and robustness. This noise robustness characteristic of the proposed BRINT is evaluated quantitatively with different artificially generated types and levels of noise (including Gaussian, salt and pepper, and speckle noise) in natural texture images.

Multi-modal image registration using structural features.

Multi-modal image registration has been a challenging task in medical images because of the complex intensity relationship between images to be aligned. Registration methods often rely on the statistical intensity relationship between the images which suffers from problems such as statistical insufficiency. The proposed registration method works based on extracting structural features by utilizing the complex phase and gradient-based information. By employing structural relationships between different modalities instead of complex similarity measures, the multi-modal registration problem is converted into a mono-modal one. Therefore, conventional mono-modal similarity measures can be utilized to evaluate the registration results. This new registration paradigm has been tested on magnetic resonance (MR) brain images of different modes. The method has been evaluated based on target registration error (TRE) to determine alignment accuracy. Quantitative results demonstrate that the proposed method is capable of achieving comparable registration accuracy compared to the conventional mutual information.

A new Mercer sigmoid kernel for clinical data classification.

In classification with Support Vector Machines, only Mercer kernels, i.e. valid kernels, such as the Gaussian RBF kernel, are widely accepted and thus suitable for clinical data. Practitioners would also like to use the sigmoid kernel, a non-Mercer kernel, but its range of validity is difficult to determine, and even within range its validity is in dispute. Despite these shortcomings the sigmoid kernel is used by some, and two kernels in the literature attempt to emulate and improve upon it. We propose the first Mercer sigmoid kernel, that is therefore trustworthy for the classification of clinical data. We show the similarity between the Mercer sigmoid kernel and the sigmoid kernel and, in the process, identify a normalization technique that improves the classification accuracy of the latter. The Mercer sigmoid kernel achieves the best mean accuracy on three clinical data sets, detecting melanoma in skin lesions better than the most popular kernels; while with non-clinical data sets it has no significant difference in median accuracy as compared with the Gaussian RBF kernel. It consistently classifies some points correctly that the Gaussian RBF kernel does not and vice versa.