Hepatocellular Carcinoma Recognition from Ultrasound Images Using Combinations of Conventional and Deep Learning Techniques.

Hepatocellular Carcinoma (HCC) is the most frequent malignant liver tumor and the third cause of cancer-related deaths worldwide. For many years, the golden standard for HCC diagnosis has been the needle biopsy, which is invasive and carries risks. Computerized methods are due to achieve a noninvasive, accurate HCC detection process based on medical images. We developed image analysis and recognition methods to perform automatic and computer-aided diagnosis of HCC. Conventional approaches that combined advanced texture analysis, mainly based on Generalized Co-occurrence Matrices (GCM) with traditional classifiers, as well as deep learning approaches based on Convolutional Neural Networks (CNN) and Stacked Denoising Autoencoders (SAE), were involved in our research. The best accuracy of 91% was achieved for B-mode ultrasound images through CNN by our research group. In this work, we combined the classical approaches with CNN techniques, within B-mode ultrasound images. The combination was performed at the classifier level. The CNN features obtained at the output of various convolution layers were combined with powerful textural features, then supervised classifiers were employed. The experiments were conducted on two datasets, acquired with different ultrasound machines. The best performance, above 98%, overpassed our previous results, as well as representative state-of-the-art results.

Enhanced Perception for Autonomous Driving Using Semantic and Geometric Data Fusion.

Environment perception remains one of the key tasks in autonomous driving for which solutions have yet to reach maturity. Multi-modal approaches benefit from the complementary physical properties specific to each sensor technology used, boosting overall performance. The added complexity brought on by data fusion processes is not trivial to solve, with design decisions heavily influencing the balance between quality and latency of the results. In this paper we present our novel real-time, 360∘ enhanced perception component based on low-level fusion between geometry provided by the LiDAR-based 3D point clouds and semantic scene information obtained from multiple RGB cameras, of multiple types. This multi-modal, multi-sensor scheme enables better range coverage, improved detection and classification quality with increased robustness. Semantic, instance and panoptic segmentations of 2D data are computed using efficient deep-learning-based algorithms, while 3D point clouds are segmented using a fast, traditional voxel-based solution. Finally, the fusion obtained through point-to-image projection yields a semantically enhanced 3D point cloud that allows enhanced perception through 3D detection refinement and 3D object classification. The planning and control systems of the vehicle receives the individual sensors' perception together with the enhanced one, as well as the semantically enhanced 3D points. The developed perception solutions are successfully integrated onto an autonomous vehicle software stack, as part of the UP-Drive project.

Fast Panoptic Segmentation with Soft Attention Embeddings.

Panoptic segmentation provides a rich 2D environment representation by unifying semantic and instance segmentation. Most current state-of-the-art panoptic segmentation methods are built upon two-stage detectors and are not suitable for real-time applications, such as automated driving, due to their high computational complexity. In this work, we introduce a novel, fast and accurate single-stage panoptic segmentation network that employs a shared feature extraction backbone and three network heads for object detection, semantic segmentation, instance-level attention masks. Guided by object detections, our new panoptic segmentation head learns instance specific soft attention masks based on spatial embeddings. The semantic masks for stuff classes and soft instance masks for things classes are pixel-wise coherent and can be easily integrated in a panoptic output. The training and inference pipelines are simplified and no post-processing of the panoptic output is necessary. Benefiting from fast inference speed, the network can be deployed in automated vehicles or robotic applications. We perform extensive experiments on COCO and Cityscapes datasets and obtain competitive results in both accuracy and time. On the Cityscapes dataset we achieve 59.7 panoptic quality with an inference speed of more than 10 FPS on high resolution 1024 × 2048 images.

Robust Data Association Using Fusion of Data-Driven and Engineered Features for Real-Time Pedestrian Tracking in Thermal Images.

Object tracking is an essential problem in computer vision that has been extensively researched for decades. Tracking objects in thermal images is particularly difficult because of the lack of color information, low image resolution, or high similarity between objects of the same class. One of the main challenges in multi-object tracking, also referred to as the data association problem, is finding the correct correspondences between measurements and tracks and adapting the object appearance changes over time. We addressed this challenge of data association for thermal images by proposing three contributions. The first contribution consisted of the creation of a data-driven appearance score using five Siamese Networks, which operate on the image detection and on parts of it. Secondly, we engineered an original edge-based descriptor that improves the data association process. Lastly, we proposed a dataset consisting of pedestrian instances that were recorded in different scenarios and are used for training the Siamese Networks. The data-driven part of the data association score offers robustness, while feature engineering offers adaptability to unknown scenarios and their combination leads to a more powerful tracking solution. Our approach had a running time of 25 ms and achieved an average precision of 86.2% on publicly available benchmarks, containing real-world scenarios, as shown in the evaluation section.

Hepatocellular Carcinoma Automatic Diagnosis within CEUS and B-Mode Ultrasound Images Using Advanced Machine Learning Methods.

Hepatocellular Carcinoma (HCC) is the most common malignant liver tumor, being present in 70% of liver cancer cases. It usually evolves on the top of the cirrhotic parenchyma. The most reliable method for HCC diagnosis is the needle biopsy, which is an invasive, dangerous method. In our research, specific techniques for non-invasive, computerized HCC diagnosis are developed, by exploiting the information from ultrasound images. In this work, the possibility of performing the automatic diagnosis of HCC within B-mode ultrasound and Contrast-Enhanced Ultrasound (CEUS) images, using advanced machine learning methods based on Convolutional Neural Networks (CNN), was assessed. The recognition performance was evaluated separately on B-mode ultrasound images and on CEUS images, respectively, as well as on combined B-mode ultrasound and CEUS images. For this purpose, we considered the possibility of combining the input images directly, performing feature level fusion, then providing the resulted data at the entrances of representative CNN classifiers. In addition, several multimodal combined classifiers were experimented, resulted by the fusion, at classifier, respectively, at the decision levels of two different branches based on the same CNN architecture, as well as on different CNN architectures. Various combination methods, and also the dimensionality reduction method of Kernel Principal Component Analysis (KPCA), were involved in this process. These results were compared with those obtained on the same dataset, when employing advanced texture analysis techniques in conjunction with conventional classification methods and also with equivalent state-of-the-art approaches. An accuracy above 97% was achieved when our new methodology was applied.

Integration of Real-Time Image Fusion in the Robotic-Assisted Treatment of Hepatocellular Carcinoma.

Hepatocellular carcinoma (HCC) is one of the leading causes of cancer-related deaths worldwide, with its mortality rate correlated with the tumor staging; i.e., early detection and treatment are important factors for the survival rate of patients. This paper presents the development of a novel visualization and detection system for HCC, which is a composing module of a robotic system for the targeted treatment of HCC. The system has two modules, one for the tumor visualization that uses image fusion (IF) between computerized tomography (CT) obtained preoperatively and real-time ultrasound (US), and the second module for HCC automatic detection from CT images. Convolutional neural networks (CNN) are used for the tumor segmentation which were trained using 152 contrast-enhanced CT images. Probabilistic maps are shown as well as 3D representation of HCC within the liver tissue. The development of the visualization and detection system represents a milestone in testing the feasibility of a novel robotic system in the targeted treatment of HCC. Further optimizations are planned for the tumor visualization and detection system with the aim of introducing more relevant functions and increase its accuracy.

Comparison of Deep-Learning and Conventional Machine-Learning Methods for the Automatic Recognition of the Hepatocellular Carcinoma Areas from Ultrasound Images.

The emergence of deep-learning methods in different computer vision tasks has proved to offer increased detection, recognition or segmentation accuracy when large annotated image datasets are available. In the case of medical image processing and computer-aided diagnosis within ultrasound images, where the amount of available annotated data is smaller, a natural question arises: are deep-learning methods better than conventional machine-learning methods? How do the conventional machine-learning methods behave in comparison with deep-learning methods on the same dataset? Based on the study of various deep-learning architectures, a lightweight multi-resolution Convolutional Neural Network (CNN) architecture is proposed. It is suitable for differentiating, within ultrasound images, between the Hepatocellular Carcinoma (HCC), respectively the cirrhotic parenchyma (PAR) on which HCC had evolved. The proposed deep-learning model is compared with other CNN architectures that have been adapted by transfer learning for the ultrasound binary classification task, but also with conventional machine-learning (ML) solutions trained on textural features. The achieved results show that the deep-learning approach overcomes classical machine-learning solutions, by providing a higher classification performance.

Stabilization and Validation of 3D Object Position Using Multimodal Sensor Fusion and Semantic Segmentation.

The stabilization and validation process of the measured position of objects is an important step for high-level perception functions and for the correct processing of sensory data. The goal of this process is to detect and handle inconsistencies between different sensor measurements, which result from the perception system. The aggregation of the detections from different sensors consists in the combination of the sensorial data in one common reference frame for each identified object, leading to the creation of a super-sensor. The result of the data aggregation may end up with errors such as false detections, misplaced object cuboids or an incorrect number of objects in the scene. The stabilization and validation process is focused on mitigating these problems. The current paper proposes four contributions for solving the stabilization and validation task, for autonomous vehicles, using the following sensors: trifocal camera, fisheye camera, long-range RADAR (Radio detection and ranging), and 4-layer and 16-layer LIDARs (Light Detection and Ranging). We propose two original data association methods used in the sensor fusion and tracking processes. The first data association algorithm is created for tracking LIDAR objects and combines multiple appearance and motion features in order to exploit the available information for road objects. The second novel data association algorithm is designed for trifocal camera objects and has the objective of finding measurement correspondences to sensor fused objects such that the super-sensor data are enriched by adding the semantic class information. The implemented trifocal object association solution uses a novel polar association scheme combined with a decision tree to find the best hypothesis-measurement correlations. Another contribution we propose for stabilizing object position and unpredictable behavior of road objects, provided by multiple types of complementary sensors, is the use of a fusion approach based on the Unscented Kalman Filter and a single-layer perceptron. The last novel contribution is related to the validation of the 3D object position, which is solved using a fuzzy logic technique combined with a semantic segmentation image. The proposed algorithms have a real-time performance, achieving a cumulative running time of 90 ms, and have been evaluated using ground truth data extracted from a high-precision GPS (global positioning system) with 2 cm accuracy, obtaining an average error of 0.8 m.

Systems of nonlinear algebraic equations with positive solutions.

We are concerned with the positive solutions of an algebraic system depending on a parameter [Formula: see text] and arising in economics. For [Formula: see text] we prove that the system has at least a solution. For [Formula: see text] we give three proofs of the existence and a proof of the uniqueness of the solution. Brouwer's theorem and inequalities involving convex functions are essential tools in our proofs.

Detecting Curvilinear Features Using Structure Tensors.

Few published articles on curvilinear structures exist compared with works on detecting lines or corners with high accuracy. In medical ultrasound imaging, the structures that need to be detected appear as a collection of microstructures correlated along a path. In this paper, we investigated techniques that extract meaningful low-level information for curvilinear structures, using techniques based on structure tensor. We proposed a novel structure tensor enhancement inspired by bilateral filtering. We compared the proposed approach with five state-of-the-art curvilinear structure detectors. We tested the algorithms against simulated images with known ground truth and real images from three different domains (medical ultrasound, scanning electron microscope, and astronomy). For the real images, we employed experts to delineate the ground truth for each domain. Techniques borrowed from machine learning robustly assessed the performance of the methods (area under curve and cross validation). As a practical application, we used the proposed method to label a set of 5000 ultrasound images. We conclude that the proposed tensor-based approach outperforms the state-of-the-art methods in providing magnitude and orientation information for curvilinear structures. The evaluation methodology ensures that the employed feature-detection method will yield reproducible performance on new, unseen images. We published all the implemented methods as open-source software.

Semi-supervised segmentation of ultrasound images based on patch representation and continuous min cut.

Ultrasound segmentation is a challenging problem due to the inherent speckle and some artifacts like shadows, attenuation and signal dropout. Existing methods need to include strong priors like shape priors or analytical intensity models to succeed in the segmentation. However, such priors tend to limit these methods to a specific target or imaging settings, and they are not always applicable to pathological cases. This work introduces a semi-supervised segmentation framework for ultrasound imaging that alleviates the limitation of fully automatic segmentation, that is, it is applicable to any kind of target and imaging settings. Our methodology uses a graph of image patches to represent the ultrasound image and user-assisted initialization with labels, which acts as soft priors. The segmentation problem is formulated as a continuous minimum cut problem and solved with an efficient optimization algorithm. We validate our segmentation framework on clinical ultrasound imaging (prostate, fetus, and tumors of the liver and eye). We obtain high similarity agreement with the ground truth provided by medical expert delineations in all applications (94% DICE values in average) and the proposed algorithm performs favorably with the literature.

Label transfer by measuring compactness.

This paper presents a new automatic image annotation algorithm. First, we introduce a new similarity measure between images: compactness. This uses low level visual descriptors for determining the similarity between two images. Compactness shows how close test image features lie to training image feature cluster centers. The measure provides the core for a k-nearest neighbor type image annotation method. Afterward, a formalism for defining different transfer techniques is devised and several label transfer techniques are provided. The method as whole is evaluated on four image annotation benchmarks. The results on these sets validate the accuracy of the approach, which outperforms many state-of-the-art annotation methods. The method presented here requires a simple training process, efficiently combines different feature types and performs better than complex learning algorithms, even in this incipient form. The main contributions of this paper are the usage of compactness as a similarity measure that enables efficient low level feature comparison and an annotation algorithm based on label transfer.

Motion estimation using the correlation transform.

The zero-mean normalized cross-correlation is shown to improve the accuracy of optical flow, but its analytical form is quite complicated for the variational framework. This paper addresses this issue and presents a new direct approach to this matching measure. Our approach uses the correlation transform to define very discriminative descriptors that are precomputed and that have to be matched in the target frame. It is equivalent to the computation of the optical flow for the correlation transforms of the images. The smoothness energy is non-local and uses a robust penalty in order to preserve motion discontinuities. The model is associated with a fast and parallelizable minimization procedure based on the projected-proximal point algorithm. The experiments confirm the strength of this model and implicitly demonstrate the correctness of our solution. The results demonstrate that the involved data term is very robust with respect to changes in illumination, especially where large illumination exists.

Iterative methods for obtaining energy-minimizing parametric snakes with applications to medical imaging.

After a brief survey on the parametric deformable models, we develop an iterative method based on the finite difference schemes in order to obtain energy-minimizing snakes. We estimate the approximation error, the residue, and the truncature error related to the corresponding algorithm, then we discuss its convergence, consistency, and stability. Some aspects regarding the prosthetic sugical methods that implement the above numerical methods are also pointed out.

Abdominal tumor characterization and recognition using superior-order cooccurrence matrices, based on ultrasound images.

The noninvasive diagnosis of the malignant tumors is an important issue in research nowadays. Our purpose is to elaborate computerized, texture-based methods for performing computer-aided characterization and automatic diagnosis of these tumors, using only the information from ultrasound images. In this paper, we considered some of the most frequent abdominal malignant tumors: the hepatocellular carcinoma and the colonic tumors. We compared these structures with the benign tumors and with other visually similar diseases. Besides the textural features that proved in our previous research to be useful in the characterization and recognition of the malignant tumors, we improved our method by using the grey level cooccurrence matrix and the edge orientation cooccurrence matrix of superior order. As resulted from our experiments, the new textural features increased the malignant tumor classification performance, also revealing visual and physical properties of these structures that emphasized the complex, chaotic structure of the corresponding tissue.

Influence of expert-dependent variability over the performance of noninvasive fibrosis assessment in patients with chronic hepatitis C by means of texture analysis.

Texture analysis is viewed as a method to enhance the diagnosis power of classical B-mode ultrasound image. The present paper aims to evaluate and eliminate the dependence between the human expert and the performance of such a texture analysis system in predicting the cirrhosis in chronic hepatitis C patients. 125 consecutive chronic hepatitis C patients were included in this study. Ultrasound images were acquired from each patient and four human experts established regions of interest. Textural analysis tool was evaluated. The performance of this approach depends highly on the human expert that establishes the regions of interest (P < 0.05). The novel algorithm that automatically establishes regions of interest can be compared with a trained radiologist. In classical form met in the literature, the noninvasive diagnosis through texture analysis has limited utility in clinical practice. The automatic ROI establishment tool is very useful in eliminating the expert-dependent variability.

Design of interpolation functions for subpixel-accuracy stereo-vision systems.

Traditionally, subpixel interpolation in stereo-vision systems was designed for the block-matching algorithm. During the evaluation of different interpolation strategies, a strong correlation was observed between the type of the stereo algorithm and the subpixel accuracy of the different solutions. Subpixel interpolation should be adapted to each stereo algorithm to achieve maximum accuracy. In consequence, it is more important to propose methodologies for interpolation function generation than specific function shapes. We propose two such methodologies based on data generated by the stereo algorithms. The first proposal uses a histogram to model the environment and applies histogram equalization to an existing solution adapting it to the data. The second proposal employs synthetic images of a known environment and applies function fitting to the resulted data. The resulting function matches the algorithm and the data as best as possible. An extensive evaluation set is used to validate the findings. Both real and synthetic test cases were employed in different scenarios. The test results are consistent and show significant improvements compared with traditional solutions.

Usefulness of textural analysis as a tool for noninvasive liver fibrosis staging.

PURPOSE: Noninvasive diagnosis of liver fibrosis is a popular topic in the medical literature. Textural analysis on B-mode ultrasound is viewed as a noninvasive tool for fibrosis staging. A liver tissue model is proposed and used to simulate ultrasound images. METHODS: One hundred and twenty-five patients with chronic hepatitis C were included in this study. Patients were investigated using B-mode ultrasound and liver biopsy (Metavir scoring). A texture analysis tool consisting of 12 algorithms and a logistic regression classifier was implemented and validated. Tissue model parameters were varied and ultrasound images were generated. RESULTS: Texture analysis can discriminate between stages F0 and F4 using actual patient data (accuracy 69.5%) and synthetic images (accuracy 76.6%). A human expert is less sensitive than texture analysis in discriminating subtle changes in ultrasound images. High fibrosis detection accuracies are correlated with larger differences in portal space density (r (2) = 0.5). Accuracies measured when we varied only the fibrosis stage and kept the rest of the tissue parameters constant showed high detection rates only in a narrow parameter interval. CONCLUSION: The texture analysis system shows limited performance in staging fibrosis and it cannot be used for accurate monitoring of fibrosis evolution over time.

Ultrasonography contribution to the detection and characterization of hepatic restructuring: is the "virtual biopsy" taken into consideration?

Generally, the evolution of diffuse liver diseases is variable but quite long. Even the severe types of chronic hepatitis have a slow progression which implies decades, often over 20-30 years. Cirrhosis is the principal long time complication of chronic hepatopathies. It represents a major risk factor for the development of hepatocellular carcinoma. Ultrasonography plays an important role among the methods used for detecting diffuse liver diseases, for placing them and identifying supplementary risk factors for carcinogenesis and of hepatocellular carcinoma itself. The two- and especially the three-dimensional exploration allow the characterization of hepatic texture and the identification of certain changes which may suggest hepatic restructuring.