Towards dental diagnostic systems: Synergizing wavelet transform with generative adversarial networks for enhanced image data fusion.

The advent of precision diagnostics in pediatric dentistry is shifting towards ensuring early detection of dental diseases, a critical factor in safeguarding the oral health of the younger population. In this study, an innovative approach is introduced, wherein Discrete Wavelet Transform (DWT) and Generative Adversarial Networks (GANs) are synergized within an Image Data Fusion (IDF) framework to enhance the accuracy of dental disease diagnosis through dental diagnostic systems. Dental panoramic radiographs from pediatric patients were utilized to demonstrate how the integration of DWT and GANs can significantly improve the informativeness of dental images. In the IDF process, the original images, GAN-augmented images, and wavelet-transformed images are combined to create a comprehensive dataset. DWT was employed for the decomposition of images into frequency components to enhance the visibility of subtle pathological features. Simultaneously, GANs were used to augment the dataset with high-quality, synthetic radiographic images indistinguishable from real ones, to provide robust data training. These integrated images are then fed into an Artificial Neural Network (ANN) for the classification of dental diseases. The utilization of the ANN in this context demonstrates the system's robustness and culminates in achieving an unprecedented accuracy rate of 0.897, 0.905 precision, recall of 0.897, and specificity of 0.968. Additionally, this study explores the feasibility of embedding the diagnostic system into dental X-ray scanners by leveraging lightweight models and cloud-based solutions to minimize resource constraints. Such integration is posited to revolutionize dental care by providing real-time, accurate disease detection capabilities, which significantly reduces diagnostical delays and enhances treatment outcomes.

Feasibility study on discrimination of Polygonatum kingianum origins by NIR and MIR spectra data.

Most existing studies have focused on identifying the origin of species with protected geographical indications while neglecting to determine the proximate geographical origin of different species. In this study, we investigated the feasibility of using near- and mid-infrared spectroscopy to identify the origin of 156 Polygonatum kingianum samples from six regions in Yunnan, China. In this work, spectral images of different modes reveal more information about the P. kingianum. Comparing the performance of traditional machine learning models according to single spectrum and data fusion, the middle-level data fusion-principal component model has the best performance, and its sensitivity, specificity, and accuracy are all 1, and the model has the least number of variables. The residual convolutional neural network (ResNet) model constructed in the 1050-850 cm-1 band confirms that fewer variables are beneficial in improving the accuracy of the model. In conclusion, this study verifies the feasibility of the proposed strategy and establishes a practical model to determine the source of P. kingianum.

Comparison of deep-learning multimodality data fusion strategies in mandibular osteoradionecrosis NTCP modelling using clinical variables and radiation dose distribution volumes.

OBJECTIVE: Normal tissue complication probability (NTCP) modelling is rapidly embracing deep learning (DL) methods, acknowledging the importance of spatial dose information. Finding effective ways to combine information from radiation dose distribution maps (dosiomics) and clinical data involves technical challenges and requires domain knowledge. We propose different multi-modality data fusion strategies to facilitate future DL-based NTCP studies. Approach. Early, joint and late DL multi-modality fusion strategies were compared using clinical and mandibular radiation dose distribution volumes. These were contrasted with single-modality models: a random forest trained on non-image data (clinical, demographic and dose-volume metrics) and a 3D DenseNet-40 trained on image data (mandibular dose distribution maps). The study involved a matched cohort of 92 ORN cases and 92 controls from a single institution. Main results. The late fusion model exhibited superior discrimination and calibration performance, while the join fusion achieved a more balanced distribution of the predicted probabilities. Discrimination performance did not significantly differ between strategies. Late fusion, though less technically complex, lacks crucial inter-modality interactions for NTCP modelling. In contrast, joint fusion, despite its complexity, resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation. Significance. This study is a pioneering effort in comparing different strategies for including image data into DL-based NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. Multiple data fusion strategies should be compared and reported in multi-modality NTCP modelling using DL. .

Point Cloud Densification Algorithm for Multiple Cameras and Lidars Data Fusion.

Fusing data from many sources helps to achieve improved analysis and results. In this work, we present a new algorithm to fuse data from multiple cameras with data from multiple lidars. This algorithm was developed to increase the sensitivity and specificity of autonomous vehicle perception systems, where the most accurate sensors measuring the vehicle's surroundings are cameras and lidar devices. Perception systems based on data from one type of sensor do not use complete information and have lower quality. The camera provides two-dimensional images; lidar produces three-dimensional point clouds. We developed a method for matching pixels on a pair of stereoscopic images using dynamic programming inspired by an algorithm to match sequences of amino acids used in bioinformatics. We improve the quality of the basic algorithm using additional data from edge detectors. Furthermore, we also improve the algorithm performance by reducing the size of matched pixels determined by available car speeds. We perform point cloud densification in the final step of our method, fusing lidar output data with stereo vision output. We implemented our algorithm in C++ with Python API, and we provided the open-source library named Stereo PCD. This library very efficiently fuses data from multiple cameras and multiple lidars. In the article, we present the results of our approach to benchmark databases in terms of quality and performance. We compare our algorithm with other popular methods.

A Fusion Tracking Algorithm for Electro-Optical Theodolite Based on the Three-State Transition Model.

This study presents a novel approach to address the autonomous stable tracking issue in electro-optical theodolite operating in closed-loop mode. The proposed methodology includes a multi-sensor adaptive weighted fusion algorithm and a fusion tracking algorithm based on a three-state transition model. A refined recursive formula for error covariance estimation is developed by integrating attenuation factors and least squares extrapolation. This formula is employed to formulate a multi-sensor weighted fusion algorithm that utilizes error covariance estimation. By assigning weighted coefficients to calculate the residual of the newly introduced error term and defining the sensor's unique states based on these coefficients, a fusion tracking algorithm grounded on the three-state transition model is introduced. In cases of interference or sensor failure, the algorithm either computes the weighted fusion value of the multi-sensor measurement or triggers autonomous sensor switching to ensure the autonomous and stable measurement of the theodolite. Experimental results indicate that when a specific sensor is affected by interference or the off-target amount cannot be extracted, the algorithm can swiftly switch to an alternative sensor. This capability facilitates the precise and consistent generation of data, thereby ensuring the stable operation of the tracking system. Furthermore, the algorithm demonstrates robustness across various measurement scenarios.

Lycopene detection in cherry tomatoes with feature enhancement and data fusion.

Lycopene, a biologically active phytochemical with health benefits, is a key quality indicator for cherry tomatoes. While ultraviolet/visible/near-infrared (UV/Vis/NIR) spectroscopy holds promise for large-scale online lycopene detection, capturing its characteristic signals is challenging due to the low lycopene concentration in cherry tomatoes. This study improved the prediction accuracy of lycopene by supplementing spectral data with image information through spectral feature enhancement and spectra-image fusion. The feasibility of using UV/Vis/NIR spectra and image features to predict lycopene content was validated. By enhancing spectral bands corresponding to colors correlated with lycopene, the performance of the spectral model was improved. Additionally, direct spectra-image fusion further enhanced the prediction accuracy, achieving RP2, RMSEP, and RPD as 0.95, 8.96 mg/kg, and 4.25, respectively. Overall, this research offers valuable insights into supplementing spectral data with image information to improve the accuracy of non-destructive lycopene detection, providing practical implications for online fruit quality prediction.

Vibrational spectroscopy data fusion for enhanced classification of different milk types.

The aim of this study is to classify seven types of Irish milk (butter, fresh, heart active, lactose free, light, protein, and slimline), supplied by a specific company, using vibrational spectroscopy methods: Near infrared (NIR), mid infrared (MIR), and Raman spectroscopy. In this regard, chemometric methods were used, and the impact of spectral data fusion on prediction accuracy was evaluated. A total of 105 samples were tested, with 21 used in the test set. The study assessed principal component analysis (PCA), partial least square discriminant analysis (PLS-DA), and sequential and orthogonalized partial least squares linear discriminant analysis (SO-PLS-LDA) for classifying different milk types. The prediction accuracy, when applying PLS-DA on individual blocks of data and low-level fused data, did not exceed 85.71 %. However, implementing the SO-PLS-LDA strategy significantly improved the accuracy to 95 %, suggesting a promising method for the development of classification models for milk using data fusion strategies.

Artificial intelligence decision making tools in food metabolomics: Data fusion unravels synergies within the hazelnut (Corylus avellana L.) metabolome and improves quality prediction.

This study investigates the metabolome of high-quality hazelnuts (Corylus avellana L.) by applying untargeted and targeted metabolome profiling techniques to predict industrial quality. Utilizing comprehensive two-dimensional gas chromatography and liquid chromatography coupled with high-resolution mass spectrometry, the research characterizes the non-volatile (primary and specialized metabolites) and volatile metabolomes. Data fusion techniques, including low-level (LLDF) and mid-level (MLDF), are applied to enhance classification performance. Principal component analysis (PCA) and partial least squares discriminant analysis (PLS-DA) reveal that geographical origin and postharvest practices significantly impact the specialized metabolome, while storage conditions and duration influence the volatilome. The study demonstrates that MLDF approaches, particularly supervised MLDF, outperform single-fraction analyses in predictive accuracy. Key findings include the identification of metabolites patterns causally correlated to hazelnut's quality attributes, of them aldehydes, alcohols, terpenes, and phenolic compounds as most informative. The integration of multiple analytical platforms and data fusion methods shows promise in refining quality assessments and optimizing storage and processing conditions for the food industry.

Authentication of Argentinean extra-virgin olive oils using three-way fluorescence and two-way near-infrared data fused with multi-block DD-SIMCA.

A trending problem of Extra Virgin Olive Oil (EVOO) adulteration is investigated using two analytical platforms, involving: (1) Near Infrared (NIR) spectroscopy, resulting in a two-way data set, and (2) Fluorescence Excitation-Emission Matrix (EEFM) spectroscopy, producing three-way data. The related instruments were employed to study genuine and adulterated samples. Each data set was first separately analyzed using the Data Driven-Soft Independent Modeling of Class Analogies (DD-SIMCA) method, based on Principal Component Analysis (for the two-way NIR data) and PARallel FACtor analysis (for the three-way EEFM data). The data sets were then processed together using the multi-block fusion method, based on the concept of Cumulative Analytical Signal (CAS). A comparison of the data processing methods in terms of sensitivity, specificity and selectivity showed the following order of excellence: NIR < EEFM < NIR + EEFM. This finding confirms the effectiveness of multi-block data fusion, which cumulatively improves the model performance.

iP3T: an interpretable multimodal time-series model for enhanced gait phase prediction in wearable exoskeletons.

Introduction: Wearable exoskeletons assist individuals with mobility impairments, enhancing their gait and quality of life. This study presents the iP3T model, designed to optimize gait phase prediction through the fusion of multimodal time-series data. Methods: The iP3T model integrates data from stretch sensors, inertial measurement units (IMUs), and surface electromyography (sEMG) to capture comprehensive biomechanical and neuromuscular signals. The model's architecture leverages transformer-based attention mechanisms to prioritize crucial data points. A series of experiments were conducted on a treadmill with five participants to validate the model's performance. Results: The iP3T model consistently outperformed traditional single-modality approaches. In the post-stance phase, the model achieved an RMSE of 1.073 and an R2 of 0.985. The integration of multimodal data enhanced prediction accuracy and reduced metabolic cost during assisted treadmill walking. Discussion: The study highlights the critical role of each sensor type in providing a holistic understanding of the gait cycle. The attention mechanisms within the iP3T model contribute to its interpretability, allowing for effective optimization of sensor configurations and ultimately improving mobility and quality of life for individuals with gait impairments.

Bridged Treatment Comparisons: an Illustrative Application in HIV Treatment.

Comparisons of treatments, interventions, or exposures are of central interest in epidemiology, but direct comparisons are not always possible due to practical or ethical reasons. Here, we detail a fusion approach to compare treatments across studies. The motivating example entails comparing the risk of the composite outcome of death, AIDS, or greater than a 50% CD4 cell count decline in people with HIV when assigned triple versus mono antiretroviral therapy, using data from the AIDS Clinical Trial Group (ACTG) 175 (mono versus dual therapy) and ACTG 320 (dual versus triple therapy). We review a set of identification assumptions and estimate the risk difference using an inverse probability weighting estimator that leverages the shared trial arms (dual therapy). A fusion diagnostic based on comparing the shared arms is proposed that may indicate violation of the identification assumptions. Application of the data fusion estimator and diagnostic to the ACTG trials indicates triple therapy results in a reduction in risk compared to monotherapy in individuals with baseline CD4 counts between 50 and 300 cells/mm3. Bridged treatment comparisons address questions that none of the constituent data sources could address alone, but valid fusion-based inference requires careful consideration of the underlying assumptions.

Regulating Modality Utilization within Multimodal Fusion Networks.

Multimodal fusion networks play a pivotal role in leveraging diverse sources of information for enhanced machine learning applications in aerial imagery. However, current approaches often suffer from a bias towards certain modalities, diminishing the potential benefits of multimodal data. This paper addresses this issue by proposing a novel modality utilization-based training method for multimodal fusion networks. The method aims to guide the network's utilization on its input modalities, ensuring a balanced integration of complementary information streams, effectively mitigating the overutilization of dominant modalities. The method is validated on multimodal aerial imagery classification and image segmentation tasks, effectively maintaining modality utilization within ±10% of the user-defined target utilization and demonstrating the versatility and efficacy of the proposed method across various applications. Furthermore, the study explores the robustness of the fusion networks against noise in input modalities, a crucial aspect in real-world scenarios. The method showcases better noise robustness by maintaining performance amidst environmental changes affecting different aerial imagery sensing modalities. The network trained with 75.0% EO utilization achieves significantly better accuracy (81.4%) in noisy conditions (noise variance = 0.12) compared to traditional training methods with 99.59% EO utilization (73.7%). Additionally, it maintains an average accuracy of 85.0% across different noise levels, outperforming the traditional method's average accuracy of 81.9%. Overall, the proposed approach presents a significant step towards harnessing the full potential of multimodal data fusion in diverse machine learning applications such as robotics, healthcare, satellite imagery, and defense applications.

Multimodal Image-Based Indoor Localization with Machine Learning-A Systematic Review.

Outdoor positioning has become a ubiquitous technology, leading to the proliferation of many location-based services such as automotive navigation and asset tracking. Meanwhile, indoor positioning is an emerging technology with many potential applications. Researchers are continuously working towards improving its accuracy, and one general approach to achieve this goal includes using machine learning to combine input data from multiple available sources, such as camera imagery. For this active research area, we conduct a systematic literature review and identify around 40 relevant research papers. We analyze contributions describing indoor positioning methods based on multimodal data, which involves combinations of images with motion sensors, radio interfaces, and LiDARs. The conducted survey allows us to draw conclusions regarding the open research areas and outline the potential future evolution of multimodal indoor positioning.

Calibration Methods for Large-Scale and High-Precision Globalization of Local Point Cloud Data Based on iGPS.

The point cloud is one of the measurement results of local measurement and is widely used because of its high measurement accuracy, high data density, and low environmental impact. However, since point cloud data from a single measurement are generally small in spatial extent, it is necessary to accurately globalize the local point cloud to measure large components. In this paper, the method of using an iGPS (indoor Global Positioning System) as an external measurement device to realize high-accuracy globalization of local point cloud data is proposed. Two calibration models are also discussed for different application scenarios. Verification experiments prove that the average calibration errors of these two calibration models are 0.12 mm and 0.17 mm, respectively. The proposed method can maintain calibration precision in a large spatial range (about 10 m × 10 m × 5 m), which is of high value for engineering applications.

Monitoring of meat quality and change-point detection by a sensor array and profiling of bacterial communities.

BACKGROUND: Real-time monitoring of food consumer quality remains challenging due to diverse bio-chemical processes taking place in the food matrices, and hence it requires accurate analytical methods. Thresholds to determine spoiled food are often difficult to set. The existing analytical methods are too complicated for rapid in situ screening of foodstuff. RESULTS: We have studied the dynamics of meat spoilage by electronic nose (e-nose) for digitizing the smell associated with volatile spoilage markers of meat, comparing the results with changes in the microbiome composition of the spoiling meat samples. We apply the time series analysis to follow dynamic changes in the gas profile extracted from the e-nose responses and to identify the change-point window of the meat state. The obtained e-nose features correlate with changes in the microbiome composition such as increase in the proportion of Brochothrix and Pseudomonas spp. and disappearance of Mycoplasma spp., and with representative gas sensors towards hydrogen, ammonia, and alcohol vapors with R2 values of 0.98, 0.93, and 0.91, respectively. Integration of e-nose and computer vision into a single analytical panel improved the meat state identification accuracy up to 0.85, allowing for more reliable meat state assessment. SIGNIFICANCE: Accurate identification of the change-point in the meat state achieved by digitalizing volatile spoilage markers from the e-nose unit holds promises for application of smart miniaturized devices in food industry.

The development and application of a novel E-commerce recommendation system used in electric power B2B sector.

The advent of the digital era has transformed E-commerce platforms into critical tools for industry, yet traditional recommendation systems often fall short in the specialized context of the electric power industry. These systems typically struggle with the industry's unique challenges, such as infrequent and high-stakes transactions, prolonged decision-making processes, and sparse data. This research has developed a novel recommendation engine tailored to these specific conditions, such as to handle the low frequency and long cycle nature of Business-to-Business (B2B) transactions. This approach includes algorithmic enhancements to better process and interpret the limited data available, and data pre-processing techniques designed to enrich the sparse datasets characteristic of this industry. This research also introduces a methodological innovation that integrates multi-dimensional data, combining user E-commerce activities, product specifics, and essential non-tendering information. The proposed engine employs advanced machine learning techniques to provide more accurate and relevant recommendations. The results demonstrate a marked improvement over traditional models, offering a more robust and effective tool for facilitating B2B transactions in the electric power industry. This research not only addresses the sector's unique challenges but also provides a blueprint for adapting recommendation systems to other industries with similar B2B characteristics.

The Application of GPS-Based Friend/Foe Localization and Identification to Enhance Security in Restricted Areas.

This paper is devoted to the application of object localization and identification with information combined from a radar system and a dedicated portable/mobile electronic device equipped with a global positioning system (GPS) receiver. This device is able to provide object's (staff member, and staff vehicle) rough location and identification. Such systems are required in very restrictive security areas like airports (e.g., open-air area and apron). Currently, the outdoor area of the airport is typically protected by a surveillance system operated by security guards. Surveillance systems are composed of different sensors, video and infrared cameras, and microwave radars. The sheer number of events generated via the system can lead to fatigue among staff, potentially resulting in the omission of critical events. To address this issue, we propose an electronic system equipped with a wireless module and a GPS module. This approach enables automatic identification of objects through the fusion of data from two independent systems (GPS and radar). The radar system is capable of precisely localizing and tracking objects, while the described system is able to identify registered objects. This paper contains a description of the subsystems of a portable/mobile electronic device. The fusion of information from the proposed system (rough location and identification) with the precise location obtained from short-range radar is intended to reduce the number of false alerts in the surveillance system.

The Artificial Neural Twin - Process optimization and continual learning in distributed process chains.

Industrial process optimization and control is crucial to increase economic and ecologic efficiency. However, data sovereignty, differing goals, or the required expert knowledge for implementation impede holistic implementation. Further, the increasing use of data-driven AI-methods in process models and industrial sensory often requires regular fine-tuning to accommodate distribution drifts. We propose the Artificial Neural Twin, which combines concepts from model predictive control, deep learning, and sensor networks to address these issues. Our approach introduces decentral, differentiable data fusion to estimate the state of distributed process steps and their dependence on input data. By treating the interconnected process steps as a quasi neural-network, we can backpropagate loss gradients for process optimization or model fine-tuning to process parameters or AI models respectively. The concept is demonstrated on a virtual machine park simulated in Unity, consisting of bulk material processes in plastic recycling.

Enhanced river suspended sediment concentration identification via multimodal video image, optical flow, and water temperature data fusion.

Monitoring suspended sediment concentration (SSC) in rivers is pivotal for water quality management and sustainable river ecosystem development. However, achieving continuous and precise SSC monitoring is fraught with challenges, including low automation, lengthy measurement processes, and high cost. This study proposes an innovative approach for SSC identification in rivers using multimodal data fusion. We developed a robust model by harnessing colour features from video images, motion characteristics from the Lucas-Kanade (LK) optical flow method, and temperature data. By integrating ResNet with a mixed density network (MDN), our method fused the image and optical flow fields, and temperature data to enhance accuracy and reliability. Validated at a hydropower station in the Xinjiang Uygur Autonomous Region, China, the results demonstrated that while the image field alone offers a baseline level of SSC identification, it experiences local errors under specific conditions. The incorporation of optical flow and water temperature information enhanced model robustness, particularly when coupling the image and optical flow fields, yielding a Nash-Sutcliffe efficiency (NSE) of 0.91. Further enhancement was observed with the combined use of all three data types, attaining an NSE of 0.93. This integrated approach offers a more accurate SSC identification solution, enabling non-contact, low-cost measurements, facilitating remote online monitoring, and supporting water resource management and river water-sediment element monitoring.

Digital modelling method of coal-mine roadway based on millimeter-wave radar array.

The roadway space of coal mine is narrow, and the illumination is low and uneven. Dynamic mining is accompanied by dust and water mist. The accuracy and reliability of roadway data collected by vision and laser sensors are poor. Based on this, a digital modeling method of coal mine roadway based on millimeter-wave radar array is proposed. Firstly, aiming at the problem of complex environmental interference, combined with the characteristics of small amount of single frame data of millimeter-wave point cloud, a multi-layer filtering noise reduction processing and dynamic subgraph registration method of millimeter-wave point cloud is proposed to filter out interference points and realize single radar point cloud registration. Secondly, aiming at the problem that a single millimeter-wave radar cannot scan the complete roadway information at one time, combined with the characteristics of small elevation field of view of millimeter-wave radar, a millimeter-wave radar array acquisition system is built, and an improved iterative closest point (ICP) registration algorithm based on point cloud features is established to construct the roadway point cloud fusion model. Finally, aiming at the problem of uneven and sparse point cloud after array fusion, a Poisson surface reconstruction method based on point cloud density weighted interpolation is proposed to refine the roadway structure and realize the accurate reconstruction of digital roadway model. The experimental results show that the digital modeling method of millimeter-wave radar array can accurately obtain the information of roadway surrounding rock, the density of roadway point cloud is increased by 22.4%, and the average absolute error percentage of the width and height of the reconstructed roadway model is only 0.82% and 0.72%, which provides a new research method for the reconstruction of underground roadway and an important basis for the digital modeling method of coal mine roadway.