Improved Chinese Giant Salamander Parental Care Behavior Detection Based on YOLOv8.

Optimizing the breeding techniques and increasing the hatching rate of Andrias davidianus offspring necessitates a thorough understanding of its parental care behaviors. However, A. davidianus' nocturnal and cave-dwelling tendencies pose significant challenges for direct observation. To address this problem, this study constructed a dataset for the parental care behavior of A. davidianus, applied the target detection method to this behavior for the first time, and proposed a detection model for A. davidianus' parental care behavior based on the YOLOv8s algorithm. Firstly, a multi-scale feature fusion convolution (MSConv) is proposed and combined with a C2f module, which significantly enhances the feature extraction capability of the model. Secondly, the large separable kernel attention is introduced into the spatial pyramid pooling fast (SPPF) layer to effectively reduce the interference factors in the complex environment. Thirdly, to address the problem of low quality of captured images, Wise-IoU (WIoU) is used to replace CIoU in the original YOLOv8 to optimize the loss function and improve the model's robustness. The experimental results show that the model achieves 85.7% in the mAP50-95, surpassing the YOLOv8s model by 2.1%. Compared with other mainstream models, the overall performance of our model is much better and can effectively detect the parental care behavior of A. davidianus. Our research method not only offers a reference for the behavior recognition of A. davidianus and other amphibians but also provides a new strategy for the smart breeding of A. davidianus.

PMSTD-Net: A Neural Prediction Network for Perceiving Multi-Scale Spatiotemporal Dynamics.

With the continuous advancement of sensing technology, applying large amounts of sensor data to practical prediction processes using artificial intelligence methods has become a developmental direction. In sensing images and remote sensing meteorological data, the dynamic changes in the prediction targets relative to their background information often exhibit more significant dynamic characteristics. Previous prediction methods did not specifically analyze and study the dynamic change information of prediction targets at spatiotemporal multi-scale. Therefore, this paper proposes a neural prediction network based on perceptual multi-scale spatiotemporal dynamic changes (PMSTD-Net). By designing Multi-Scale Space Motion Change Attention Unit (MCAU) to perceive the local situation and spatial displacement dynamic features of prediction targets at different scales, attention is ensured on capturing the dynamic information in their spatial dimensions adequately. On this basis, this paper proposes Multi-Scale Spatiotemporal Evolution Attention (MSEA) unit, which further integrates the spatial change features perceived by MCAU units in higher channel dimensions, and learns the spatiotemporal evolution characteristics at different scales, effectively predicting the dynamic characteristics and regularities of targets in sensor information.Through experiments on spatiotemporal prediction standard datasets such as Moving MNIST, video prediction dataset KTH, and Human3.6m, PMSTD-Net demonstrates prediction performance surpassing previous methods. We construct the GPM satellite remote sensing precipitation dataset, demonstrating the network's advantages in perceiving multi-scale spatiotemporal dynamic changes in remote sensing meteorological data. Finally, through extensive ablation experiments, the performance of each module in PMSTD-Net is thoroughly validated.

MS-YOLOv8-Based Object Detection Method for Pavement Diseases.

Detection of pavement diseases is crucial for road maintenance. Traditional methods are costly, time-consuming, and less accurate. This paper introduces an enhanced pavement disease recognition algorithm, MS-YOLOv8, which modifies the YOLOv8 model by incorporating three novel mechanisms to improve detection accuracy and adaptability to varied pavement conditions. The Deformable Large Kernel Attention (DLKA) mechanism adjusts convolution kernels dynamically, adapting to multi-scale targets. The Large Separable Kernel Attention (LSKA) enhances the SPPF feature extractor, boosting multi-scale feature extraction capabilities. Additionally, Multi-Scale Dilated Attention in the network's neck performs Spatially Weighted Dilated Convolution (SWDA) across different dilatation rates, enhancing background distinction and detection precision. Experimental results show that MS-YOLOv8 increases background classification accuracy by 6%, overall precision by 1.9%, and mAP by 1.4%, with specific disease detection mAP up by 2.9%. Our model maintains comparable detection speeds. This method offers a significant reference for automatic road defect detection.

Gearbox Fault Diagnosis Based on MSCNN-LSTM-CBAM-SE.

Ensuring the safety of mechanical equipment, gearbox fault diagnosis is crucial for the stable operation of the whole system. However, existing diagnostic methods still have limitations, such as the analysis of single-scale features and insufficient recognition of global temporal dependencies. To address these issues, this article proposes a new method for gearbox fault diagnosis based on MSCNN-LSTM-CBAM-SE. The output of the CBAM-SE module is deeply integrated with the multi-scale features from MSCNN and the temporal features from LSTM, constructing a comprehensive feature representation that provides richer and more precise information for fault diagnosis. The effectiveness of this method has been validated with two sets of gearbox datasets and through ablation studies on this model. Experimental results show that the proposed model achieves excellent performance in terms of accuracy and F1 score, among other metrics. Finally, a comparison with other relevant fault diagnosis methods further verifies the advantages of the proposed model. This research offers a new solution for accurate fault diagnosis of gearboxes.

A Dual-Branch Fusion of a Graph Convolutional Network and a Convolutional Neural Network for Hyperspectral Image Classification.

Semi-supervised graph convolutional networks (SSGCNs) have been proven to be effective in hyperspectral image classification (HSIC). However, limited training data and spectral uncertainty restrict the classification performance, and the computational demands of a graph convolution network (GCN) present challenges for real-time applications. To overcome these issues, a dual-branch fusion of a GCN and convolutional neural network (DFGCN) is proposed for HSIC tasks. The GCN branch uses an adaptive multi-scale superpixel segmentation method to build fusion adjacency matrices at various scales, which improves the graph convolution efficiency and node representations. Additionally, a spectral feature enhancement module (SFEM) enhances the transmission of crucial channel information between the two graph convolutions. Meanwhile, the CNN branch uses a convolutional network with an attention mechanism to focus on detailed features of local areas. By combining the multi-scale superpixel features from the GCN branch and the local pixel features from the CNN branch, this method leverages complementary features to fully learn rich spatial-spectral information. Our experimental results demonstrate that the proposed method outperforms existing advanced approaches in terms of classification efficiency and accuracy across three benchmark data sets.

Evolutionary Implications of Self-Assembling Cybernetic Materials with Collective Problem-Solving Intelligence at Multiple Scales.

In recent years, the scientific community has increasingly recognized the complex multi-scale competency architecture (MCA) of biology, comprising nested layers of active homeostatic agents, each forming the self-orchestrated substrate for the layer above, and, in turn, relying on the structural and functional plasticity of the layer(s) below. The question of how natural selection could give rise to this MCA has been the focus of intense research. Here, we instead investigate the effects of such decision-making competencies of MCA agential components on the process of evolution itself, using in silico neuroevolution experiments of simulated, minimal developmental biology. We specifically model the process of morphogenesis with neural cellular automata (NCAs) and utilize an evolutionary algorithm to optimize the corresponding model parameters with the objective of collectively self-assembling a two-dimensional spatial target pattern (reliable morphogenesis). Furthermore, we systematically vary the accuracy with which the uni-cellular agents of an NCA can regulate their cell states (simulating stochastic processes and noise during development). This allows us to continuously scale the agents' competency levels from a direct encoding scheme (no competency) to an MCA (with perfect reliability in cell decision executions). We demonstrate that an evolutionary process proceeds much more rapidly when evolving the functional parameters of an MCA compared to evolving the target pattern directly. Moreover, the evolved MCAs generalize well toward system parameter changes and even modified objective functions of the evolutionary process. Thus, the adaptive problem-solving competencies of the agential parts in our NCA-based in silico morphogenesis model strongly affect the evolutionary process, suggesting significant functional implications of the near-ubiquitous competency seen in living matter.

Investigation of the effect of ultrasonication on starch-fatty acid complexes and the stabilization mechanism.

The complexation of physically modified starch with fatty acids is favorable for the production of resistant starch. However, there is a lack of information on the effect of ultrasonication (UC) on the structure and properties of starch complexes and the molecular mechanism of the stabilization. Here, the multi-scale structure and in vitro digestive properties of starch-fatty acid complexes before and after UC were investigated, and the stabilization mechanisms of starch and fatty acids were explored. The results showed that the physicochemical properties and multi-scale structure of the starch-fatty acid complexes significantly changed with the type of fatty acids. The solubility and swelling power of the starch-fatty acid complexes were significantly decreased after UC (P < 0.05), which facilitated the binding of starch with fatty acids. The XRD results revealed that after the addition of fatty acids, the starch-fatty acid complexes showed typical V-shaped complexes. In addition, the starch-fatty acid complexes showed a significant increase in complexing index, improved short-range ordering and enhanced thermal stability. However, the differences in the structure and properties of the fatty acids themselves resulted in no significant improvement in the multi-scale structure of maize starch-palmitic acid by UC. In terms of digestibility, especially the complexes after UC were more compact in structure, which increased the difficulty of enzymatic digestion and thus slowed down the digestion process. DFT calculations and combined with FT-IR analysis showed that non-covalent interactions such as hydrogen bonding and hydrophobic interactions were the main driving force for the formation of the complexes, with binding energies (lauric acid, myristic acid and palmitic acid) of -30.50, -22.14 and -14.10 kcal/mol, respectively. Molecular dynamics simulations further confirmed the molecular mechanism of inclusion complex formation and stabilization. This study is important for the regulation of starchy foods by controlling processing conditions, and provides important information on the role of fatty acids in the regulation of starch complexes and the binding mechanism.

SB-Net: Synergizing CNN and LSTM networks for uncovering retrosynthetic pathways in organic synthesis.

Retrosynthesis is vital in synthesizing target products, guiding reaction pathway design crucial for drug and material discovery. Current models often neglect multi-scale feature extraction, limiting efficacy in leveraging molecular descriptors. Our proposed SB-Net model, a deep-learning architecture tailored for retrosynthesis prediction, addresses this gap. SB-Net combines CNN and Bi-LSTM architectures, excelling in capturing multi-scale molecular features. It integrates parallel branches for processing one-hot encoded descriptors and ECFP, merging through dense layers. Experimental results demonstrate SB-Net's superiority, achieving 73.6 % top-1 and 94.6 % top-10 accuracy on USPTO-50k data. Versatility is validated on MetaNetX, with rates of 52.8 % top-1, 74.3 % top-3, 79.8 % top-5, and 83.5 % top-10. SB-Net's success in bioretrosynthesis prediction tasks indicates its efficacy. This research advances computational chemistry, offering a robust deep-learning model for retrosynthesis prediction. With implications for drug discovery and synthesis planning, SB-Net promises innovative and efficient pathways.

Multi-scale variational autoencoder for imputation of missing values in untargeted metabolomics using whole-genome sequencing data.

BACKGROUND: Missing data is a common challenge in mass spectrometry-based metabolomics, which can lead to biased and incomplete analyses. The integration of whole-genome sequencing (WGS) data with metabolomics data has emerged as a promising approach to enhance the accuracy of data imputation in metabolomics studies. METHOD: In this study, we propose a novel method that leverages the information from WGS data and reference metabolites to impute unknown metabolites. Our approach utilizes a multi-scale variational autoencoder to jointly model the burden score, polygenetic risk score (PGS), and linkage disequilibrium (LD) pruned single nucleotide polymorphisms (SNPs) for feature extraction and missing metabolomics data imputation. By learning the latent representations of both omics data, our method can effectively impute missing metabolomics values based on genomic information. RESULTS: We evaluate the performance of our method on empirical metabolomics datasets with missing values and demonstrate its superiority compared to conventional imputation techniques. Using 35 template metabolites derived burden scores, PGS and LD-pruned SNPs, the proposed methods achieved R2-scores > 0.01 for 71.55 % of metabolites. CONCLUSION: The integration of WGS data in metabolomics imputation not only improves data completeness but also enhances downstream analyses, paving the way for more comprehensive and accurate investigations of metabolic pathways and disease associations. Our findings offer valuable insights into the potential benefits of utilizing WGS data for metabolomics data imputation and underscore the importance of leveraging multi-modal data integration in precision medicine research.

Improved YOLO-FastestV2 wheat spike detection model based on a multi-stage attention mechanism with a LightFPN detection head.

The number of wheat spikes has an important influence on wheat yield, and the rapid and accurate detection of wheat spike numbers is of great significance for wheat yield estimation and food security. Computer vision and machine learning have been widely studied as potential alternatives to human detection. However, models with high accuracy are computationally intensive and time consuming, and lightweight models tend to have lower precision. To address these concerns, YOLO-FastestV2 was selected as the base model for the comprehensive study and analysis of wheat sheaf detection. In this study, we constructed a wheat target detection dataset comprising 11,451 images and 496,974 bounding boxes. The dataset for this study was constructed based on the Global Wheat Detection Dataset and the Wheat Sheaf Detection Dataset, which was published by PP Flying Paddle. We selected three attention mechanisms, Large Separable Kernel Attention (LSKA), Efficient Channel Attention (ECA), and Efficient Multi-Scale Attention (EMA), to enhance the feature extraction capability of the backbone network and improve the accuracy of the underlying model. First, the attention mechanism was added after the base and output phases of the backbone network. Second, the attention mechanism that further improved the model accuracy after the base and output phases was selected to construct the model with a two-phase added attention mechanism. On the other hand, we constructed SimLightFPN to improve the model accuracy by introducing SimConv to improve the LightFPN module. The results of the study showed that the YOLO-FastestV2-SimLightFPN-ECA-EMA hybrid model, which incorporates the ECA attention mechanism in the base stage and introduces the EMA attention mechanism and the combination of SimLightFPN modules in the output stage, has the best overall performance. The accuracy of the model was P=83.91%, R=78.35%, AP= 81.52%, and F1 = 81.03%, and it ranked first in the GPI (0.84) in the overall evaluation. The research examines the deployment of wheat ear detection and counting models on devices with constrained resources, delivering novel solutions for the evolution of agricultural automation and precision agriculture.

Application of physics encoded neural networks to improve predictability of properties of complex multi-scale systems.

Predicting physical properties of complex multi-scale systems is a common challenge and demands analysis of various temporal and spatial scales. However, physics alone is often not sufficient due to lack of knowledge on certain details of the system. With sufficient data, however, machine learning techniques may aid. If data are yet relatively cumbersome to obtain, hybrid methods may come to the rescue. We focus in this report on using various types of neural networks (NN) including NN's into which physics information is encoded (PeNN's) and also studied effects of NN's hyperparameters. We apply the networks to predict the viscosity of an emulsion as a function of shear rate. We show that using various network performance metrics as the mean squared error and the coefficient of determination ( R 2 ) that the PeNN's always perform better than the NN's, as also confirmed by a Friedman test with a p-value smaller than 0.0002. The PeNN's capture extrapolation and interpolation very well, contrary to the NN's. In addition, we have found that the NN's hyperparameters including network complexity and optimization methods do not have any effect on the above conclusions. We suggest that encoding NN's with any disciplinary system based information yields promise to better predict properties of complex systems than NN's alone, which will be in particular advantageous for small numbers of data. Such encoding would also be scalable, allowing different properties to be combined, without repetitive training of the NN's.

Multi-scale brain attributes contribute to the distribution of diffuse glioma subtypes.

Gliomas are primary brain tumors and are among the most malignant types. Adult-type diffuse gliomas can be classified based on their histological and molecular signatures as IDH-wildtype glioblastoma, IDH-mutant astrocytoma, and IDH-mutant and 1p/19q-codeleted oligodendroglioma. Recent studies have shown that each subtype of glioma has its own specific distribution pattern. However, the mechanisms underlying the specific distributions of glioma subtypes are not entirely clear despite partial explanations such as cell origin. To investigate the impact of multi-scale brain attributes on glioma distribution, we constructed cumulative frequency maps for diffuse glioma subtypes based on T1w structural images and evaluated the spatial correlation between tumor frequency and diverse brain attributes, including postmortem gene expression, functional connectivity metrics, cerebral perfusion, glucose metabolism, and neurotransmitter signaling. Regression models were constructed to evaluate the contribution of these factors to the anatomic distribution of different glioma subtypes. Our findings revealed that the three different subtypes of gliomas had distinct distribution patterns, showing spatial preferences toward different brain environmental attributes. Glioblastomas were especially likely to occur in regions enriched with synapse-related pathways and diverse neurotransmitter receptors. Astrocytomas and oligodendrogliomas preferentially occurred in areas enriched with genes associated with neutrophil-mediated immune responses. The functional network characteristics and neurotransmitter distribution also contributed to oligodendroglioma distribution. Our results suggest that different brain transcriptomic, neurotransmitter, and connectomic attributes are the factors that determine the specific distributions of glioma subtypes. These findings highlight the importance of bridging diverse scales of biological organization when studying neurological dysfunction.

Single-Shot Direct Transmission Terahertz Imaging Based on Intense Broadband Terahertz Radiation.

There are numerous applications of terahertz (THz) imaging in many fields. However, current THz imaging is generally based on scanning technique due to the limited intensity of the THz sources. Thus, it takes a long time to obtain a frame image of the target and cannot meet the requirement of fast THz imaging. Here, we demonstrate a single-shot direct THz imaging strategy based on a broadband intense THz source with a frequency range of 0.1~23 THz and a THz camera with a frequency response range of 1~7 THz. This THz source was generated from the laser-plasma interaction, with its central frequency at ~12 THz. The frame rate of this imaging system was 8.5 frames per second. The imaging resolution reached 146.2 µm. With this imaging system, a single-shot THz image for a target object with a size of more than 7 cm was routinely obtained, showing a potential application for fast THz imaging. Furthermore, we proposed and tested an image enhancement algorithm based on an improved dark channel prior (DCP) theory and multi-scale retinex (MSR) theory to optimize the image brightness, contrast, entropy and peak signal-to-noise ratio (PSNR).

An Improved U-Net Infrared Small Target Detection Algorithm Based on Multi-Scale Feature Decomposition and Fusion and Attention Mechanism.

Infrared small target detection technology plays a crucial role in various fields such as military reconnaissance, power patrol, medical diagnosis, and security. The advancement of deep learning has led to the success of convolutional neural networks in target segmentation. However, due to challenges like small target scales, weak signals, and strong background interference in infrared images, convolutional neural networks often face issues like leakage and misdetection in small target segmentation tasks. To address this, an enhanced U-Net method called MST-UNet is proposed, the method combines multi-scale feature decomposition and fusion and attention mechanisms. The method involves using Haar wavelet transform instead of maximum pooling for downsampling in the encoder to minimize feature loss and enhance feature utilization. Additionally, a multi-scale residual unit is introduced to extract contextual information at different scales, improving sensory field and feature expression. The inclusion of a triple attention mechanism in the encoder structure further enhances multidimensional information utilization and feature recovery by the decoder. Experimental analysis on the NUDT-SIRST dataset demonstrates that the proposed method significantly improves target contour accuracy and segmentation precision, achieving IoU and nIoU values of 80.09% and 80.19%, respectively.

A Dynamic Multi-Scale Convolution Model for Face Recognition Using Event-Related Potentials.

With the development of data mining technology, the analysis of event-related potential (ERP) data has evolved from statistical analysis of time-domain features to data-driven techniques based on supervised and unsupervised learning. However, there are still many challenges in understanding the relationship between ERP components and the representation of familiar and unfamiliar faces. To address this, this paper proposes a model based on Dynamic Multi-Scale Convolution for group recognition of familiar and unfamiliar faces. This approach uses generated weight masks for cross-subject familiar/unfamiliar face recognition using a multi-scale model. The model employs a variable-length filter generator to dynamically determine the optimal filter length for time-series samples, thereby capturing features at different time scales. Comparative experiments are conducted to evaluate the model's performance against SOTA models. The results demonstrate that our model achieves impressive outcomes, with a balanced accuracy rate of 93.20% and an F1 score of 88.54%, outperforming the methods used for comparison. The ERP data extracted from different time regions in the model can also provide data-driven technical support for research based on the representation of different ERP components.

Multi-scale object equalization learning network for intracerebral hemorrhage region segmentation.

Segmentation and the subsequent quantitative assessment of the target object in computed tomography (CT) images provide valuable information for the analysis of intracerebral hemorrhage (ICH) pathology. However, most existing methods lack a reasonable strategy to explore the discriminative semantics of multi-scale ICH regions, making it difficult to address the challenge of complex morphology in clinical data. In this paper, we propose a novel multi-scale object equalization learning network (MOEL-Net) for accurate ICH region segmentation. Specifically, we first introduce a shallow feature extraction module (SFEM) for obtaining shallow semantic representations to maintain sufficient and effective detailed location information. Then, a deep feature extraction module (DFEM) is leveraged to extract the deep semantic information of the ICH region from the combination of SFEM and original image features. To further achieve equalization learning in different scales of ICH regions, we introduce a multi-level semantic feature equalization fusion module (MSFEFM), which explores the equalized fusion features of the described objects with the assistance of shallow and deep semantic information provided by SFEM and DFEM. Driven by the above three designs, MOEL-Net shows a solid capacity to capture more discriminative features in various ICH region segmentation. To promote the research of clinical automatic ICH region segmentation, we collect two datasets, VMICH and FRICH (divided into Test A and Test B) for evaluation. Experimental results show that the proposed model achieves the Dice scores of 88.28%, 90.92%, and 90.95% on the VMICH, FRICH Test A, and Test B, respectively, which outperform fourteen competing methods.

Multi-scale assessment of rainwater harvesting availability across the continental U.S.

Addressing resilience, sustainability, and water resource conservation has become increasingly important in the modern world. Challenges arise due to periodic droughts, climate change, and seasonal variability in areas with limited freshwater availability. Therefore, implementing and promoting water reuse is essential. Rainwater harvesting (RWH) is one such alternative, offering benefits in conserving water resources and mitigating droughts while reducing urban flooding and costs by generating alternative lower-cost water sources. Providing users with knowledge of available volumes for harvesting, including homeowners and governmental entities, is key to encouraging this practice. Hydrological data and geographic information systems are fundamental for managing, designing, and projecting rainwater harvesting practices. However, no tools currently integrate this information at multiple scales with current and future climate scenarios. This research aimed to develop a multi-scale assessment tool named H2O HARVEST, for evaluating the availability and potential of rainwater harvesting. Additional benefits of the H2O HARVEST app include aiding decision-making by national governmental entities and analyzing potential future scenarios for homeowner users. The app also provides regulatory policy information at the state level. We offer an app with the necessary capabilities to bridge the technology gap and promote rainwater harvesting practices. Our research demonstrated that RWH has the potential to be a sustainable water reuse practice. For more than 50% of the states, the RWH could supply at least 50% of the water demand. The regions of the US with the greatest potential are the Central and East.

Driving risk identification of urban arterial and collector roads based on multi-scale data.

Urban arterial and collector roads, while interconnected within the urban transportation network, serve distinct purposes, leading to different driving risk profiles. Investigating these differences using advanced methods is of paramount significance. This study aims to achieve this by primarily collecting and processing relevant vehicle trajectory data alongside driver-vehicle-road-environment data. A comprehensive risk assessment matrix is constructed to assess driving risks, incorporating multiple conflict and traffic flow indicators with statistically temporal stability. The Entropy weight-TOPSIS method and the K-means algorithm are employed to determine the risk scores and levels of the target arterial and collector roads. Using risk levels as the outcome variables and multi-scale features as the explanatory variables, random parameters models with heterogeneity in means and variances are developed to identify the determinants of driving risks at different levels. Likelihood ratio tests and comparisons of out-of-sample and within-sample prediction are conducted. Results reveal significant statistical differences in the risk profiles between arterial and collector roads. The marginal effects of significant parameters are then calculated separately for arterial and collector roads, indicating that several factors have different impacts on the probability of risk levels for arterial and collector roads, such as the number of movable elements in road landscape pictures, the standard deviation of the vehicle's lateral acceleration, the average standard deviation of speed for all vehicles on the road segment, and the number of one-way lanes on the road segment. Some practical implications are provided based on the findings. Future research can be implemented by expanding the collected data to different regions and cities over longer periods.

Eigen-entropy based time series signatures to support multivariate time series classification.

Most current algorithms for multivariate time series classification tend to overlook the correlations between time series of different variables. In this research, we propose a framework that leverages Eigen-entropy along with a cumulative moving window to derive time series signatures to support the classification task. These signatures are enumerations of correlations among different time series considering the temporal nature of the dataset. To manage dataset's dynamic nature, we employ preprocessing with dense multi scale entropy. Consequently, the proposed framework, Eigen-entropy-based Time Series Signatures, captures correlations among multivariate time series without losing its temporal and dynamic aspects. The efficacy of our algorithm is assessed using six binary datasets sourced from the University of East Anglia, in addition to a publicly available gait dataset and an institutional sepsis dataset from the Mayo Clinic. We use recall as the evaluation metric to compare our approach against baseline algorithms, including dependent dynamic time warping with 1 nearest neighbor and multivariate multi-scale permutation entropy. Our method demonstrates superior performance in terms of recall for seven out of the eight datasets.

TransMVAN: Multi-view Aggregation Network with Transformer for Pneumonia Diagnosis.

Automated and accurate classification of pneumonia plays a crucial role in improving the performance of computer-aided diagnosis systems for chest X-ray images. Nevertheless, it is a challenging task due to the difficulty of learning the complex structure information of lung abnormality from chest X-ray images. In this paper, we propose a multi-view aggregation network with Transformer (TransMVAN) for pneumonia classification in chest X-ray images. Specifically, we propose to incorporate the knowledge from glance and focus views to enrich the feature representation of lung abnormality. Moreover, to capture the complex relationships among different lung regions, we propose a bi-directional multi-scale vision Transformer (biMSVT), with which the informative messages between different lung regions are propagated through two directions. In addition, we also propose a gated multi-view aggregation (GMVA) to adaptively select the feature information from glance and focus views for further performance enhancement of pneumonia diagnosis. Our proposed method achieves AUCs of 0.9645 and 0.9550 for pneumonia classification on two different chest X-ray image datasets. In addition, it achieves an AUC of 0.9761 for evaluating positive and negative polymerase chain reaction (PCR). Furthermore, our proposed method also attains an AUC of 0.9741 for classifying non-COVID-19 pneumonia, COVID-19 pneumonia, and normal cases. Experimental results demonstrate the effectiveness of our method over other methods used for comparison in pneumonia diagnosis from chest X-ray images.