Temporal-Spatial Correlation Attention Network for Clinical Data Analysis in Intensive Care Unit.

Recent advancements in medical information technology have enabled electronic health records (EHRs) to store comprehensive clinical data which has ushered healthcare into the era of "big data". However, medical data are rather complicated, making problem-solving in healthcare be limited in scope and comprehensiveness. The rapid development of deep learning in recent years has opened up opportunities for leveraging big data in healthcare. In this article we introduce a temporal-spatial correlation attention network (TSCAN) to address various clinical characteristic prediction problems, including mortality prediction, length of stay prediction, physiologic decline detection, and phenotype classification. Leveraging the attention mechanism model's design, our approach efficiently identifies relevant items in clinical data and temporally correlated nodes based on specific tasks, resulting in improved prediction accuracy. Additionally, our method identifies crucial clinical indicators associated with significant outcomes, which can inform and enhance treatment options. Our experiments utilize data from the publicly accessible Medical Information Mart for Intensive Care (MIMIC-IV) database. Finally, our approach demonstrates notable performance improvements of 2.0% (metric) compared to other SOTA prediction methods. Specifically, we achieved an impressive 90.7% mortality rate prediction accuracy and 45.1% accuracy in length of stay prediction.

[Retrospective study on the modified Uhl technique of closed reduction and percutaneous pin in the treatment of Colles' fracture].

OBJECTIVE: To retrospectively assess the advantages of the modified Uhl technique in the treatment of Colles' fracture guided by the principles of Chinese osteosynthesis (CO) concept. METHODS: A retrospective study was conducted on 358 patients with Colles' fracture treated with the modified Uhl technique of closed reduction and percutaneous pin between January 2016 and June 2021. Out of these, 120 eligible cases were selected and categorized into two groups according to different surgical methods:the closed reduction and percutaneous pin group, and the open reduction group. Sixty-eight patients in the closed reduction and percutaneous pin group were treated with the modified Uhl technique, while fifty-two patients in the open reduction group were treated with open reduction and internal fixation using plates. The modified Sarmiento imaging score, Gartland-Werley wrist score, operation time, hospital stay, and treatment costs between the two groups were compared at a 6-month postoperative follow-up. RESULTS: There were no significant differences in terms of gender, age, affected side, injure factors, time of injury to surgery, Sarmiento imaging score, and Gartland-Werley wrist joint score (P>0.05). The closed reduction and percutaneous pin group exhibited an operation time of (35.88±14.11) minutes, hospitalization stay of (9.78±2.48) days, and treatment costs of (16 074.91±1 964.48) yuan, while the open reduction group demonstrated comparatively longer operation time of (65.48±14.26) minutes, hospitalization stay of (15.88±2.00) days, and treatment costs of (20 451.27±1 760.22) yuan (P<0.01). CONCLUSION: The modified Uhl technique presents notable advantages in the management of Colles' fracture, including reliable fixation, less trauma, shorter operation time, less pain, shorter hospital stay, and cost-effectiveness. This technique exhibits promising potential for broader clinical application. However, it is important to note that the pin could potentially damage tendons, and in cases of Colles' fractures with osteoporosis and comminuted fragments, additional techniques may be required for reliable fixation.

Deep reinforcement learning framework for thoracic diseases classification via prior knowledge guidance.

The chest X-ray is commonly employed in the diagnosis of thoracic diseases. Over the years, numerous approaches have been proposed to address the issue of automatic diagnosis based on chest X-rays. However, the limited availability of labeled data for related diseases remains a significant challenge in achieving accurate diagnoses. This paper focuses on the diagnostic problem of thorax diseases and presents a novel deep reinforcement learning framework. This framework incorporates prior knowledge to guide the learning process of diagnostic agents, and the model parameters can be continually updated as more data becomes available, mimicking a person's learning process. Specifically, our approach offers two key contributions: (1) prior knowledge can be acquired from pre-trained models using old data or similar data from other domains, effectively reducing the dependence on target domain data; and (2) the reinforcement learning framework enables the diagnostic agent to be as exploratory as a human, leading to improved diagnostic accuracy through continuous exploration. Moreover, this method effectively addresses the challenge of learning models with limited data, enhancing the model's generalization capability. We evaluate the performance of our approach using the well-known NIH ChestX-ray 14 and CheXpert datasets, and achieve competitive results. More importantly, in clinical application, we make considerable progress. The source code for our approach can be accessed at the following URL: https://github.com/NeaseZ/MARL.

HMNet: Hierarchical Multi-Scale Brain Tumor Segmentation Network.

An accurate and efficient automatic brain tumor segmentation algorithm is important for clinical practice. In recent years, there has been much interest in automatic segmentation algorithms that use convolutional neural networks. In this paper, we propose a novel hierarchical multi-scale segmentation network (HMNet), which contains a high-resolution branch and parallel multi-resolution branches. The high-resolution branch can keep track of the brain tumor's spatial details, and the multi-resolution feature exchange and fusion allow the network's receptive fields to adapt to brain tumors of different shapes and sizes. In particular, to overcome the large computational overhead caused by expensive 3D convolution, we propose a lightweight conditional channel weighting block to reduce GPU memory and improve the efficiency of HMNet. We also propose a lightweight multi-resolution feature fusion (LMRF) module to further reduce model complexity and reduce the redundancy of the feature maps. We run tests on the BraTS 2020 dataset to determine how well the proposed network would work. The dice similarity coefficients of HMNet for ET, WT, and TC are 0.781, 0.901, and 0.823, respectively. Many comparative experiments on the BraTS 2020 dataset and other two datasets show that our proposed HMNet has achieved satisfactory performance compared with the SOTA approaches.

Deep Correlated Joint Network for 2-D Image-Based 3-D Model Retrieval.

In this article, we propose a novel deep correlated joint network (DCJN) approach for 2-D image-based 3-D model retrieval. First, the proposed method can jointly learn two distinct deep neural networks, which are trained for individual modalities to learn two deep nonlinear transformations for visual feature extraction from the co-embedding feature space. Second, we propose the global loss function for the DCJN, consisting of a discriminative loss and a correlation loss. The discriminative loss aims to minimize the intraclass distance of the extracted features and maximize the interclass distance of such features to a large margin within each modality, while the correlation loss focuses on mitigating the distribution discrepancy across different modalities. Consequently, the proposed method can realize cross-modality feature extraction guided by the defined global loss function to benefit the similarity measure between 2-D images and 3-D models. For a comparison experiment, we contribute the current largest 2-D image-based 3-D model retrieval dataset. Moreover, the proposed method was further evaluated on three popular benchmarks, including the 3-D Shape Retrieval Contest 2014, 2016, and 2018 benchmarks. The extensive comparison experimental results demonstrate the superiority of this method over the state-of-the-art methods.

Monocular Image-Based 3-D Model Retrieval: A Benchmark.

Monocular image-based 3-D model retrieval aims to search for relevant 3-D models from a dataset given one RGB image captured in the real world, which can significantly benefit several applications, such as self-service checkout, online shopping, etc. To help advance this promising yet challenging research topic, we built a novel dataset and organized the first international contest for monocular image-based 3-D model retrieval. Moreover, we conduct a thorough analysis of the state-of-the-art methods. Existing methods can be classified into supervised and unsupervised methods. The supervised methods can be analyzed based on several important aspects, such as the strategies of domain adaptation, view fusion, loss function, and similarity measure. The unsupervised methods focus on solving this problem with unlabeled data and domain adaptation. Seven popular metrics are employed to evaluate the performance, and accordingly, we provide a thorough analysis and guidance for future work. To the best of our knowledge, this is the first benchmark for monocular image-based 3-D model retrieval, which aims to help related research in multiview feature learning, domain adaptation, and information retrieval.

Toward Region-Aware Attention Learning for Scene Graph Generation.

Scene graph generation (SGGen) is a challenging task due to a complex visual context of an image. Intuitively, the human visual system can volitionally focus on attended regions by salient stimuli associated with visual cues. For example, to infer the relationship between man and horse, the interaction between human leg and horseback can provide strong visual evidence to predict the predicate ride. Besides, the attended region face can also help to determine the object man. Till now, most of the existing works studied the SGGen by extracting coarse-grained bounding box features while understanding fine-grained visual regions received limited attention. To mitigate the drawback, this article proposes a region-aware attention learning method. The key idea is to explicitly construct the attention space to explore salient regions with the object and predicate inferences. First, we extract a set of regions in an image with the standard detection pipeline. Each region regresses to an object. Second, we propose the object-wise attention graph neural network (GNN), which incorporates attention modules into the graph structure to discover attended regions for object inference. Third, we build the predicate-wise co-attention GNN to jointly highlight subject's and object's attended regions for predicate inference. Particularly, each subject-object pair is connected with one of the latent predicates to construct one triplet. The proposed intra-triplet and inter-triplet learning mechanism can help discover the pair-wise attended regions to infer predicates. Extensive experiments on two popular benchmarks demonstrate the superiority of the proposed method. Additional ablation studies and visualization further validate its effectiveness.

DAN: Deep-Attention Network for 3D Shape Recognition.

Due to the wide applications in a rapidly increasing number of different fields, 3D shape recognition has become a hot topic in the computer vision field. Many approaches have been proposed in recent years. However, there remain huge challenges in two aspects: exploring the effective representation of 3D shapes and reducing the redundant complexity of 3D shapes. In this paper, we propose a novel deep-attention network (DAN) for 3D shape representation based on multiview information. More specifically, we introduce the attention mechanism to construct a deep multiattention network that has advantages in two aspects: 1) information selection, in which DAN utilizes the self-attention mechanism to update the feature vector of each view, effectively reducing the redundant information, and 2) information fusion, in which DAN applies attention mechanism that can save more effective information by considering the correlations among views. Meanwhile, deep network structure can fully consider the correlations to continuously fuse effective information. To validate the effectiveness of our proposed method, we conduct experiments on the public 3D shape datasets: ModelNet40, ModelNet10, and ShapeNetCore55. Experimental results and comparison with state-of-the-art methods demonstrate the superiority of our proposed method. Code is released on https://github.com/RiDang/DANN.

Sequential Saliency Guided Deep Neural Network for Joint Mitosis Identification and Localization in Time-Lapse Phase Contrast Microscopy Images.

The analysis of cell mitotic behavior plays important role in many biomedical research and medical diagnostic applications. To improve the accuracy of mitosis detection in automated analysis systems, this paper proposes the sequential saliency guided deep neural network (SSG-DNN) to jointly identify and localize mitotic events in time-lapse phase contrast microscopy images. It consists of three key modules. First, the module of visual context learning extracts static visual feature and dynamic visual transition within individual volumetric cell regions. Secondly, with these information, the module of sequential saliency modeling aims to discover the saliency distribution over all successive frames in each volumetric region. Finally, the module of sequence structure modeling can leverage both visual context and saliency distribution for mitosis identification and localization. SSG-DNN can jointly realize visual feature learning and sequential structure modeling in the end-to-end framework. Moreover, the proposed method is independent of complicated preconditioning methods for mitotic candidate extraction and can be applied for mitosis detection in one-shot manner. To our knowledge, it is the first weakly supervised work to realize joint mitosis identification and localization only with sequence-wise labels. In our experiments, we evaluate its performances of both tasks on the popular C3H10 dataset and a novel and large-scale dataset, C2C12-16, which contains much more mitotic events and is more challenging owing to diverse cell culture conditions. Experimental results can demonstrate the superiority of the proposed method.

Multi-Domain & Multi-Task Learning for Human Action Recognition.

Domain-invariant (view-invariant & modalityinvariant) feature representation is essential for human action recognition. Moreover, given a discriminative visual representation, it is critical to discover the latent correlations among multiple actions in order to facilitate action modeling. To address these problems, we propose a multi-domain & multi-task learning (MDMTL) method to (1) extract domain-invariant information for multi-view and multi-modal action representation and (2) explore the relatedness among multiple action categories. Specifically, we present a sparse transfer learning-based method to co-embed multi-domain (multi-view & multi-modality) data into a single common space for discriminative feature learning. Additionally, visual feature learning is incorporated into the multitask learning framework, with the Frobenius-norm regularization term and the sparse constraint term, for joint task modeling and task relatedness-induced feature learning. To the best of our knowledge, MDMTL is the first supervised framework to jointly realize domain-invariant feature learning and task modeling for multi-domain action recognition. Experiments conducted on the INRIA Xmas Motion Acquisition Sequences (IXMAS) dataset, the MSR Daily Activity 3D (DailyActivity3D) dataset, and the Multi-modal & Multi-view & Interactive (M2I) dataset, which is the most recent and largest multi-view and multi-model action recognition dataset, demonstrate the superiority of MDMTL over the state-of-the-art approaches.

[A comparative study of titanium elastic intramedullary nail internal fixation and bone plate internal fixation in the treatment of adult Galeazzi fracture].

Objective: To explore the effectiveness difference between titanium elastic intramedullary nail internal fixation and bone plate internal fixation in the treatment of adult Galeazzi fracture. Methods: Ninety-seven patients of Galeazzi fracture according with the selection criteria were divided into 2 groups by prospective cohort study, who were admitted between January 2012 and November 2015. In the patients, 59 were treated with open reduction and bone plate internal fixation (plate group), and 38 with titanium elastic intramedullary nail internal fixation (minimally invasive group). There was no significant difference in the gender, age, cause of injury, fracture site, type of fracture, and time from injury to operation between 2 groups ( P>0.05). The operation time, intraoperative blood loss, fracture healing time, and complications were recorded and compared between 2 groups, and the forearm function was evaluated by Anderson score. Results: All the patients were followed up 12-23 months (mean, 17 months). The operation time, intraoperative blood loss, fracture healing time of minimally invasive group were significantly less than those in plate group ( P<0.05). There were 1 case of fracture nonunion, 1 case of wound infection in plate group, and 1 case of nail tail slight infection in minimally invasive group, which were all cured after the corresponding treatment. The remaining patients had good fracture healing, and no vascular injury, internal fixation failure, deep infection, or other complications occurred. According to Anderson score at 12 months after operation, the forearm function results were excellent in 46 cases, good in 12 cases, and poor in 1 case, with an excellent and good rate of 98.3% in plate group; and the results were excellent in 26 cases, good in 11 cases, and poor in 1 case, with an excellent and good rate of 97.4% in minimally invasive group; showing no significant difference ( χ2=0.10, P=0.75). Conclusion: Minimally invasive fixation with titanium elastic nail has such advantages as small damage, quick recovery, no skin scarring, etc. As long as the correct indication is selected, minimally invasive titanium intramedullary nail internal fixation of Galeazzi fractures can also get good effectiveness.

View-Based 3-D Model Retrieval: A Benchmark.

View-based 3-D model retrieval is one of the most important techniques in numerous applications of computer vision. While many methods have been proposed in recent years, to the best of our knowledge, there is no benchmark to evaluate the state-of-the-art methods. To tackle this problem, we systematically investigate and evaluate the related methods by: 1) proposing a clique graph-based method and 2) reimplementing six representative methods. Moreover, we concurrently evaluate both hand-crafted visual features and deep features on four popular datasets (NTU60, NTU216, PSB, and ETH) and one challenging real-world multiview model dataset (MV-RED) prepared by our group with various evaluation criteria to understand how these algorithms perform. By quantitatively analyzing the performances, we discover the graph matching-based method with deep features, especially the clique graph matching algorithm with convolutional neural networks features, can usually outperform the others. We further discuss the future research directions in this field.

Multi-Grained Random Fields for Mitosis Identification in Time-Lapse Phase Contrast Microscopy Image Sequences.

This paper proposes a multi-grained random fields (MGRFs) model for mitosis identification. To deal with the difficulty in hidden state discovery and sequential structure modeling in mitosis sequences only containing gradual visual pattern changes, we design the graphical structure to transform individual sequence into a set of coarse-to-fine grained sequencesconveying diverse temporal dynamics. Furthermore, we propose the corresponding probabilistic model for joint temporal learning and feature learning. To deal with the non-convex formulation of MGRF, we decomposemodel training into two sub-tasks, layer-wise sequential learning of both temporal dynamics and visual feature and new layer generation by graph-based sequential grouping, and optimize the model by alternating between them iteratively. The proposed method is validated on very challenging mitosis data set of C3H10T1/2 and C2C12 stem cells. Extensive comparison experiments demonstrate its superiority to the state of the arts.

Hierarchical Clustering Multi-Task Learning for Joint Human Action Grouping and Recognition.

This paper proposes a hierarchical clustering multi-task learning (HC-MTL) method for joint human action grouping and recognition. Specifically, we formulate the objective function into the group-wise least square loss regularized by low rank and sparsity with respect to two latent variables, model parameters and grouping information, for joint optimization. To handle this non-convex optimization, we decompose it into two sub-tasks, multi-task learning and task relatedness discovery. First, we convert this non-convex objective function into the convex formulation by fixing the latent grouping information. This new objective function focuses on multi-task learning by strengthening the shared-action relationship and action-specific feature learning. Second, we leverage the learned model parameters for the task relatedness measure and clustering. In this way, HC-MTL can attain both optimal action models and group discovery by alternating iteratively. The proposed method is validated on three kinds of challenging datasets, including six realistic action datasets (Hollywood2, YouTube, UCF Sports, UCF50, HMDB51 & UCF101), two constrained datasets (KTH & TJU), and two multi-view datasets (MV-TJU & IXMAS). The extensive experimental results show that: 1) HC-MTL can produce competing performances to the state of the arts for action recognition and grouping; 2) HC-MTL can overcome the difficulty in heuristic action grouping simply based on human knowledge; 3) HC-MTL can avoid the possible inconsistency between the subjective action grouping depending on human knowledge and objective action grouping based on the feature subspace distributions of multiple actions. Comparison with the popular clustered multi-task learning further reveals that the discovered latent relatedness by HC-MTL aids inducing the group-wise multi-task learning and boosts the performance. To the best of our knowledge, ours is the first work that breaks the assumption that all actions are either independent for individual learning or correlated for joint modeling and proposes HC-MTL for automated, joint action grouping and modeling.

Multi-Modal Clique-Graph Matching for View-Based 3D Model Retrieval.

Multi-view matching is an important but a challenging task in view-based 3D model retrieval. To address this challenge, we propose an original multi-modal clique graph (MCG) matching method in this paper. We systematically present a method for MCG generation that is composed of cliques, which consist of neighbor nodes in multi-modal feature space and hyper-edges that link pairwise cliques. Moreover, we propose an image set-based clique/edgewise similarity measure to address the issue of the set-to-set distance measure, which is the core problem in MCG matching. The proposed MCG provides the following benefits: 1) preserves the local and global attributes of a graph with the designed structure; 2) eliminates redundant and noisy information by strengthening inliers while suppressing outliers; and 3) avoids the difficulty of defining high-order attributes and solving hyper-graph matching. We validate the MCG-based 3D model retrieval using three popular single-modal data sets and one novel multi-modal data set. Extensive experiments show the superiority of the proposed method through comparisons. Moreover, we contribute a novel real-world 3D object data set, the multi-view RGB-D object data set. To the best of our knowledge, it is the largest real-world 3D object data set containing multi-modal and multi-view information.

Jointly Learning Multiple Sequential Dynamics for Human Action Recognition.

Discovering visual dynamics during human actions is a challenging task for human action recognition. To deal with this problem, we theoretically propose the multi-task conditional random fields model and explore its application on human action recognition. For visual representation, we propose the part-induced spatiotemporal action unit sequence to represent each action sample with multiple partwise sequential feature subspaces. For model learning, we propose the multi-task conditional random fields (MTCRFs) model to discover the sequence-specific structure and the sequence-shared relationship. Specifically, the multi-chain graph structure and the corresponding probabilistic model are designed to represent the interaction among multiple part-induced action unit sequences. Moreover we propose the model learning and inference methods to discover temporal context within individual action unit sequence and the latent correlation among different body parts. Extensive experiments are implemented to demonstrate the superiority of the proposed method on two popular RGB human action datasets, KTH & TJU, and the depth dataset in MSR Daily Activity 3D.

[Comparison between mini-traumatic bone-grafting and non-bone-grafting in percutaneous K-wire fixation to treat the calcaneal fractures].

OBJECTIVE: To compare the effect between mini-traumatic bone-grafting and non-bone-grafting in percutaneous K-wire fixation for treating the calcaneal fractures. METHODS: From 2002 to 2006, 112 patients with the type II (Paley type) fractures of calcaneus were studied. There were 56 cases in bone-grafting group involving 36 males and 20 famales,aged from 21 to 65, averaged (42.0 +/- 2.3) years; 11 cases were in type II a and 45 were in type II b; the course was from 3 to 14 days, averaged (6.0 +/- 1.2) days. And there were 56 cases in non-bone-grafting group involving 38 males and 18 famales,aged from 22 to 67, averaged (43.0 +/- 2.5)years; 13 cases were in type II a and 43 were in type II b; the course was from 2 to 15 days, averaged (5.0-2.1) days. All the cases were treated by closed reduction and percutaneous K-wire fixation, and bone-grafting group(56 cases) were treated by mini-traumatic bone-grafting, but the other group (56 cases) were not. The collapsing rate and fineness rate were compared. RESULTS: All the cases were followed up from 5 to 52 months. There were no collapsing cases in the bone-grafting group after operation, but 3 cases occurrenced re-collapsing in the non-bone-grafting group. According to the Zhang Tie-liang's evaluation criterion, in the bone-grafting group,the results were excellent in 43 cases, good in 12, fair in 1, the fineness rate was 98.2%. In the non-bone-grafting group,the results were excellent in 37 cases, good in 16, fair in 2, poor in 1, the fineness rate was 94.7%. CONCLUSION: Treatment of the type II fracture of calcaneus with closed reduction, percutaneous K-wire fixation and mini-traumatic bone-grafting can prevent the posterior talar articular surface of caltaneus from collapsing again after operation, enhance the union of fracture, elevate the curative effect, thus it should be taken with the standard therapeutic regimen.