Zero-Shot Video Grounding With Pseudo Query Lookup and Verification.

Video grounding, the process of identifying a specific moment in an untrimmed video based on a natural language query, has become a popular topic in video understanding. However, fully supervised learning approaches for video grounding that require large amounts of annotated data can be expensive and time-consuming. Recently, zero-shot video grounding (ZS-VG) methods that leverage pre-trained object detectors and language models to generate pseudo-supervision for training video grounding models have been developed. However, these approaches have limitations in recognizing diverse categories and capturing specific dynamics and interactions in the video context. To tackle these challenges, we introduce a novel two-stage ZS-VG framework called Lookup-and-Verification (LoVe), which treats the pseudo-query generation procedure as a video-to-concept retrieval problem. Our approach allows for the extraction of diverse concepts from an open-concept pool and employs a verification process to ensure the relevance of the retrieved concepts to the objects or events of interest in the video proposals. Comprehensive experimental results on the Charades-STA, ActivityNet-Captions, and DiDeMo datasets demonstrate the effectiveness of the LoVe framework.

Collaborative Contrastive Refining for Weakly Supervised Person Search.

Weakly supervised person search involves training a model with only bounding box annotations, without human-annotated identities. Clustering algorithms are commonly used to assign pseudo-labels to facilitate this task. However, inaccurate pseudo-labels and imbalanced identity distributions can result in severe label and sample noise. In this work, we propose a novel Collaborative Contrastive Refining (CCR) weakly-supervised framework for person search that jointly refines pseudo-labels and the sample-learning process with different contrastive strategies. Specifically, we adopt a hybrid contrastive strategy that leverages both visual and context clues to refine pseudo-labels, and leverage the sample-mining and noise-contrastive strategy to reduce the negative impact of imbalanced distributions by distinguishing positive samples and noise samples. Our method brings two main advantages: 1) it facilitates better clustering results for refining pseudo-labels by exploring the hybrid similarity; 2) it is better at distinguishing query samples and noise samples for refining the sample-learning process. Extensive experiments demonstrate the superiority of our approach over the state-of-the-art weakly supervised methods by a large margin (more than 3% mAP on CUHK-SYSU). Moreover, by leveraging more diverse unlabeled data, our method achieves comparable or even better performance than the state-of-the-art supervised methods.

High expression of IL6 and decrease in immune cells in COVID-19 patients combined with myocardial injury.

Purpose: Myocardial injury, as a serious complication of coronavirus disease-2019 (COVID-19), increases the occurrence of adverse outcomes. Identification of key regulatory molecules of myocardial injury may help formulate corresponding treatment strategies and improve the prognosis of COVID-19 patients. Methods: Gene Set Enrichment Analysis (GSEA) was conducted to identify co-regulatory pathways. Differentially expressed genes (DEGs) in GSE150392 and GSE169241 were screened and an intersection analysis with key genes of the co-regulatory pathway was conducted. A protein-protein interaction (PPI) network was constructed to screen for key regulatory genes. Preliminarily screened genes were verified using other datasets to identify genes with consistent expression. Based on the hierarchical cluster, we divided the patients from GSE177477 into high- and low-risk groups and compared the proportion of immune cells. A total of 267 COVID-19 patients from the Zhejiang Provincial Hospital of Chinese Medicine from December 26, 2022, to January 11, 2023, were enrolled to verify the bioinformatics results. Univariate and multivariate analyses were performed to analyze the risk factors for myocardial injury. According to high-sensitivity troponin (hsTnI) levels, patients with COVID-19 were divided into high- and low-sensitivity groups, and interleukin 6 (IL6) expression and lymphocyte subsets were compared. Patients were also divided into high and low groups according to the IL6 expression, and hsTnI levels were compared. Results: Interleukin signaling pathway and GPCR ligand binding were shown to be co-regulatory pathways in myocardial injury associated with COVID-19. According to the hierarchical cluster analysis of seven genes (IL6, NFKBIA, CSF1, CXCL1, IL1R1, SOCS3, and CASP1), patients with myocardial injury could be distinguished from those without myocardial injury. Age, IL6 levels, and hospital stay may be factors influencing myocardial injury caused by COVID-19. Compared with COVID-19 patients without myocardial injury, the levels of IL6 in patients with myocardial injury increased, while the number of CD4+ T cells, CD8+ T cells, B cells, and NK cells decreased (P<0.05). The hsTnI levels in COVID-19 patients with high IL6 levels were higher than those in patients with low IL6 (P<0.05). Conclusions: The COVID-19 patients with myocardial injury had elevated IL6 expression and decreased lymphocyte counts. IL6 may participate in myocardial injury through the interleukin signaling pathway.

Divide and Retain: A Dual-Phase Modeling for Long-Tailed Visual Recognition.

This work explores visual recognition models on real-world datasets exhibiting a long-tailed distribution. Most of previous works are based on a holistic perspective that the overall gradient for training model is directly obtained by considering all classes jointly. However, due to the extreme data imbalance in long-tailed datasets, joint consideration of different classes tends to induce the gradient distortion problem; i.e., the overall gradient tends to suffer from shifted direction toward data-rich classes and enlarged variances caused by data-poor classes. The gradient distortion problem impairs the training of our models. To avoid such drawbacks, we propose to disentangle the overall gradient and aim to consider the gradient on data-rich classes and that on data-poor classes separately. We tackle the long-tailed visual recognition problem via a dual-phase-based method. In the first phase, only data-rich classes are concerned to update model parameters, where only separated gradient on data-rich classes is used. In the second phase, the rest data-poor classes are involved to learn a complete classifier for all classes. More importantly, to ensure the smooth transition from phase I to phase II, we propose an exemplar bank and a memory-retentive loss. In general, the exemplar bank reserves a few representative examples from data-rich classes. It is used to maintain the information of data-rich classes when transiting. The memory-retentive loss constrains the change of model parameters from phase I to phase II based on the exemplar bank and data-poor classes. The extensive experimental results on four commonly used long-tailed benchmarks, including CIFAR100-LT, Places-LT, ImageNet-LT, and iNaturalist 2018, highlight the excellent performance of our proposed method.

Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding.

Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. This task has achieved significant momentum in the computer vision community as it enables activity grounding beyond pre-defined activity classes by utilizing the semantic diversity of natural language descriptions. The semantic diversity is rooted in the principle of compositionality in linguistics, where novel semantics can be systematically described by combining known words in novel ways (compositional generalization). However, existing temporal grounding datasets are not carefully designed to evaluate the compositional generalizability. To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i.e., Charades-CG and ActivityNet-CG. We empirically find that they fail to generalize to queries with novel combinations of seen words. We argue that the inherent compositional structure (i.e., composition constituents and their relationships) inside the videos and language is the crucial factor to achieve compositional generalization. Based on this insight, we propose a variational cross-graph reasoning framework that explicitly decomposes video and language into hierarchical semantic graphs, respectively, and learns fine-grained semantic correspondence between the two graphs. Meanwhile, we introduce a novel adaptive structured semantics learning approach to derive the structure-informed and domain-generalizable graph representations, which facilitate the fine-grained semantic correspondence reasoning between the two graphs. To further evaluate the understanding of the compositional structure, we also introduce a more challenging setting, where one of the components in the novel composition is unseen. This requires more sophisticated understanding of the compositional structure to infer the potential semantics of the unseen word based on the other learned composition constituents appearing in both the video and language context, and their relationships. Extensive experiments validate the superior compositional generalizability of our approach, demonstrating its ability to handle queries with novel combinations of seen words as well as novel words in the testing composition.

Co-Learning Meets Stitch-Up for Noisy Multi-Label Visual Recognition.

In real-world scenarios, collected and annotated data often exhibit the characteristics of multiple classes and long-tailed distribution. Additionally, label noise is inevitable in large-scale annotations and hinders the applications of learning-based models. Although many deep learning based methods have been proposed for handling long-tailed multi-label recognition or label noise respectively, learning with noisy labels in long-tailed multi-label visual data has not been well-studied because of the complexity of long-tailed distribution entangled with multi-label correlation. To tackle such a critical yet thorny problem, this paper focuses on reducing noise based on some inherent properties of multi-label classification and long-tailed learning under noisy cases. In detail, we propose a Stitch-Up augmentation to synthesize a cleaner sample, which directly reduces multi-label noise by stitching up multiple noisy training samples. Equipped with Stitch-Up, a Heterogeneous Co-Learning framework is further designed to leverage the inconsistency between long-tailed and balanced distributions, yielding cleaner labels for more robust representation learning with noisy long-tailed data. To validate our method, we build two challenging benchmarks, named VOC-MLT-Noise and COCO-MLT-Noise, respectively. Extensive experiments are conducted to demonstrate the effectiveness of our proposed method. Compared to a variety of baselines, our method achieves superior results.

Bilaterally Normalized Scale-Consistent Sinkhorn Distance for Few-Shot Image Classification.

Few-shot image classification aims at exploring transferable features from base classes to recognize images of the unseen novel classes with only a few labeled images. Existing methods usually compare the support features and query features, which are implemented by either matching the global feature vectors or matching the local feature maps at the same position. However, few labeled images fail to capture all the diverse context and intraclass variations, leading to mismatch issues for existing methods. On one hand, due to the misaligned position and cluttered background, existing methods suffer from the object mismatch issue. On the other hand, due to the scale inconsistency between images, existing methods suffer from the scale mismatch issue. In this article, we propose the bilaterally normalized scale-consistent Sinkhorn distance (BSSD) to solve these issues. First, instead of same-position matching, we use the Sinkhorn distance to find an optimal matching between images, mitigating the object mismatch caused by misaligned position. Meanwhile, we propose the intraimage and interimage attentions as the bilateral normalization on the Sinkhorn distance to suppress the object mismatch caused by background clutter. Second, local feature maps are enhanced with the multiscale pooling strategy, making the Sinkhorn distance possible to find a consistent matching scale between images. Experimental results show the effectiveness of the proposed approach, and we achieve the state-of-the-art on three few-shot benchmarks.

Discriminative Radial Domain Adaptation.

Domain adaptation methods reduce domain shift typically by learning domain-invariant features. Most existing methods are built on distribution matching, e.g., adversarial domain adaptation, which tends to corrupt feature discriminability. In this paper, we propose Discriminative Radial Domain Adaptation (DRDR) which bridges source and target domains via a shared radial structure. It's motivated by the observation that as the model is trained to be progressively discriminative, features of different categories expand outwards in different directions, forming a radial structure. We show that transferring such an inherently discriminative structure would enable to enhance feature transferability and discriminability simultaneously. Specifically, we represent each domain with a global anchor and each category a local anchor to form a radial structure and reduce domain shift via structure matching. It consists of two parts, namely isometric transformation to align the structure globally and local refinement to match each category. To enhance the discriminability of the structure, we further encourage samples to cluster close to the corresponding local anchors based on optimal-transport assignment. Extensively experimenting on multiple benchmarks, our method is shown to consistently outperforms state-of-the-art approaches on varied tasks, including the typical unsupervised domain adaptation, multi-source domain adaptation, domain-agnostic learning, and domain generalization.

Symbiotic Attention for Egocentric Action Recognition With Object-Centric Alignment.

In this paper, we propose to tackle egocentric action recognition by suppressing background distractors and enhancing action-relevant interactions. The existing approaches usually utilize two independent branches to recognize egocentric actions, i.e., a verb branch and a noun branch. However, the mechanism to suppress distracting objects and exploit local human-object correlations is missing. To this end, we introduce two extra sources of information, i.e., the candidate objects spatial location and their discriminative features, to enable concentration on the occurring interactions. We design a Symbiotic Attention with Object-centric feature Alignment framework (SAOA) to provide meticulous reasoning between the actor and the environment. First, we introduce an object-centric feature alignment method to inject the local object features to the verb branch and noun branch. Second, we propose a symbiotic attention mechanism to encourage the mutual interaction between the two branches and select the most action-relevant candidates for classification. The framework benefits from the communication among the verb branch, the noun branch, and the local object information. Experiments based on different backbones and modalities demonstrate the effectiveness of our method. Notably, our framework achieves the state-of-the-art on the largest egocentric video dataset.

Filter Pruning by Switching to Neighboring CNNs With Good Attributes.

Filter pruning is effective to reduce the computational costs of neural networks. Existing methods show that updating the previous pruned filter would enable large model capacity and achieve better performance. However, during the iterative pruning process, even if the network weights are updated to new values, the pruning criterion remains the same. In addition, when evaluating the filter importance, only the magnitude information of the filters is considered. However, in neural networks, filters do not work individually, but they would affect other filters. As a result, the magnitude information of each filter, which merely reflects the information of an individual filter itself, is not enough to judge the filter importance. To solve the above problems, we propose meta-attribute-based filter pruning (MFP). First, to expand the existing magnitude information-based pruning criteria, we introduce a new set of criteria to consider the geometric distance of filters. Additionally, to explicitly assess the current state of the network, we adaptively select the most suitable criteria for pruning via a meta-attribute, a property of the neural network at the current state. Experiments on two image classification benchmarks validate our method. For ResNet-50 on ILSVRC-2012, we could reduce more than 50% FLOPs with only 0.44% top-5 accuracy loss.

Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification.

Though significant progress has been achieved on fine-grained visual classification (FGVC), severe overfitting still hinders model generalization. A recent study shows that hard samples in the training set can be easily fit, but most existing FGVC methods fail to classify some hard examples in the test set. The reason is that the model overfits those hard examples in the training set, but does not learn to generalize to unseen examples in the test set. In this article, we propose a moderate hard example modulation (MHEM) strategy to properly modulate the hard examples. MHEM encourages the model to not overfit hard examples and offers better generalization and discrimination. First, we introduce three conditions and formulate a general form of a modulated loss function. Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods. Moreover, we demonstrate that our baseline can be readily incorporated into the existing methods and empower these methods to be more discriminative. Equipped with our strong baseline, we achieve consistent improvements on three typical FGVC datasets, i.e., CUB-200-2011, Stanford Cars, and FGVC-Aircraft. We hope the idea of moderate hard example modulation will inspire future research work toward more effective fine-grained visual recognition.

Removal of p-Nitrophenol from simulated sewage using steel slag: Capability and mechanism.

The steel slag was investigated for the removal of p-nitrophenol (4-NP) from simulated sewage by batch adsorption and fixed-bed column absorption experiments. The results showed that the maximum adsorption capacity was 109.66 mg/g at 298 K, pH of 7, initial concentration 100 mg/L, and dose 0.8 g/L. The adsorption process fitted the Langmuir isothermal adsorption model and followed pseudo-second-order kinetic models, the activation energy of adsorption (Ea) was 10.78 kJ/mol, which indicated that the adsorption was single-molecule layer physical adsorption. The regeneration efficiency was still maintained at 84.20% after five adsorption-desorption cycles. The column adsorption experiments showed that the adsorption capacity of the Thomas model reached 13.69 mg/g and the semi-penetrating time of the Yoon-Nelson model was 205 min at 298 K. Fe3O4 was identified as the main adsorption site by adsorption energy calculation, XRD and XPS analysis. The FT-IR, Zeta potential, and ionic strength analysis indicated that the adsorption mechanism was hydrogen bonding interaction and electrostatic interaction. This work proved that steel slag could be utilized as a potential adsorbent for phenol-containing wastewater treatment.

Weakly Supervised RGB-D Salient Object Detection With Prediction Consistency Training and Active Scribble Boosting.

RGB-D salient object detection (SOD) has attracted increasingly more attention as it shows more robust results in complex scenes compared with RGB SOD. However, state-of-the-art RGB-D SOD approaches heavily rely on a large amount of pixel-wise annotated data for training. Such densely labeled annotations are often labor-intensive and costly. To reduce the annotation burden, we investigate RGB-D SOD from a weakly supervised perspective. More specifically, we use annotator-friendly scribble annotations as supervision signals for model training. Since scribble annotations are much sparser compared to ground-truth masks, some critical object structure information might be neglected. To preserve such structure information, we explicitly exploit the complementary edge information from two modalities (i.e., RGB and depth). Specifically, we leverage the dual-modal edge guidance and introduce a new network architecture with a dual-edge detection module and a modality-aware feature fusion module. In order to use the useful information of unlabeled pixels, we introduce a prediction consistency training scheme by comparing the predictions of two networks optimized by different strategies. Moreover, we develop an active scribble boosting strategy to provide extra supervision signals with negligible annotation cost, leading to significant SOD performance improvement. Extensive experiments on seven benchmarks validate the superiority of our proposed method. Remarkably, the proposed method with scribble annotations achieves competitive performance in comparison to fully supervised state-of-the-art methods.

Label Independent Memory for Semi-Supervised Few-Shot Video Classification.

In this paper, we propose to leverage freely available unlabeled video data to facilitate few-shot video classification. In this semi-supervised few-shot video classification task, millions of unlabeled data are available for each episode during training. These videos can be extremely imbalanced, while they have profound visual and motion dynamics. To tackle the semi-supervised few-shot video classification problem, we make the following contributions. First, we propose a label independent memory (LIM) to cache label related features, which enables a similarity search over a large set of videos. LIM produces a class prototype for few-shot training. This prototype is an aggregated embedding for each class, which is more robust to noisy video features. Second, we integrate a multi-modality compound memory network to capture both RGB and flow information. We propose to store the RGB and flow representation in two separate memory networks, but they are jointly optimized via a unified loss. In this way, mutual communications between the two modalities are leveraged to achieve better classification performance. Third, we conduct extensive experiments on the few-shot Kinetics-100, Something-Something-100 datasets, which validates the effectiveness of leveraging the accessible unlabeled data for few-shot classification.

Instance-Invariant Domain Adaptive Object Detection Via Progressive Disentanglement.

Most state-of-the-art methods of object detection suffer from poor generalization ability when the training and test data are from different domains. To address this problem, previous methods mainly explore to align distribution between source and target domains, which may neglect the impact of the domain-specific information existing in the aligned features. Besides, when transferring detection ability across different domains, it is important to extract the instance-level features that are domain-invariant. To this end, we explore to extract instance-invariant features by disentangling the domain-invariant features from the domain-specific features. Particularly, a progressive disentangled mechanism is proposed to decompose domain-invariant and domain-specific features, which consists of a base disentangled layer and a progressive disentangled layer. Then, with the help of Region Proposal Network (RPN), the instance-invariant features are extracted based on the output of the progressive disentangled layer. Finally, to enhance the disentangled ability, we design a detached optimization to train our model in an end-to-end fashion. Experimental results on four domain-shift scenes show our method is separately 2.3, 3.6, 4.0, and 2.0 percent higher than the baseline method. Meanwhile, visualization analysis demonstrates that our model owns well disentangled ability.

Untargeted metabolomic approach to study the serum metabolites in women with polycystic ovary syndrome.

BACKGROUND: Polycystic ovary syndrome (PCOS) is not only a kind of common endocrine syndrome but also a metabolic disorder, which harms the reproductive system and the whole body metabolism of the PCOS patients worldwide. In this study, we aimed to investigate the differences in serum metabolic profiles of the patients with PCOS compared to the healthy controls. MATERIAL AND METHODS: 31 PCOS patients and 31 matched healthy female controls were recruited in this study, the clinical characteristics data were recorded, the laboratory biochemical data were detected. Then, we utilized the metabolomics approach by UPLC-HRMS technology to study the serum metabolic changes between PCOS and controls. RESULTS: The metabolomics analysis showed that there were 68 downregulated and 78 upregulated metabolites in PCOS patients serum compared to those in the controls. These metabolites mainly belong to triacylglycerols, glycerophosphocholines, acylcarnitines, diacylglycerols, peptides, amino acids, glycerophosphoethanolamines and fatty acid. Pathway analysis showed that these metabolites were enriched in pathways including glycerophospholipid metabolism, fatty acid degradation, fatty acid biosynthesis, ether lipid metabolism, etc. Diagnosis value assessed by ROC analysis showed that the changed metabolites, including Leu-Ala/Ile-Ala, 3-(4-Hydroxyphenyl) propionic acid, Ile-Val/Leu-Val, Gly-Val/Val-Gly, aspartic acid, DG(34:2)_DG(16:0/18:2), DG(34:1)_DG(16:0/18:1), Phe-Trp, DG(36:1)_DG(18:0/18:1), Leu-Leu/Leu-Ile, had higher AUC values, indicated a significant role in PCOS. CONCLUSION: The present study characterized the difference of serum metabolites and related pathway profiles in PCOS patients, this finding hopes to provide potential metabolic markers for the prognosis and diagnosis of this disease.

Training Robust Object Detectors From Noisy Category Labels and Imprecise Bounding Boxes.

Object detection has gained great improvements with the advances of convolutional neural networks and the availability of large amounts of accurate training data. Though the amount of data is increasing significantly, the quality of data annotations is not guaranteed from the existing crowd-sourcing labeling platforms. In addition to noisy category labels, imprecise bounding box annotations are commonly existed for object detection data. When the quality of training data degenerates, the performance of the typical object detectors is severely impaired. In this paper, we propose a Meta-Refine-Net (MRNet) to train object detectors from noisy category labels and imprecise bounding boxes. First, MRNet learns to adaptively assign lower weights to proposals with incorrect labels so as to suppress large loss values generated by these proposals on the classification branch. Second, MRNet learns to dynamically generate more accurate bounding box annotations to overcome the misleading of imprecisely annotated bounding boxes. Thus, the imprecise bounding boxes could impose positive impacts on the regression branch rather than simply be ignored. Third, we propose to refine the imprecise bounding box annotations by jointly learning from both the category and the localization information. By doing this, the approximation of ground-truth bounding boxes is more accurate while the misleading would be further alleviated. Our MRNet is model-agnostic and is capable of learning from noisy object detection data with only a few clean examples (less than 2%). Extensive experiments on PASCAL VOC 2012 and MS COCO 2017 demonstrate the effectiveness and efficiency of our method.

DIA proteomics analysis through serum profiles reveals the significant proteins as candidate biomarkers in women with PCOS.

BACKGROUND: The aim of this study was to apply proteomic methodology for the analysis of proteome changes in women with polycystic ovary syndrome (PCOS). MATERIAL AND METHODS: All the participators including 31 PCOS patients and 31 healthy female as controls were recruited, the clinical characteristics data was recorded at the time of recruitment, the laboratory biochemical data was detected. Then, a data-independent acquisition (DIA)-based proteomics method was performed to compare the serum protein changes between PCOS patients and controls. In addition, Western blotting was used to validate the expression of identified proteomic biomarkers. RESULTS: There were 80 proteins differentially expressed between PCOS patients and controls significantly, including 54 downregulated and 26 upregulated proteins. Gene ontology and Kyoto Encyclopedia of Genes and Genomes analysis showed that downregulated proteins were enriched in platelet degranulation, cell adhesion, cell activation, blood coagulation, hemostasis, defense response and inflammatory response terms; upregulated proteins were enriched in cofactor catabolic process, hydrogen peroxide catabolic process, antioxidant activity, cellular oxidant detoxification, cellular detoxification, antibiotic catabolic process and hydrogen peroxide metabolic process. Receiver operating characteristic curves analysis showed that the area under curve of Histone H4 (H4), Histone H2A (H2A), Trem-like transcript 1 protein (TLT-1) were all over than 0.9, indicated promising diagnosis values of these proteins. Western blotting results proved that the detected significant proteins, including H4, H2A, TLT-1, Peroxiredoxin-1, Band 3 anion transport protein were all differently expressed in PCOS and control groups significantly. CONCLUSION: These proteomic biomarkers provided the potentiality to help us understand PCOS better, but future studies comparing systemic expression and exact role of these candidate biomarkers in PCOS are essential for confirmation of this hypothesis.

Few-Shot Common-Object Reasoning Using Common-Centric Localization Network.

In the few-shot common-localization task, given few support images without bounding box annotations at each episode, the goal is to localize the common object in the query image of unseen categories. The few-shot common-localization task involves common object reasoning from the given images, predicting the spatial locations of the object with different shapes, sizes, and orientations. In this work, we propose a common-centric localization (CCL) network for few-shot common-localization. The motivation of our common-centric localization network is to learn the common object features by dynamic feature relation reasoning via a graph convolutional network with conditional feature aggregation. First, we propose a local common object region generation pipeline to reduce background noises due to feature misalignment. Each support image predicts more accurate object spatial locations by replacing the query with the images in the support set. Second, we introduce a graph convolutional network with dynamic feature transformation to enforce the common object reasoning. To enhance the discriminability during feature matching and enable a better generalization in unseen scenarios, we leverage a conditional feature encoding function to alter visual features according to the input query adaptively. Third, we introduce a common-centric relation structure to model the correlation between the common features and the query image feature. The generated common features guide the query image feature towards a more common object-related representation. We evaluate our common-centric localization network on four datasets, i.e., CL-VOC-07, CL-VOC-12, CL-COCO, CL-VID. We obtain significant improvements compared to state-of-the-art. Our quantitative results confirm the effectiveness of our network.

Holistic LSTM for Pedestrian Trajectory Prediction.

Accurate predictions of future pedestrian trajectory could prevent a considerable number of traffic injuries and improve pedestrian safety. It involves multiple sources of information and real-time interactions, e.g., vehicle speed and ego-motion, pedestrian intention and historical locations. Existing methods directly apply a simple concatenation operation to combine multiple cues while their dynamics over time are less studied. In this paper, we propose a novel Long Short-Term Memory (LSTM), namely, to incorporate multiple sources of information from pedestrians and vehicles adaptively. Different from LSTM, our considers mutual interactions and explores intrinsic relations among multiple cues. First, we introduce extra memory cells to improve the transferability of LSTMs in modeling future variations. These extra memory cells include a speed cell to explicitly model vehicle speed dynamics, an intention cell to dynamically analyze pedestrian crossing intentions and a correlation cell to exploit correlations among temporal frames. These three individual cells uncover the future movement of vehicles, pedestrians and global scenes. Second, we propose a gated shifting operation to learn the movement of pedestrians. The intention of crossing the road or not would significantly affect pedestrian's spatial locations. To this end, global scene dynamics and pedestrian intention information are leveraged to model the spatial shifts. Third, we integrate the speed variations to the output gate and dynamically reweight the output channels via the scaling of vehicle speed. The movement of the vehicle would alter the scale of the predicted pedestrian bounding box: as the vehicle gets closer to the pedestrian, the bounding box is enlarging. Our rescaling process captures the relative movement and updates the size of pedestrian bounding boxes accordingly. Experiments conducted on three pedestrian trajectory forecasting benchmarks show that our achieves state-of-the-art performance.