Aggregated Contextual Transformations for High-Resolution Image Inpainting.

Image inpainting that completes large free-form missing regions in images is a promising yet challenging task. State-of-the-art approaches have achieved significant progress by taking advantage of generative adversarial networks (GAN). However, these approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512×512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release codes and models in https://github.com/researchmm/AOT-GAN-for-Inpainting.

A Low Rank Promoting Prior for Unsupervised Contrastive Learning.

Unsupervised learning is just at a tipping point where it could really take off. Among these approaches, contrastive learning has led to state-of-the-art performance. In this paper, we construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning, referred to as LORAC. In contrast to the existing conventional self-supervised approaches that only considers independent learning, our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. This heuristic poses particular joint learning constraints to reduce the degree of freedom of the problem during the search of the optimal network parameterization. Most importantly, we argue that the low rank prior employed here is not unique, and many different priors can be invoked in a similar probabilistic way, corresponding to different hypotheses about underlying truth behind the contrastive features. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks, including image classification, object detection, instance segmentation and keypoint detection. Code is available: https://github.com/ssl-codelab/lorac.

Reference-Based Defect Detection Network.

The defect detection task can be regarded as a realistic scenario of object detection in the computer vision field and it is widely used in the industrial field. Directly applying vanilla object detector to defect detection task can achieve promising results, while there still exists challenging issues that have not been solved. The first issue is the texture shift which means a trained defect detector model will be easily affected by unseen texture, and the second issue is partial visual confusion which indicates that a partial defect box is visually similar with a complete box. To tackle these two problems, we propose a Reference-based Defect Detection Network (RDDN). Specifically, we introduce template reference and context reference to against those two problems, respectively. Template reference can reduce the texture shift from image, feature or region levels, and encourage the detectors to focus more on the defective area as a result. We can use either well-aligned template images or the outputs of a pseudo template generator as template references in this work, and they are jointly trained with detectors by the supervision of normal samples. To solve the partial visual confusion issue, we propose to leverage the carried context information of context reference, which is the concentric bigger box of each region proposal, to perform more accurate region classification and regression. Experiments on two defect detection datasets demonstrate the effectiveness of our proposed approach.

oComm: Overlapping Community Detection in Multi-View Brain Network.

Many efforts have been made on developing multi-view network community detection approaches. However, most of them can only reveal non-overlapping community structure. In this paper, we propose a novel approach for Overlapping Community Detection in Multi-view Brain Network (oComm). For modeling the overlapping community structure, a community membership strength vector is introduced for each node in each view, based on which a network generative model is designed to measure the within-view community quality. For measuring the consistency of overlapping community structures across different views, the Jaccard similarity is adopted to measure the first-order structural consistency of one node across different views, based on which a cross-view community consistency model is established. One objective function is defined by integrating the above two components. By solving the objective function via the alternative coordinate gradient ascent method, the optimal community membership strength vectors are generated, from which the multi-view overlapping community structure is obtained. Additionally, this study collects a set of EEG data of 147 subjects from Department of Otolaryngology of Sun Yat-sen Memorial Hospital, Sun Yat-sen University, based on which three multi-view brain networks are constructed. Comparison results with several existing approaches have confirmed the effectiveness of the proposed method.

MVStream: Multiview Data Stream Clustering.

This article studies a new problem of data stream clustering, namely, multiview data stream (MVStream) clustering. Although many data stream clustering algorithms have been developed, they are restricted to the single-view streaming data, and clustering MVStreams still remains largely unsolved. In addition to the many issues encountered by the conventional single-view data stream clustering, such as capturing cluster evolution and discovering clusters of arbitrary shapes under the limited computational resources, the main challenge of MVStream clustering lies in integrating information from multiple views in a streaming manner and abstracting summary statistics from the integrated features simultaneously. In this article, we propose a novel MVStream clustering algorithm for the first time. The main idea is to design a multiview support vector domain description (MVSVDD) model, by which the information from multiple insufficient views can be integrated, and the outputting support vectors (SVs) are utilized to abstract the summary statistics of the historical multiview data objects. Based on the MVSVDD model, a new multiview cluster labeling method is designed, whereby clusters of arbitrary shapes can be discovered for each view. By tracking the cluster labels of SVs in each view, the cluster evolution associated with concept drift can be captured. Since the SVs occupy only a small portion of data objects, the proposed MVStream algorithm is quite efficient with the limited computational resources. Extensive experiments are conducted to demonstrate the effectiveness and efficiency of the proposed method.

Slanted support window-based stereo matching for surface reconstruction of microscopic samples.

This work is to address the limitations of 2D Scanning Electron Microscopy (SEM) micrographs in providing 3D topographical information necessary for various types of analysis in biological and biomedical sciences as well as mechanical and material engineering by investigating modern stereo vision methodologies for 3D surface reconstruction of microscopic samples. To achieve this, micrograph pairs of the microscopic samples are acquired by utilizing an SEM equipped with motor controlled specimen stage capable of precise translational, rotational movements and tilting of the specimen stage. After pre-processing of the micrographs by SIFT feature detection/description followed by RANSAC for matching outlier removal and stereo rectification, a dense stereo matching methodology is utilized which takes advantage of slanted support window formulation for sub-pixel accuracy stereo matching of the input images. This results in a dense disparity map which is used to determine the true depth/elevation of individual surface points. This is a major improvement in comparison to previous matching methodologies which require additional post-processing refinement steps to reduce the negative effects of discrete disparity assignment or the blurring artifacts in near the edge regions. The provided results are great representatives of the superior performance of the slanted support window assumption employed here for surface reconstruction of microscopic samples.

Effective Clipart Image Vectorization through Direct Optimization of Bezigons.

Bezigons, i.e., closed paths composed of Bézier curves, have been widely employed to describe shapes in image vectorization results. However, most existing vectorization techniques infer the bezigons by simply approximating an intermediate vector representation (such as polygons). Consequently, the resultant bezigons are sometimes imperfect due to accumulated errors, fitting ambiguities, and a lack of curve priors, especially for low-resolution images. In this paper, we describe a novel method for vectorizing clipart images. In contrast to previous methods, we directly optimize the bezigons rather than using other intermediate representations; therefore, the resultant bezigons are not only of higher fidelity compared with the original raster image but also more reasonable because they were traced by a proficient expert. To enable such optimization, we have overcome several challenges and have devised a differentiable data energy as well as several curve-based prior terms. To improve the efficiency of the optimization, we also take advantage of the local control property of bezigons and adopt an overlapped piecewise optimization strategy. The experimental results show that our method outperforms both the current state-of-the-art method and commonly used commercial software in terms of bezigon quality.

Pattern Visualization of Human Connectome Data.

The human brain is a complex network with countless connected neurons, and can be described as a "connectome". Existing studies on analyzing human connectome data are primarily focused on characterizing the brain networks with a small number of easily computable measures that may be inadequate for revealing complex relationship between brain function and its structural substrate. To facilitate large-scale connectomic analysis, in this paper, we propose a powerful and flexible volume rendering scheme to effectively visualize and interactively explore thousands of network measures in the context of brain anatomy, and to aid pattern discovery. We demonstrate the effectiveness of the proposed scheme by applying it to a real connectome data set.