Transformer based Pluralistic Image Completion with Reduced Information Loss.

Transformer based methods have achieved great success in image inpainting recently. However, we find that these solutions regard each pixel as a token, thus suffering from an information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration. 2) They quantize 2563 RGB values to a small number (such as 512) of quantized color values. The indices of quantized pixels are used as tokens for the inputs and prediction targets of the transformer. To mitigate these issues, we propose a new transformer based framework called "PUT". Specifically, to avoid input downsampling while maintaining computation efficiency, we design a patch-based auto-encoder P-VQVAE. The encoder converts the masked image into non-overlapped patch tokens and the decoder recovers the masked regions from the inpainted tokens while keeping the unmasked regions unchanged. To eliminate the information loss caused by input quantization, an Un-quantized Transformer is applied. It directly takes features from the P-VQVAE encoder as input without any quantization and only regards the quantized tokens as prediction targets.Furthermore, to make the inpainting process more controllable, we introduce semantic and structural conditions as extra guidance. Extensive experiments show that our method greatly outperforms existing transformer based methods on image fidelity and achieves much higher diversity and better fidelity than state-of-the-art pluralistic inpainting methods on complex large-scale datasets (e.g., ImageNet). Codes are available at https://github.com/liuqk3/PUT.

Exploring the Application of Large-Scale Pre-Trained Models on Adverse Weather Removal.

Image restoration under adverse weather conditions (e.g., rain, snow, and haze) is a fundamental computer vision problem that has important implications for various downstream applications. Distinct from early methods that are specially designed for specific types of weather, recent works tend to simultaneously remove various adverse weather effects based on either spatial feature representation learning or semantic information embedding. Inspired by various successful applications incorporating large-scale pre-trained models (e.g., CLIP), in this paper, we explore their potential benefits for leveraging large-scale pre-trained models in this task based on both spatial feature representation learning and semantic information embedding aspects: 1) spatial feature representation learning, we design a Spatially Adaptive Residual (SAR) encoder to adaptively extract degraded areas. To facilitate training of this model, we propose a Soft Residual Distillation (CLIP-SRD) strategy to transfer spatial knowledge from CLIP between clean and adverse weather images; 2) semantic information embedding, we propose a CLIP Weather Prior (CWP) embedding module to enable the network to adaptively respond to different weather conditions. This module integrates the sample-specific weather priors extracted by the CLIP image encoder with the distribution-specific information (as learned by a set of parameters) and embeds these elements using a cross-attention mechanism. Extensive experiments demonstrate that our proposed method can achieve state-of-the-art performance under various and severe adverse weather conditions. The code will be made available.

Causal Relationship Between Gut Microbiota and Aneurysm: A Mendelian Randomization Study.

Observational studies have suggested a possible relationship between gut microbiota (GM) and aneurysm development. However, the nature of this association remains unclear due to the inherent limitations of observational research, such as reverse causation and confounding factors. To address this knowledge deficit, this study aimed to investigate and establish a causal link between GM and aneurysm development.

Time-restricted eating with calorie restriction on weight loss and cardiometabolic risk: a systematic review and meta-analysis.

The effect of time-restricted eating (TRE) has been summarized in previous studies, but its benefits in combination with calorie restriction (CR) still need to be determined. The present meta-analysis aimed to evaluate the efficacy of TRE with CR on weight loss and cardiometabolic risk. PubMed, Embase, Cochrane Library, and gray literature databases were searched from inception to October 18, 2022, for potential randomized controlled trial (RCT) studies based on predefined inclusion and exclusion criteria. Body weight and other cardiometabolic risk factors were described as weighted mean difference (WMD) with a 95% confidence interval (CI). Eight RCTs involving 579 participants were enrolled in the present analysis. The pooled results showed that TRE with CR reduced the body weight, fat mass, and waist circumference significantly (WMD: -1.40, 95% CI: -1.81 to -1.00, and I2: 0%; WMD: -0.73, 95% CI: -1.39 to -0.07, and I2: 0%; WMD: -1.87, 95% CI: -3.47 to -0.26, and I2: 67.25%, respectively). However, compared with CR alone, TRE plus CR exhibited no significant benefit on the blood pressure, glucose profile, and lipid profile. Subgroup analysis suggested that early TRE is more effective in weight loss (WMD: -1.42, 95% CI: -1.84 to -1.01, and I2: 0%) and improving fat mass (WMD: -1.06, 95% CI: -1.91 to -0.22, and I2: 0%) than delayed or broader TRE when combined with CR. Although the combination of TRE and CR can effectively decrease body weight, fat mass, and waist circumference, the long-term effects, particularly those on cardiometabolic risk in participants with chronic cardiovascular disease and diabetes, remain to be explored.

Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis.

Semantic image synthesis, translating semantic layouts to photo-realistic images, is a one-to-many mapping problem. Though impressive progress has been recently made, diverse semantic synthesis that can efficiently produce semantic-level or even instance-level multimodal results, still remains a challenge. In this article, we propose a novel diverse semantic image synthesis framework from the perspective of semantic class distributions, which naturally supports diverse generation at both semantics and instance level. We achieve this by modeling class-level conditional modulation parameters as continuous probability distributions instead of discrete values, and sampling per-instance modulation parameters through instance-adaptive stochastic sampling that is consistent across the network. Moreover, we propose prior noise remapping, through linear perturbation parameters encoded from paired references, to facilitate supervised training and exemplar-based instance style control at test time. To further extend the user interaction function of the proposed method, we also introduce sketches into the network. In addition, specially designed generator modules, Progressive Growing Module and Multi-Scale Refinement Module, can be used as a general module to improve the performance of complex scene generation. Extensive experiments on multiple datasets show that our method can achieve superior diversity and comparable quality compared to state-of-the-art methods. Codes are available at https://github.com/tzt101/INADE.git.

Efficient Semantic Image Synthesis via Class-Adaptive Normalization.

Spatially-adaptive normalization (SPADE) is remarkably successful recently in conditional semantic image synthesis in T. Park et al. 2019 which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to prevent the semantic information from being washed away. Despite its impressive performance, a more thorough understanding of the advantages inside the box is still highly demanded to help reduce the significant computation and parameter overhead introduced by this novel structure. In this paper, from a return-on-investment point of view, we conduct an in-depth analysis of the effectiveness of this spatially-adaptive normalization and observe that its modulation parameters benefit more from semantic-awareness rather than spatial-adaptiveness, especially for high-resolution input masks. Inspired by this observation, we propose class-adaptive normalization (CLADE), a lightweight but equally-effective variant that is only adaptive to semantic class. In order to further improve spatial-adaptiveness, we introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE and propose a truly spatially-adaptive variant of CLADE, namely CLADE-ICPE. Through extensive experiments on multiple challenging datasets, we demonstrate that the proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE, but it is much more efficient with fewer extra parameters and lower computational cost. The code and pretrained models are available at https://github.com/tzt101/CLADE.git.