Bridging Global Context Interactions for High-Fidelity Pluralistic Image Completion.

We introduce PICFormer, a novel framework for Pluralistic Image Completion using a transFormer based architecture, that achieves both high quality and diversity at a much faster inference speed. Our key contribution is to introduce a code-shared codebook learning using a restrictive CNN on small and non-overlapping receptive fields (RFs) for the local visible token representation. This results in a compact yet expressive discrete representation, facilitating efficient modeling of global visible context relations by the transformer. Unlike the prevailing autoregressive approaches, we proposed to sample all tokens simultaneously, leading to more than 100× faster inference speed. To enhance appearance consistency between visible and generated regions, we further propose a novel attention-aware layer (AAL), designed to better exploit distantly related high-frequency features. Through extensive experiments, we demonstrate that the efficiently learns semantically-rich discrete codes, resulting in significantly improved image quality. Moreover, our diverse image completion framework surpasses state-of-the-art methods on multiple image completion datasets. The project page is available at https://chuanxiaz.com/picformer/.

Clinical phenotypes of adult-onset Behçet's syndrome: a comprehensive cross-sectional study in China.

Behçet's syndrome (BS) is a variant vasculitis that can involve multiple organs with inflammatory manifestations. This study aimed to provide a more comprehensive analysis of the clinical phenotypes and characteristics of BS patients. We enrolled 2792 BS patients referred from China nationwide to Huadong Hospital Affiliated to Fudan University from October 2012 to December 2022. Detailed assessments of demographic information, clinical manifestations, laboratory results, gastroscopy, and medical imaging were conducted. Cluster analysis was performed based on 13 variables to determine the clinical phenotypes, and each phenotype was characterized according to the features of BS patients. A total of 1834 BS patients were included, while 958 invalid patients were excluded. The median age at onset was 31 years (IQR, 24-40 years), and the median disease duration was 10 years (IQR, 5-15 years). Eight clusters were identified, including mucocutaneous (n = 655, 35.7%), gastrointestinal (n = 363, 19.8%), articular (n = 184, 10%), ocular (n = 223, 12.2%), cardiovascular (n = 119, 6.5%), neurological (n = 118, 6.4%), vascular (n = 114, 6.2%), and hematological phenotype (n = 58, 3.2%). Ocular (RR = 1.672 (95% CI, 1.327-2.106); P < 0.001), gastrointestinal (RR = = 1.194 (95% CI, 1.031-1.383); P = 0.018), cardiovascular (RR = = 2.582 (95% CI, 1.842-3.620); P < 0.001), and vascular (RR = = 2.288 (95% CI, 1.600-3.272); P < 0.001) involvement were more prevalent in male BS patients, while the hematological (RR = 0.528 (95% CI, 0.360-0.776); P = 0.001) involvement was more common among female patients. BS presents significant heterogeneity and gender differences. The eight phenotypes of BS patients we propose hold the potential to assist clinicians in devising more personalized treatment and follow-up strategies. Key Points • This cluster analysis divided adult-onset BS into eight clinical phenotypes. • BS demonstrates a high level of clinical heterogeneity and gender differences. • Hematologic phenotypes of BS present distinctive clinical characteristics.

PLK1-activating IFI16-STING-TBK1 pathway induces apoptosis of intestinal epithelial cells in patients with intestinal Behçet's syndrome.

Inflammatory signals from immunological cells may cause damage to intestinal epithelial cells (IECs), resulting in intestinal inflammation and tissue impairment. Interferon-γ-inducible protein 16 (IFI16) was reported to be involved in the pathogenesis of Behçet's syndrome (BS). This study aimed to investigate how inflammatory cytokines released by immunological cells and IFI16 participate in the pathogenesis of intestinal BS. RNA sequencing and real-time quantitative PCR (qPCR) showed that the positive regulation of tumor necrosis factor-α (TNF-α) production in peripheral blood mononuclear cells (PBMCs) of intestinal BS patients may be related to the upregulation of polo like kinase 1 (PLK1) in PBMCs (P = 0.012). The plasma TNF-α protein level in intestinal BS was significantly higher than in healthy controls (HCs; P = 0.009). PBMCs of intestinal BS patients and HCs were co-cultured with human normal IECs (NCM460) to explore the interaction between immunological cells and IECs. Using IFI16 knockdown, PBMC-NCM460 co-culture, TNF-α neutralizing monoclonal antibody (mAb), stimulator of interferon genes (STING) agonist 2'3'-cGAMP, and the PLK1 inhibitor SBE 13 HCL, we found that PLK1 promotes the secretion of TNF-α from PBMCs of intestinal BS patients, which causes overexpression of IFI16 and induces apoptosis of IECs via the STING-TBK1 pathway. The expressions of IFI16, TNF-α, cleaved caspase 3, phosphorylated STING (pSTING) and phosphorylated tank binding kinase 1 (pTBK1) in the intestinal ulcer tissue of BS patients were significantly higher than that of HCs (all P < 0.05). PLK1 in PBMCs of intestinal BS patients increased TNF-α secretion, inducing IEC apoptosis via activation of the IFI16-STING-TBK1 pathway. PLK1 and the IFI16-STING-TBK1 pathway may be new therapeutic targets for intestinal BS.

Clinical heterogeneity and five phenotypes identified in pediatric Behçet's syndrome: a cohort study from Shanghai Behçet's syndrome database.

OBJECTIVES: Behçet's syndrome (BS) is a rare disease of unknown etiology, with limited reports especially in pediatric BS. The clinical characteristics and phenotypes of pediatric BS as a highly heterogeneous variable vessel vasculitis were investigated in this study. METHODS: A cross-sectional study was conducted to compare clinical variables and descriptive characteristics of BS by age of onset and gender. Cluster analysis was then performed to identify the phenotypes of pediatric BS. RESULTS: A total of 2082 BS patients were included in this study, 1834 adults and 248 children. Compared with adult-onset BS, pediatric BS had a higher incidence of folliculitis [relative risks (RR) and 95% confidence interval (CI) 1.3 (1.0-1.5)], uveitis of the left eye [RR and 95% CI 2.3 (1.0-5.0)], intestinal ulcer complications [RR and 95% CI 2.1 (1.1-4.2)], pericarditis [RR and 95% CI 2.5 (1.0-6.2)], and psychiatric disorders [RR and 95% CI 2.8(1.0-7.9)], while the incidence of thrombocytopenia was lower [RR 0.2 (0.1-1.0)]. Among pediatric BS, females had more genital ulcers, while males were more likely to have skin lesions, panuveitis, vascular involvement, venous lesions, cardiac involvement, and aortic aneurysms. Cluster analysis classified pediatric BS into five clusters (C1-C5): C1 (n = 61, 24.6%) showed gastrointestinal (GI) involvement; C2 (n = 44, 17.7%) was the central nervous system (CNS) type where 23 cases overlapped joint involvement; in C3 (n = 35, 14.1%), all patients presented with arthritis or arthralgia; all patients in C4 (n = 29, 11.7%) manifested ocular involvement, with a few patients overlapping with GI involvement or joint damage; C5 (n = 79, 31.9%) was the mucocutaneous type, presenting both oral ulcers, genital ulcers, and skin lesions. CONCLUSIONS: The clinical features of pediatric and adult BS differ significantly. Male and female pediatric BS also have a distinct demography. Five phenotypes including GI, CNS, joint, ocular, and mucocutaneous types were identified for pediatric BS.

Pruning Self-Attentions Into Convolutional Layers in Single Path.

Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. However, modeling global correlations with multi-head self-attention (MSA) layers leads to two widely recognized issues: the massive computational resource consumption and the lack of intrinsic inductive bias for modeling local visual patterns. To solve both issues, we devise a simple yet effective method named Single-Path Vision Transformer pruning (SPViT), to efficiently and automatically compress the pre-trained ViTs into compact models with proper locality added. Specifically, we first propose a novel weight-sharing scheme between MSA and convolutional operations, delivering a single-path space to encode all candidate operations. In this way, we cast the operation search problem as finding which subset of parameters to use in each MSA layer, which significantly reduces the computational cost and optimization difficulty, and the convolution kernels can be well initialized using pre-trained MSA parameters. Relying on the single-path space, we introduce learnable binary gates to encode the operation choices in MSA layers. Similarly, we further employ learnable gates to encode the fine-grained MLP expansion ratios of FFN layers. In this way, our SPViT optimizes the learnable gates to automatically explore from a vast and unified search space and flexibly adjust the MSA-FFN pruning proportions for each individual dense model. We conduct extensive experiments on two representative ViTs showing that our SPViT achieves a new SOTA for pruning on ImageNet-1 k. For example, our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.

FastCAT: A framework for fast routing table calculation incorporating multiple protocols.

Currently, most network outages occur because of manual configuration errors. Therefore, it is essential to verify the correctness of network configurations before deployment. Computing the network control plane is a key technology for network configuration verification. We can verify the correctness of network configurations for fault tolerance by generating routing tables, as well as connectivity. However, existing routing table calculation tools have disadvantages such as lack of user-friendliness, limited expressiveness, and slower speed of routing table generation. In this paper, we present FastCAT, a framework for computing routing tables incorporating multiple protocols. FastCAT can simulate the interaction of multiple routing protocols and quickly generate routing tables based on configuration files and topology information. The key to FastCAT's performance is that FastCAT focuses only on the final stable state of the OSPF and IS-IS protocols, disregarding the transient states during protocol convergence. For RIPv2 and BGP, FastCAT computes the current protocol routing tables based on the protocol's previous state, retaining only the most recent protocol routing tables in the latest state. Experimental evaluations have shown that FastCAT generates routing tables more quickly and accurately than the state-of-the-art routing simulation tool, in a general network of around 200 routers.

End-to-End One-Shot Human Parsing.

Previous human parsing methods are limited to parsing humans into pre-defined classes, which is inflexible for practical fashion applications that often have new fashion item classes. In this paper, we define a novel one-shot human parsing (OSHP) task that requires parsing humans into an open set of classes defined by any test example. During training, only base classes are exposed, which only overlap with part of the test-time classes. To address three main challenges in OSHP, i.e., small sizes, testing bias, and similar parts, we devise an End-to-end One-shot human Parsing Network (EOP-Net). Firstly, an end-to-end human parsing framework is proposed to parse the query image into both coarse-grained and fine-grained human classes, which embeds rich semantic information that is shared across different granularities to identify the small-sized human classes. Then, we gradually smooth the training-time static prototypes to get robust class representations. Moreover, we employ a dynamic objective to encourage the network enhancing features' representational capability in the early training phase while improving features' transferability in the late training phase. Therefore, our method can quickly adapt to the novel classes and mitigate the testing bias issue. In addition, we add a contrastive loss at the prototype level to enforce inter-class distances, thereby discriminating the similar parts. For comprehensive evaluations on the new task, we tailor three existing popular human parsing benchmarks to the OSHP task. Experiments demonstrate that EOP-Net outperforms representative one-shot segmentation models by large margins and serves as a strong baseline for further research.

Unifying Flow, Stereo and Depth Estimation.

We present a unified formulation and model for three motion and 3D perception tasks: optical flow, rectified stereo matching and unrectified stereo depth estimation from posed images. Unlike previous specialized architectures for each specific task, we formulate all three tasks as a unified dense correspondence matching problem, which can be solved with a single model by directly comparing feature similarities. Such a formulation calls for discriminative feature representations, which we achieve using a Transformer, in particular the cross-attention mechanism. We demonstrate that cross-attention enables integration of knowledge from another image via cross-view interactions, which greatly improves the quality of the extracted features. Our unified model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks. We outperform RAFT with our unified model on the challenging Sintel dataset, and our final model that uses a few additional task-specific refinement steps outperforms or compares favorably to recent state-of-the-art methods on 10 popular flow, stereo and depth datasets, while being simpler and more efficient in terms of model design and inference speed.

Facial Action Unit Detection via Adaptive Attention and Relation.

Facial action unit (AU) detection is challenging due to the difficulty in capturing correlated information from subtle and dynamic AUs. Existing methods often resort to the localization of correlated regions of AUs, in which predefining local AU attentions by correlated facial landmarks often discards essential parts, or learning global attention maps often contains irrelevant areas. Furthermore, existing relational reasoning methods often employ common patterns for all AUs while ignoring the specific way of each AU. To tackle these limitations, we propose a novel adaptive attention and relation (AAR) framework for facial AU detection. Specifically, we propose an adaptive attention regression network to regress the global attention map of each AU under the constraint of attention predefinition and the guidance of AU detection, which is beneficial for capturing both specified dependencies by landmarks in strongly correlated regions and facial globally distributed dependencies in weakly correlated regions. Moreover, considering the diversity and dynamics of AUs, we propose an adaptive spatio-temporal graph convolutional network to simultaneously reason the independent pattern of each AU, the inter-dependencies among AUs, as well as the temporal dependencies. Extensive experiments show that our approach (i) achieves competitive performance on challenging benchmarks including BP4D, DISFA, and GFT in constrained scenarios and Aff-Wild2 in unconstrained scenarios, and (ii) can precisely learn the regional correlation distribution of each AU.

Single-Path Bit Sharing for Automatic Loss-Aware Model Compression.

Network pruning and quantization are proven to be effective ways for deep model compression. To obtain a highly compact model, most methods first perform network pruning and then conduct quantization based on the pruned model. However, this strategy may ignore that the pruning and quantization would affect each other and thus performing them separately may lead to sub-optimal performance. To address this, performing pruning and quantization jointly is essential. Nevertheless, how to make a trade-off between pruning and quantization is non-trivial. Moreover, existing compression methods often rely on some pre-defined compression configurations (i.e., pruning rates or bitwidths). Some attempts have been made to search for optimal configurations, which however may take unbearable optimization cost. To address these issues, we devise a simple yet effective method named Single-path Bit Sharing (SBS) for automatic loss-aware model compression. To this end, we consider the network pruning as a special case of quantization and provide a unified view for model pruning and quantization. We then introduce a single-path model to encode all candidate compression configurations, where a high bitwidth value will be decomposed into the sum of a lowest bitwidth value and a series of re-assignment offsets. Relying on the single-path model, we introduce learnable binary gates to encode the choice of configurations and learn the binary gates and model parameters jointly. More importantly, the configuration search problem can be transformed into a subset selection problem, which helps to significantly reduce the optimization difficulty and computation cost. In this way, the compression configurations of each layer and the trade-off between pruning and quantization can be automatically determined. Extensive experiments on CIFAR-100 and ImageNet show that SBS significantly reduces computation cost while achieving promising performance. For example, our SBS compressed MobileNetV2 achieves 22.6× Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.

PCSK9 promotes T helper 1 and T helper 17 cell differentiation by activating the nuclear factor-κB pathway in ankylosing spondylitis.

OBJECTIVE: Our previous study reveals that proprotein convertase subtilisin/kexin type 9 (PCSK9) is positively related to inflammatory markers, T helper (Th)-17 cells, and treatment response in ankylosing spondylitis (AS) patients. Subsequently, this study aimed to explore the effect of PCSK9 on Th cell differentiation and its potential molecular mechanism in AS. METHODS: Serum PCSK9 was determined by enzyme-linked immunosorbent assay in 20 AS patients and 20 healthy controls (HCs). Then naïve CD4+ T cells were isolated from AS patients and infected with PCSK9 overexpression or knockdown adenovirus followed by polarization assay. Afterward, PMA (an NF-κB activator) was administrated. RESULTS: PCSK9 was increased in AS patients compared to HCs (p < .001), and it was positively related to Th1 cells (p = .050) and Th17 cells (p = .039) in AS patients. PCSK9 overexpression increased the CD4+ IFN-γ+ cells (p < .05), CD4+ IL-17A+ cells (p < .01), IFN-γ (p < .01), and IL-17A (p < .01), while it exhibited no effect on CD4+ IL-4+ cells or IL-4 (both p > .05); its knockdown displayed the opposite function on them. Moreover, PCSK9 overexpression upregulated the p-NF-κB p65/NF-κB p65 (p < .01), while it had no effect on p-ERK/ERK or p-JNK/JNK (both p > .05); its knockdown decreased p-NF-κB p65/NF-κB p65 (p < .01) and p-JNK/JNK (p < .05). Then, PMA upregulates p-NF-κB p65/NF-κB p65 (p < .001) and increased CD4+ IFN-γ+ cells, CD4+ IL-17A+ cells, IFN-γ, and IL-17A (all p < .01), also it alleviated the effect of PCSK9 knockdown on NF-κB inhibition and Th cell differentiation (all p < .01). CONCLUSION: PCSK9 enhances Th1 and Th17 cell differentiation in an NF-κB-dependent manner in AS, while further validation is necessary.

Comprehensive analysis of T-cell receptor repertoires reveals antigen-driven T-cell clusters in patients with Behçet's syndrome.

T lymphocytes are the major components of adaptive immunity in Behçet's syndrome (BS) pathology. However, the precise mechanism of T-cell-induced inflammatory condition remains to be determined. We applied bulk sequencing of the T-cell receptor (TCR) ß chain in peripheral blood samples from 45 patients with BS and 10 healthy donors as controls. TCR repertoires in BS patients displayed more clonality and less diversity than in healthy donors. Male patients exhibited lower diversity metrics of TCR and had a larger proportion in the top 10 clones than females (p = 0.016). There were no TCR clonality differences in other clinical features, such as age, disease duration, organ involvement, disease severity, and activity. By "Grouping of Lymphocyte Interactions by Paratope Hotspots" (GLIPH2) for antigen prediction, we found distinct 2477 clusters of TCR-ß sequences that potentially recognize similar antigens shared between BS patients. We observed clonal T-cell expansion in BS patients. Sexual differences in TCR clonal expansion and public TCR groups deserve further study to reveal the underline T-cell-mediated immunity in BS.

Deconfounded Image Captioning: A Causal Retrospect.

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

Predictors of infliximab refractory intestinal Behçet's syndrome: A retrospective cohort study from the Shanghai Behçet's syndrome database.

OBJECTIVES: This retrospective cohort study aimed to find out predictors and early biomarkers of Infliximab (IFX) refractory intestinal Behçet's syndrome (intestinal BS). METHODS: We collected the baseline clinical characteristics, laboratory parameters, and concomitant therapies of intestinal BS patients treated by IFX from the Shanghai Behçet's syndrome database. After 1 year IFX therapy, intestinal BS patients with non-mucosal healing (NMH, intestinal ulcers detected by colonoscopy) and/or no clinical remission [NCR, scores of the disease activity index for intestinal Behçet's disease (DAIBD) ≥20] were defined as IFX refractory intestinal BS. Multivariate logistic regression analysis was performed to evaluate the predictors for NMH and NCR in IFX refractory intestinal BS. RESULTS: In 85 intestinal BS patients, NMH was identified in 29 (34.12%) patients, and NCR was confirmed in 20 (23.53%) patients. Erythrocyte sedimentation rate (ESR; ≥24 mm/h) and free triiodothyronine (fT3; ≤3.3pmol/L) were the independent risk factors of NMH in IFX refractory intestinal BS. Drinking alcohol and the fT3/free thyroxine ratio (fT3/fT4; ≤0.24) were independent risk factors, and thalidomide was an independent protective factor, for NCR in intestinal BS patients treated by IFX. CONCLUSION: This study may be applicable for adjusting the therapeutic strategy and sidestepping unnecessary exposure to IFX in intestinal BS patients. Routine assessments of ESR, fT3, and fT3/fT4 ratio are helpful to identify high-risk individuals of IFX refractory intestinal BS. Thalidomide is suggested to be a concomitant therapy with IFX for intestinal BS patients.

Serum PCSK9 is positively correlated with disease activity and Th17 cells, while its short-term decline during treatment reflects desirable outcomes in ankylosing spondylitis patients.

OBJECTIVE: Proprotein convertase subtilisin/kexin type 9 (PCSK9) participates in the autoimmune disease pathology by regulating T helper (Th) cell differentiation, NF-κB pathway, toll-like receptor 4, etc. This study intended to investigate the association of serum PCSK9 with disease activity, Th cells, and treatment response in ankylosing spondylitis (AS) patients. METHODS: Eighty-nine active AS patients were enrolled in this multicenter, prospective study. Serum was collected from AS patients at week (W)0, W4, W8, and W12, as well as from 20 osteoarthritis patients and 20 healthy controls after enrollment to detect PCSK9 by ELISA. Based on the ASAS40 response at W12, AS patients were classified as responders and non-responders. RESULTS: PCSK9 was increased in AS patients versus healthy controls (P < 0.001) and osteoarthritis patients (P = 0.006). In AS patients, PCSK9 was positively linked with C-reactive protein (CRP) (P = 0.003) and ASDAS-CRP (P = 0.017), but not with other clinical properties (P > 0.05). Besides, PCSK9 was negatively correlated with interleukin-4 (P = 0.034), positively associated with Th17 cells (P = 0.005) and interleukin-17A (P = 0.014), but did not relate to Th1 cells, interferon-γ, or Th2 cells (all P > 0.05). Additionally, PCSK9 was decreased from W0 to W12 in general AS patients (P < 0.001) and responders (P < 0.001) but remained unchanged in non-responders (P = 0.129). Moreover, PCSK9 was lower at W4 (P = 0.045), W8 (P = 0.008), and W12 (P = 0.004) in responders versus non-responders. Furthermore, the treatment options did not affect the PCSK9 level (P > 0.05). CONCLUSION: Serum PCSK9 is positively associated with disease activity and Th17 cells, while its short-term decline reflects desirable treatment response in AS patients.

Tofacitinib as an alternative therapy for refractory intestinal Behçet's syndrome.

Background: Intestinal Behçet's syndrome is a major cause of morbidity and mortality in Behçet's syndrome. Objectives: Current treatment challenges remain in refractory intestinal Behçet's syndrome, when patients failed first and second-line therapies. Design: We reported the efficacy and safety profiles of tofacitinib in patients with moderate-severe intestinal Behçet's syndrome in a retrospective single-center study. Methods: Treatment with glucocorticoids, immunosuppressors, or even anti-TNFα monoclonal antibodies (mAbs) had previously failed. Primary outcomes were clinical remission or low disease activity and endoscopic healing. Results: We included 13 patients; 11 were administered tofacitinib 5 mg twice daily, and 2 took tofacitinib 5 mg once daily. Nine patients achieved clinical remission after a mean treatment duration of 10.1 ± 7.0 months, and the other four had low disease activity. Follow-up endoscopy was available in 11 patients: 5 had achieved mucosal healing; the other 4 achieved marked mucosal improvement. Prednisone dosage was significantly reduced, from 30 (interquartile range: 20-30) mg/d to 2.5 (interquartile range: 0-12.5) mg/d (p < 0.001). No serious adverse event was observed. Conclusion: Tofacitinib could be an efficacious and generally well-tolerated option in patients with intestinal Behçet's syndrome refractory to conventional agents, even anti-TNFα mAbs.

Mutual consistency learning for semi-supervised medical image segmentation.

In this paper, we propose a novel mutual consistency network (MC-Net+) to effectively exploit the unlabeled data for semi-supervised medical image segmentation. The MC-Net+ model is motivated by the observation that deep models trained with limited annotations are prone to output highly uncertain and easily mis-classified predictions in the ambiguous regions (e.g., adhesive edges or thin branches) for medical image segmentation. Leveraging these challenging samples can make the semi-supervised segmentation model training more effective. Therefore, our proposed MC-Net+ model consists of two new designs. First, the model contains one shared encoder and multiple slightly different decoders (i.e., using different up-sampling strategies). The statistical discrepancy of multiple decoders' outputs is computed to denote the model's uncertainty, which indicates the unlabeled hard regions. Second, we apply a novel mutual consistency constraint between one decoder's probability output and other decoders' soft pseudo labels. In this way, we minimize the discrepancy of multiple outputs (i.e., the model uncertainty) during training and force the model to generate invariant results in such challenging regions, aiming at regularizing the model training. We compared the segmentation results of our MC-Net+ model with five state-of-the-art semi-supervised approaches on three public medical datasets. Extension experiments with two standard semi-supervised settings demonstrate the superior performance of our model over other methods, which sets a new state of the art for semi-supervised medical image segmentation. Our code is released publicly at https://github.com/ycwu1997/MC-Net.

Auto-Encoding and Distilling Scene Graphs for Image Captioning.

We propose scene graph auto-encoder (SGAE) that incorporates the language inductive bias into the encoder-decoder image captioning framework for more human-like captions. Intuitively, we humans use the inductive bias to compose collocations and contextual inferences in discourse. For example, when we see the relation "a person on a bike", it is natural to replace "on" with "ride" and infer "a person riding a bike on a road" even when the "road" is not evident. Therefore, exploiting such bias as a language prior is expected to help the conventional encoder-decoder models reason as we humans and generate more descriptive captions. Specifically, we use the scene graph-a directed graph ( G) where an object node is connected by adjective nodes and relationship nodes-to represent the complex structural layout of both image ( I) and sentence ( S). In the language domain, we use SGAE to learn a dictionary set ( D) that helps reconstruct sentences in the S→ GS → D → S auto-encoding pipeline, where D encodes the desired language prior and the decoder learns to caption from such a prior; in the vision-language domain, we share D in the I→ GI → D → S pipeline and distill the knowledge of the language decoder of the auto-encoder to that of the encoder-decoder based image captioner to transfer the language inductive bias. In this way, the shared D provides hidden embeddings about descriptive collocations to the encoder-decoder and the distillation strategy teaches the encoder-decoder to transform these embeddings to human-like captions as the auto-encoder. Thanks to the scene graph representation, the shared dictionary set, and the Knowledge Distillation strategy, the inductive bias is transferred across domains in principle. We validate the effectiveness of SGAE on the challenging MS-COCO image captioning benchmark, where our SGAE-based single-model achieves a new state-of-the-art 129.6 CIDEr-D on the Karpathy split, and a competitive 126.6 CIDEr-D (c40) on the official server, which is even comparable to other ensemble models. Furthermore, we validate the transferability of SGAE on two more challenging settings: transferring inductive bias from other language corpora and unpaired image captioning. Once again, the results of both settings confirm the superiority of SGAE. The code is released in https://github.com/yangxuntu/SGAE.

Scalable Gaussian Process Classification With Additive Noise for Non-Gaussian Likelihoods.

Gaussian process classification (GPC) provides a flexible and powerful statistical framework describing joint distributions over function space. Conventional GPCs, however, suffer from: 1) poor scalability for big data due to the full kernel matrix and 2) intractable inference due to the non-Gaussian likelihoods. Hence, various scalable GPCs have been proposed through: 1) the sparse approximation built upon a small inducing set to reduce the time complexity and 2) the approximate inference to derive analytical evidence lower bound (ELBO). However, these scalable GPCs equipped with analytical ELBO are limited to specific likelihoods or additional assumptions. In this work, we present a unifying framework that accommodates scalable GPCs using various likelihoods. Analogous to GP regression (GPR), we introduce additive noises to augment the probability space for: 1) the GPCs with step, (multinomial) probit, and logit likelihoods via the internal variables and 2) particularly, the GPC using softmax likelihood via the noise variables themselves. This leads to unified scalable GPCs with analytical ELBO by using variational inference. Empirically, our GPCs showcase superiority on extensive binary/multiclass classification tasks with up to two million data points.

Image Co-Skeletonization via Co-Segmentation.

Recent advances in the joint processing of a set of images have shown its advantages over individual processing. Unlike the existing works geared towards co-segmentation or co-localization, in this article, we explore a new joint processing topic: image co-skeletonization, which is defined as joint skeleton extraction of the foreground objects in an image collection. It is well known that object skeletonization in a single natural image is challenging, because there is hardly any prior knowledge available about the object present in the image. Therefore, we resort to the idea of image co-skeletonization, hoping that the commonness prior that exists across the semantically similar images can be leveraged to have such knowledge, similar to other joint processing problems such as co-segmentation. Moreover, earlier research has found that augmenting a skeletonization process with the object's shape information is highly beneficial in capturing the image context. Having made these two observations, we propose a coupled framework for co-skeletonization and co-segmentation tasks to facilitate shape information discovery for our co-skeletonization process through the co-segmentation process. While image co-skeletonization is our primary goal, the co-segmentation process might also benefit, in turn, from exploiting skeleton outputs of the co-skeletonization process as central object seeds through such a coupled framework. As a result, both can benefit from each other synergistically. For evaluating image co-skeletonization results, we also construct a novel benchmark dataset by annotating nearly 1.8 K images and dividing them into 38 semantic categories. Although the proposed idea is essentially a weakly supervised method, it can also be employed in supervised and unsupervised scenarios. Extensive experiments demonstrate that the proposed method achieves promising results in all three scenarios.