Learning to Zoom-In via Learning to Zoom-Out: Real-World Super-Resolution by Generating and Adapting Degradation.

Most learning-based super-resolution (SR) methods aim to recover high-resolution (HR) image from a given low-resolution (LR) image via learning on LR-HR image pairs. The SR methods learned on synthetic data do not perform well in real-world, due to the domain gap between the artificially synthesized and real LR images. Some efforts are thus taken to capture real-world image pairs. However, the captured LR-HR image pairs usually suffer from unavoidable misalignment, which hampers the performance of end- to-end learning. Here, focusing on the real-world SR, we ask a different question: since misalignment is unavoidable, can we propose a method that does not need LR-HR image pairing and alignment at all and utilizes real images as they are? Hence we propose a framework to learn SR from an arbitrary set of unpaired LR and HR images and see how far a step can go in such a realistic and "unsupervised" setting. To do so, we firstly train a degradation generation network to generate realistic LR images and, more importantly, to capture their distribution (i.e., learning to zoom out). Instead of assuming the domain gap has been eliminated, we minimize the discrepancy between the generated data and real data while learning a degradation adaptive SR network (i.e., learning to zoom in). The proposed unpaired method achieves state-of- the-art SR results on real-world images, even in the datasets that favour the paired-learning methods more.

COVID-19 Chest CT Image Segmentation Network by Multi-Scale Fusion and Enhancement Operations.

A novel coronavirus disease 2019 (COVID-19) was detected and has spread rapidly across various countries around the world since the end of the year 2019. Computed Tomography (CT) images have been used as a crucial alternative to the time-consuming RT-PCR test. However, pure manual segmentation of CT images faces a serious challenge with the increase of suspected cases, resulting in urgent requirements for accurate and automatic segmentation of COVID-19 infections. Unfortunately, since the imaging characteristics of the COVID-19 infection are diverse and similar to the backgrounds, existing medical image segmentation methods cannot achieve satisfactory performance. In this article, we try to establish a new deep convolutional neural network tailored for segmenting the chest CT images with COVID-19 infections. We first maintain a large and new chest CT image dataset consisting of 165,667 annotated chest CT images from 861 patients with confirmed COVID-19. Inspired by the observation that the boundary of the infected lung can be enhanced by adjusting the global intensity, in the proposed deep CNN, we introduce a feature variation block which adaptively adjusts the global properties of the features for segmenting COVID-19 infection. The proposed FV block can enhance the capability of feature representation effectively and adaptively for diverse cases. We fuse features at different scales by proposing Progressive Atrous Spatial Pyramid Pooling to handle the sophisticated infection areas with diverse appearance and shapes. The proposed method achieves state-of-the-art performance. Dice similarity coefficients are 0.987 and 0.726 for lung and COVID-19 segmentation, respectively. We conducted experiments on the data collected in China and Germany and show that the proposed deep CNN can produce impressive performance effectively. The proposed network enhances the segmentation ability of the COVID-19 infection, makes the connection with other techniques and contributes to the development of remedying COVID-19 infection.

Attention-Guided Deep Neural Network With Multi-Scale Feature Fusion for Liver Vessel Segmentation.

Liver vessel segmentation is fast becoming a key instrument in the diagnosis and surgical planning of liver diseases. In clinical practice, liver vessels are normally manual annotated by clinicians on each slice of CT images, which is extremely laborious. Several deep learning methods exist for liver vessel segmentation, however, promoting the performance of segmentation remains a major challenge due to the large variations and complex structure of liver vessels. Previous methods mainly using existing UNet architecture, but not all features of the encoder are useful for segmentation and some even cause interferences. To overcome this problem, we propose a novel deep neural network for liver vessel segmentation, called LVSNet, which employs special designs to obtain the accurate structure of the liver vessel. Specifically, we design Attention-Guided Concatenation (AGC) module to adaptively select the useful context features from low-level features guided by high-level features. The proposed AGC module focuses on capturing rich complemented information to obtain more details. In addition, we introduce an innovative multi-scale fusion block by constructing hierarchical residual-like connections within one single residual block, which is of great importance for effectively linking the local blood vessel fragments together. Furthermore, we construct a new dataset containing 40 thin thickness cases (0.625 mm) which consist of CT volumes and annotated vessels. To evaluate the effectiveness of the method with minor vessels, we also propose an automatic stratification method to split major and minor liver vessels. Extensive experimental results demonstrate that the proposed LVSNet outperforms previous methods on liver vessel segmentation datasets. Additionally, we conduct a series of ablation studies that comprehensively support the superiority of the underlying concepts.

Multi-way backpropagation for training compact deep neural networks.

Depth is one of the key factors behind the success of convolutional neural networks (CNNs). Since ResNet (He et al., 2016), we are able to train very deep CNNs as the gradient vanishing issue has been largely addressed by the introduction of skip connections. However, we observe that, when the depth is very large, the intermediate layers (especially shallow layers) may fail to receive sufficient supervision from the loss due to severe transformation through long backpropagation path. As a result, the representation power of intermediate layers can be very weak and the model becomes very redundant with limited performance. In this paper, we first investigate the supervision vanishing issue in existing backpropagation (BP) methods. And then, we propose to address it via an effective method, called Multi-way BP (MW-BP), which relies on multiple auxiliary losses added to the intermediate layers of the network. The proposed MW-BP method can be applied to most deep architectures with slight modifications, such as ResNet and MobileNet. Our method often gives rise to much more compact models (denoted by "Mw+Architecture") than existing methods. For example, MwResNet-44 with 44 layers performs better than ResNet-110 with 110 layers on CIFAR-10 and CIFAR-100. More critically, the resultant models even outperform the light models obtained by state-of-the-art model compression methods. Last, our method inherently produces multiple compact models with different depths at the same time, which is helpful for model selection. Extensive experiments on both image classification and face recognition demonstrate the superiority of the proposed method.

Deep HDR Imaging via A Non-local Network.

One of the most challenging problems in reconstructing a high dynamic range (HDR) image from multiple low dynamic range (LDR) inputs is the ghosting artifacts caused by the object motion across different inputs. When the object motion is slight, most existing methods can well suppress the ghosting artifacts through aligning LDR inputs based on optical flow or detecting anomalies among them. However, they often fail to produce satisfactory results in practice, since the real object motion can be very large. In this study, we present a novel deep framework, termed NHDRRnet, which adopts an alternative direction and attempts to remove ghosting artifacts by exploiting the non-local correlation in inputs. In NHDRRnet, we first adopt an Unet architecture to fuse all inputs and map the fusion results into a low-dimensional deep feature space. Then, we feed the resultant features into a novel global non-local module which reconstructs each pixel by weighted averaging all the other pixels using the weights determined by their correspondences. By doing this, the proposed NHDRRnet is able to adaptively select the useful information (e.g., which are not corrupted by large motions or adverse lighting conditions) in the whole deep feature space to accurately reconstruct each pixel. In addition, we also incorporate a triple-pass residual module to capture more powerful local features, which proves to be effective in further boosting the performance. Extensive experiments on three benchmark datasets demonstrate the superiority of the proposed NDHRnet in terms of suppressing the ghosting artifacts in HDR reconstruction, especially when the objects have large motions.

Learning Deep Gradient Descent Optimization for Image Deconvolution.

As an integral component of blind image deblurring, non-blind deconvolution removes image blur with a given blur kernel, which is essential but difficult due to the ill-posed nature of the inverse problem. The predominant approach is based on optimization subject to regularization functions that are either manually designed or learned from examples. Existing learning-based methods have shown superior restoration quality but are not practical enough due to their restricted and static model design. They solely focus on learning a prior and require to know the noise level for deconvolution. We address the gap between the optimization- and learning-based approaches by learning a universal gradient descent optimizer. We propose a recurrent gradient descent network (RGDN) by systematically incorporating deep neural networks into a fully parameterized gradient descent scheme. A hyperparameter-free update unit shared across steps is used to generate the updates from the current estimates based on a convolutional neural network. By training on diverse examples, the RGDN learns an implicit image prior and a universal update rule through recursive supervision. The learned optimizer can be repeatedly used to improve the quality of diverse degenerated observations. The proposed method possesses strong interpretability and high generalization. Extensive experiments on synthetic benchmarks and challenging real-world images demonstrate that the proposed deep optimization method is effective and robust to produce favorable results as well as practical for real-world image deblurring applications.

Accurate Tensor Completion via Adaptive Low-Rank Representation.

Low-rank representation-based approaches that assume low-rank tensors and exploit their low-rank structure with appropriate prior models have underpinned much of the recent progress in tensor completion. However, real tensor data only approximately comply with the low-rank requirement in most cases, viz., the tensor consists of low-rank (e.g., principle part) as well as non-low-rank (e.g., details) structures, which limit the completion accuracy of these approaches. To address this problem, we propose an adaptive low-rank representation model for tensor completion that represents low-rank and non-low-rank structures of a latent tensor separately in a Bayesian framework. Specifically, we reformulate the CANDECOMP/PARAFAC (CP) tensor rank and develop a sparsity-induced prior for the low-rank structure that can be used to determine tensor rank automatically. Then, the non-low-rank structure is modeled using a mixture of Gaussians prior that is shown to be sufficiently flexible and powerful to inform the completion process for a variety of real tensor data. With these two priors, we develop a Bayesian minimum mean-squared error estimate framework for inference. The developed framework can capture the important distinctions between low-rank and non-low-rank structures, thereby enabling more accurate model, and ultimately, completion. For various applications, compared with the state-of-the-art methods, the proposed model yields more accurate completion results.

Model-free tracker for multiple objects using joint appearance and motion inference.

Model-free tracking is a widely-accepted approach to track an arbitrary object in a video using a single frame annotation with no further prior knowledge about the object of interest. Extending this problem to track multiple objects is really challenging because: a) the tracker is not aware of the objects' type while trying to distinguish them from background (detection task), and b) The tracker needs to distinguish one object from other potentially similar objects (data association task) to generate stable trajectories. In order to track multiple arbitrary objects, most existing model-free tracking approaches rely on tracking each target individually by updating their appearance model independently. Therefore, in this scenario they often fail to perform well due to confusion between the appearance of similar objects, their sudden appearance changes and occlusion. To tackle this problem, we propose to use both appearance and motion models, and to learn them jointly using graphical models and deep neural networks features. We introduce an indicator variable to predict sudden appearance change and/or occlusion. When these happen, our model does not update the appearance model thus avoiding using the background and/or incorrect object to update the appearance of the object of interest mistakenly, and relies on our motion model to track. Moreover, we consider the correlation among all targets, and seek the joint optimal locations for all targets simultaneously as a graphical model inference problem. We learn the joint parameters for both appearance model and motion model in an online fashion under the framework of LaRank. Experiment results show that our method achieved superior performance compared to the competitive methods.

MPTV: Matching Pursuit-Based Total Variation Minimization for Image Deconvolution.

Total variation (TV) regularization has proven effective for a range of computer vision tasks through its preferential weighting of sharp image edges. Existing TV-based methods, however, often suffer from the over-smoothing issue and solution bias caused by the homogeneous penalization. In this paper, we consider addressing these issues by applying inhomogeneous regularization on different image components. We formulate the inhomogeneous TV minimization problem as a convex quadratic constrained linear programming problem. Relying on this new model, we propose a matching pursuit-based total variation minimization method (MPTV), specifically for image deconvolution. The proposed MPTV method is essentially a cutting-plane method that iteratively activates a subset of nonzero image gradients and then solves a subproblem focusing on those activated gradients only. Compared with existing methods, the MPTV is less sensitive to the choice of the trade-off parameter between data fitting and regularization. Moreover, the inhomogeneity of MPTV alleviates the over-smoothing and ringing artifacts and improves the robustness to errors in blur kernel. Extensive experiments on different tasks demonstrate the superiority of the proposed method over the current state of the art.

An Adaptive Markov Random Field for Structured Compressive Sensing.

Exploiting intrinsic structures in sparse signals underpins the recent progress in compressive sensing (CS). The key for exploiting such structures is to achieve two desirable properties: generality (i.e., the ability to fit a wide range of signals with diverse structures) and adaptability (i.e., being adaptive to a specific signal). Most existing approaches, however, often only achieve one of these two properties. In this study, we propose a novel adaptive Markov random field sparsity prior for CS, which not only is able to capture a broad range of sparsity structures, but also can adapt to each sparse signal through refining the parameters of the sparsity prior with respect to the compressed measurements. To maximize the adaptability, we also propose a new sparse signal estimation where the sparse signals, support, noise and signal parameter estimation are unified into a variational optimization problem, which can be effectively solved with an alternative minimization scheme. Extensive experiments on three real-world datasets demonstrate the effectiveness of the proposed method in recovery accuracy, noise tolerance, and runtime.

A Hybrid Loss for Multiclass and Structured Prediction.

We propose a novel hybrid loss for multiclass and structured prediction problems that is a convex combination of a log loss for Conditional Random Fields (CRFs) and a multiclass hinge loss for Support Vector Machines (SVMs). We provide a sufficient condition for when the hybrid loss is Fisher consistent for classification. This condition depends on a measure of dominance between labels-specifically, the gap between the probabilities of the best label and the second best label. We also prove Fisher consistency is necessary for parametric consistency when learning models such as CRFs. We demonstrate empirically that the hybrid loss typically performs least as well as-and often better than-both of its constituent losses on a variety of tasks, such as human action recognition. In doing so we also provide an empirical comparison of the efficacy of probabilistic and margin based approaches to multiclass and structured prediction.

Dual Graph Regularized Latent Low-Rank Representation for Subspace Clustering.

Low-rank representation (LRR) has received considerable attention in subspace segmentation due to its effectiveness in exploring low-dimensional subspace structures embedded in data. To preserve the intrinsic geometrical structure of data, a graph regularizer has been introduced into LRR framework for learning the locality and similarity information within data. However, it is often the case that not only the high-dimensional data reside on a non-linear low-dimensional manifold in the ambient space, but also their features lie on a manifold in feature space. In this paper, we propose a dual graph regularized LRR model (DGLRR) by enforcing preservation of geometric information in both the ambient space and the feature space. The proposed method aims for simultaneously considering the geometric structures of the data manifold and the feature manifold. Furthermore, we extend the DGLRR model to include non-negative constraint, leading to a parts-based representation of data. Experiments are conducted on several image data sets to demonstrate that the proposed method outperforms the state-of-the-art approaches in image clustering.

Integration of the full-length HPV16 genome in cervical cancer and Caski and Siha cell lines and the possible ways of HPV integration.

Integration of high-risk human papillomavirus (HPV) into the host genome is a key event for cervical carcinogenesis. Different methods have been used to explore the physical states of the HPV genome to reveal the mechanisms for malignant transformation of the infected cells. Consensus has been reached that, although variable portions of the HPV genome are deleted in the integrated HPV sequences, common disruption of the viral E2 gene has been demonstrated in different studies. The head-to-tail concatemers of the full-length HPV16 genome is another typical integration pattern of HPV16, typically found in Caski cell lines, but its prevalence in cervical cancer has never been tested. Here, by introducing a modified PCR, we identified this head-to-tail concatemers of full-length HPV genomes in advanced cervical cancer with HPV16 single positive. Our results show that more than half of the cases contain this integrated head-to-tail concatemers of full-length HPV16 genomes. Further studies in two cervical cell lines, Caski cells and Siha cells, revealed a correlation between the prevalence of the spliced variants of integrated HPV16 sequences and the full-length transcription of the integrated head-to-tail concatemers of the full-length HPV16 genome. Based on these results, we propose that HPV16 integrated into host cells by two mechanisms: one mechanism is shared by other DNA virus and cause integration of the head-to-tail concatemers of the viral genome; another is related to the reverse transcription process, which the integrated HPV sequence is generated by the reverse transcription of the viral mRNA.

Hashing on nonlinear manifolds.

Learning-based hashing methods have attracted considerable attention due to their ability to greatly increase the scale at which existing algorithms may operate. Most of these methods are designed to generate binary codes preserving the Euclidean similarity in the original space. Manifold learning techniques, in contrast, are better able to model the intrinsic structure embedded in the original high-dimensional data. The complexities of these models, and the problems with out-of-sample data, have previously rendered them unsuitable for application to large-scale embedding, however. In this paper, how to learn compact binary embeddings on their intrinsic manifolds is considered. In order to address the above-mentioned difficulties, an efficient, inductive solution to the out-of-sample data problem, and a process by which nonparametric manifold learning may be used as the basis of a hashing method are proposed. The proposed approach thus allows the development of a range of new hashing techniques exploiting the flexibility of the wide variety of manifold learning approaches available. It is particularly shown that hashing on the basis of t-distributed stochastic neighbor embedding outperforms state-of-the-art hashing methods on large-scale benchmark data sets, and is very effective for image classification with very short code lengths. It is shown that the proposed framework can be further improved, for example, by minimizing the quantization error with learned orthogonal rotations without much computation overhead. In addition, a supervised inductive manifold hashing framework is developed by incorporating the label information, which is shown to greatly advance the semantic retrieval performance.

RandomBoost: simplified multiclass boosting through randomization.

We propose a novel boosting approach to multiclass classification problems, in which multiple classes are distinguished by a set of random projection matrices in essence. The approach uses random projections to alleviate the proliferation of binary classifiers typically required to perform multiclass classification. The result is a multiclass classifier with a single vector-valued parameter, irrespective of the number of classes involved. Two variants of this approach are proposed. The first method randomly projects the original data into new spaces, while the second method randomly projects the outputs of learned weak classifiers. These methods are not only conceptually simple but also effective and easy to implement. A series of experiments on synthetic, machine learning, and visual recognition data sets demonstrate that our proposed methods could be compared favorably with existing multiclass boosting algorithms in terms of both the convergence rate and classification accuracy.

[Tumor infiltrating regulatory T cells in human breast cancer and associated draining lymph nodes: an in-situ analysis].

OBJECTIVE: To retrospectively analyze the quantity and status of the tumor infiltrating regulatory T lymphocytes in breast cancer and the draining lymph nodes, and to elucidate the clinical pathologic significance. METHODS: Seventy-four breast cancer samples with excised axillary lymph nodes were typed and staged histopathologically. The regulatory T lymphocytes were labeled by immunohistochemistry using EnVision method with the monoclonal antibodies against CD25 and Foxp3, and the immunophenotype was analyzed. In addition, the expression of IFN-γ, IL-10 and TGF-ß1 mRNA in lymphocytes of lymph nodes draining the tumors was detected by in situ hybridization with the corresponding specific oligo nucleaic acid probes. RESULTS: The number of CD25(+)Foxp3(+) T cells infiltrating the interstitium was much higher than that in the parenchymal tissue of the cancer. In the tumor draining lymph nodes, CD25(+) cells and Foxp3(+) cells were predominantly distributed in the paracortex with a proliferative pattern. TGF-ß1, INF-γ and IL-10 mRNA positive cells showed a similar distribution pattern in the draining lymph nodes. Among the 39 cases with metastatic disease, the lymph nodes with metastases showed a much higher number of CD25(+)Foxp3(+) cells than that without metastases (23.5 vs 17.3 and 23.8 vs 15.5; P < 0.05). However, there was no difference in the density of Foxp3(+)CD25(+) cells in the draining lymph nodes between the death and survival groups (P > 0.05). Cytokine expression of TGF-ß1, IL-10 and IFN-γ mRNA in the lymphocytes of draining lymph nodes in 24 cases showed that there were more IL-10 mRNA positive cells in the dead patients than that in the survived patients. A similar trend was observed for TGF-ß1 mRNA positive cells but the difference was not statistically significant (P > 0.05). The expression rate of TGF-ß1 and IL-10 mRNA in the draining lymph nodes was proportional to that of CD25(+) and Foxp3(+) cells (P < 0.05), and the expression of TGF-ß1 positive cells was also proportional to that of IL-10 mRNA positive cells (P < 0.01). The expression of IFN-γ mRNA among these groups showed no significance (P > 0.05). CONCLUSIONS: Regulatory T cells may play important roles in inhibiting the host antitumor immunity, and the presence of increased regulatory T cells and Th2-secreting cells in paracortex with a proliferative pattern in the tumor draining lymph nodes implies that the paracortical proliferation of draining lymph nodes may not reflect positive antitumor effects.

Development of a soluble PTD-HPV18E7 fusion protein and its functional characterization in eukaryotic cells.

Though accumulated evidence has demonstrated the transformation capacity of human papillomavirus (HPV) type 18 protein E7, the underlying mechanism is still arguable. Developing a protein transduction domain (PTD)-linked E7 molecule is a suitable strategy for assessing the biological functions of the protein. In the present study, HPV18 E7 protein fused to an N-terminal PTD was expressed in the form of glutathione S-transferase fusion protein in Escherichia coli with pGEX-4T- 3 vector. After glutathione-Sepharose 4B bead affinity purification, immunoblot identification and thrombin cleavage, the PTD-18E7 protein showed structural and functional activity in that it potently transduced the cells and localized into their nuclei. The PTD-18E7 protein transduced the NIH3T3 cells in 30 min and remained stable for at least 24 h. In addition, the PTD-18E7 protein interacted with retinoblastoma protein (pRB) and caused pRB degradation in the transduced NIH3T3 cells. In contrast to the pRB level, p27 protein level was elevated in the transduced NIH3T3 cells. The PTD-18E7 protein gives us a new tool to study the biological functions of the HPV E7 protein.

[Clinicopathological significance of cytotoxic lymphocytes in breast cancer and draining lymph nodes].

OBJECTIVE: To analyze retrospectively the quantity and activation status of the tumor infiltrating cytotoxic lymphocytes in breast cancer and the draining lymph nodes, and its relation to the clinical pathological significance. METHODS: Seventy-four breast cancer samples with their corresponding axillary lymph nodes were histologically typed and staged. Cytotxic lymphocytes were analyzed by immunohistochemistry with the monoclonal antibodies against CD8, CD56, granzyme B and perforin. RESULTS: The number of infiltrating CD8(+) T cells in the cancerous interstitial tissue were much higher than that in the tumor parenchyma. Compared with the metastatic tumor samples, the CD8(+) T cells were more intensive in the primary tumors (35.7 +/- 16.0 vs. 23.7 +/- 9.6). The tumor infiltrating CD8(+) T cells of patients with 5 years survivals were more than that of the dead cases in this follow-up series death (32.9 +/- 14.1 vs. 20.1 +/- 9.9). There was no significant difference of activated tumor infiltrating cytotoxic T cell analyzed by using the activation marker granzyme B(+) and there was also no significant correlation between the intensity of CD8(+), CD56(+) cells and the clinicopathological stages. However, percentages of the activated cytotoxic lymphocytes in Stage I groups were significantly higher than those in stage III and IV. Moreover, the number of perforin(+) cells was significantly less than that of granzyme B(+) cells, particularly in the cancerous tissue, indicating a dysfunctional status of tumor infiltrating cytotoxic lymphocytes. CONCLUSIONS: Activated cytotoxic lymphocytes may play a significant role against the tumor progression and is associated with a favorable prognosis to some extent. However, a putative dysfunctional status of cytotoxic lymphocytes at tumor site may compromise the host immunity against cancer.