SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses.

Neural Radiance Fields (NeRFs) have shown impressive capabilities in synthesizing photorealistic novel views. However, their application to room-size scenes is limited by the requirement of several hundred views with accurate poses for training. To address this challenge, we propose SN 2 eRF, a framework which can reconstruct the neural radiance field with significantly fewer views and noisy poses by exploiting multiple priors. Our key insight is to leverage both multi-view and monocular priors to constrain the optimization of NeRF in the setting of sparse and noisy pose inputs. Specifically, we extract and match key points to constrain pose optimization and use Ray Transformer with a monocular depth estimator to provide dense depth prior for geometry optimization. Benefiting from these priors, our approach achieves state-of-the-art accuracy in novel view synthesis for indoor room scenarios.

SLS4D: Sparse Latent Space for 4D Novel View Synthesis.

Neural radiance fields (NeRF) have achieved great success in novel view synthesis and 3D representation for static scenarios. Existing dynamic NeRFs usually exploit a locally dense grid to fit the deformation fields; however, they fail to capture the global dynamics and concomitantly yield models of heavy parameters. We observe that the 4D space is inherently sparse. Firstly, the deformation fields are sparse in spatial but dense in temporal due to the continuity of motion. Secondly, the radiance fields are only valid on the surface of the underlying scene, usually occupying a small fraction of the whole space. We thus represent the 4D scene using a learnable sparse latent space, a.k.a. SLS4D. Specifically, SLS4D first uses dense learnable time slot features to depict the temporal space, from which the deformation fields are fitted with linear multi-layer perceptions (MLP) to predict the displacement of a 3D position at any time. It then learns the spatial features of a 3D position using another sparse latent space. This is achieved by learning the adaptive weights of each latent feature with the attention mechanism. Extensive experiments demonstrate the effectiveness of our SLS4D: It achieves the best 4D novel view synthesis using only about 6% parameters of the most recent work.

Learning Virtual View Selection for 3D Scene Semantic Segmentation.

2D-3D joint learning is essential and effective for fundamental 3D vision tasks, such as 3D semantic segmentation, due to the complementary information these two visual modalities contain. Most current 3D scene semantic segmentation methods process 2D images "as they are", i.e., only real captured 2D images are used. However, such captured 2D images may be redundant, with abundant occlusion and/or limited field of view (FoV), leading to poor performance for the current methods involving 2D inputs. In this paper, we propose a general learning framework for joint 2D-3D scene understanding by selecting informative virtual 2D views of the underlying 3D scene. We then feed both the 3D geometry and the generated virtual 2D views into any joint 2D-3D-input or pure 3D-input based deep neural models for improving 3D scene understanding. Specifically, we generate virtual 2D views based on an information score map learned from the current 3D scene semantic segmentation results. To achieve this, we formalize the learning of the information score map as a deep reinforcement learning process, which rewards good predictions using a deep neural network. To obtain a compact set of virtual 2D views that jointly cover informative surfaces of the 3D scene as much as possible, we further propose an efficient greedy virtual view coverage strategy in the normal-sensitive 6D space, including 3-dimensional point coordinates and 3-dimensional normal. We have validated our proposed framework for various joint 2D-3D-input or pure 3D-input based deep neural models on two real-world 3D scene datasets, i.e., ScanNet v2 and S3DIS, and the results demonstrate that our method obtains a consistent gain over baseline models and achieves new top accuracy for joint 2D and 3D scene semantic segmentation. Code is available at https://github.com/smy-THU/VirtualViewSelection.

GP-Recon: Online Monocular Neural 3D Reconstruction with Geometric Prior.

High-fidelity online 3D scene reconstruction from monocular videos continues to be challenging, especially for coherent and fine-grained geometry reconstruction. The previous learning-based online 3D reconstruction approaches with neural implicit representations have shown a promising ability for coherent scene reconstruction, but often fail to consistently reconstruct fine-grained geometric details during online reconstruction. This paper presents a new on-the-fly monocular 3D reconstruction approach, named GP-Recon, to perform high-fidelity online neural 3D reconstruction with fine-grained geometric details. We incorporate geometric prior (GP) into a scene's neural geometry learning to better capture its geometric details and, more importantly, propose an online volume rendering optimization to reconstruct and maintain geometric details during the online reconstruction task. The extensive comparisons with state-of-the-art approaches show that our GP-Recon consistently generates more accurate and complete reconstruction results with much better fine-grained details, both quantitatively and qualitatively.

SPS: Accurate and Real-Time Semantic Positioning System Based on Low-Cost DEM Maps.

This paper presents a Semantic Positioning System (SPS) to enhance the accuracy of mobile device geo-localization in outdoor urban environments. Although the traditional Global Positioning System (GPS) can offer a rough localization, it lacks the necessary accuracy for applications such as Augmented Reality (AR). Our SPS integrates Geographic Information System (GIS) data, GPS signals, and visual image information to estimate the 6 Degree-of-Freedom (DoF) pose through cross-view semantic matching. This approach has excellent scalability to support GIS context with Levels of Detail (LOD). The map data representation is Digital Elevation Model (DEM), a cost-effective aerial map that allows for fast deployment for large-scale areas. However, the DEM lacks geometric and texture details, making it challenging for traditional visual feature extraction to establish pixel/voxel level cross-view correspondences. To address this, we sample observation pixels from the query ground-view image using predicted semantic labels. We then propose an iterative homography estimation method with semantic correspondences. To improve the efficiency of the overall system, we further employ a heuristic search to speedup the matching process. The proposed method is robust, real-time, and automatic. Quantitative experiments on the challenging Bund dataset show that we achieve a positioning accuracy of 73.24%, surpassing the baseline skyline-based method by 20%. Compared with the state-of-the-art semantic-based approach on the Kitti dataset, we improve the positioning accuracy by an average of 5%.

Sampling Equivariant Self-Attention Networks for Object Detection in Aerial Images.

Objects in aerial images show greater variations in scale and orientation than in other images, making them harder to detect using vanilla deep convolutional neural networks. Networks with sampling equivariance can adapt sampling from input feature maps to object transformation, allowing a convolutional kernel to extract effective object features under different transformations. However, methods such as deformable convolutional networks can only provide sampling equivariance under certain circumstances, as they sample by location. We propose sampling equivariant self-attention networks, which treat self-attention restricted to a local image patch as convolution sampling by masks instead of locations, and a transformation embedding module to improve the equivariant sampling further. We further propose a novel randomized normalization module to enhance network generalization and a quantitative evaluation metric to fairly evaluate the ability of sampling equivariance of different models. Experiments show that our model provides significantly better sampling equivariance than existing methods without additional supervision and can thus extract more effective image features. Our model achieves state-of-the-art results on the DOTA-v1.0, DOTA-v1.5, and HRSC2016 datasets without additional computations or parameters.

Skeleton-CutMix: Mixing Up Skeleton With Probabilistic Bone Exchange for Supervised Domain Adaptation.

We present Skeleton-CutMix, a simple and effective skeleton augmentation framework for supervised domain adaptation and show its advantage in skeleton-based action recognition tasks. Existing approaches usually perform domain adaptation for action recognition with elaborate loss functions that aim to achieve domain alignment. However, they fail to capture the intrinsic characteristics of skeleton representation. Benefiting from the well-defined correspondence between bones of a pair of skeletons, we instead mitigate domain shift by fabricating skeleton data in a mixed domain, which mixes up bones from the source domain and the target domain. The fabricated skeletons in the mixed domain can be used to augment training data and train a more general and robust model for action recognition. Specifically, we hallucinate new skeletons by using pairs of skeletons from the source and target domains; a new skeleton is generated by exchanging some bones from the skeleton in the source domain with corresponding bones from the skeleton in the target domain, which resembles a cut-and-mix operation. When exchanging bones from different domains, we introduce a class-specific bone sampling strategy so that bones that are more important for an action class are exchanged with higher probability when generating augmentation samples for that class. We show experimentally that the simple bone exchange strategy for augmentation is efficient and effective and that distinctive motion features are preserved while mixing both action and style across domains. We validate our method in cross-dataset and cross-age settings on NTU-60 and ETRI-Activity3D datasets with an average gain of over 3% in terms of action recognition accuracy, and demonstrate its superior performance over previous domain adaptation approaches as well as other skeleton augmentation strategies.

Mesh Neural Networks Based on Dual Graph Pyramids.

Deep neural networks (DNNs) have been widely used for mesh processing in recent years. However, current DNNs can not process arbitrary meshes efficiently. On the one hand, most DNNs expect 2-manifold, watertight meshes, but many meshes, whether manually designed or automatically generated, may have gaps, non-manifold geometry, or other defects. On the other hand, the irregular structure of meshes also brings challenges to building hierarchical structures and aggregating local geometric information, which is critical to conduct DNNs. In this paper, we present DGNet, an efficient, effective and generic deep neural mesh processing network based on dual graph pyramids; it can handle arbitrary meshes. Firstly, we construct dual graph pyramids for meshes to guide feature propagation between hierarchical levels for both downsampling and upsampling. Secondly, we propose a novel convolution to aggregate local features on the proposed hierarchical graphs. By utilizing both geodesic neighbors and Euclidean neighbors, the network enables feature aggregation both within local surface patches and between isolated mesh components. Experimental results demonstrate that DGNet can be applied to both shape analysis and large-scale scene understanding. Furthermore, it achieves superior performance on various benchmarks, including ShapeNetCore, HumanBody, ScanNet and Matterport3D. Code and models will be available at https://github.com/li-xl/DGNet.

Effect of sucrose, trehalose, maltose and xylose on rheology, water mobility and microstructure of gluten-free model dough based on high hydrostatic pressure treated starches.

Due to functional and physicochemical properties, starch in its native state has limited range of applications. Simultaneously, information on effects of different sugars and their interactions with modified starch on gluten-free model dough is also limited. To better overcome these restrictions, the effects of sucrose, trehalose, maltose and xylose on rheology, water mobility and microstructure of gluten-free dough prepared with high hydrostatic pressure (HHP) treated maize (MS), potato (PS) and sweet potato starch (SS) were investigated. MS, PS and SS dough with trehalose exhibited a lower degree of dependence of G' on frequency sweep (z'), higher strength (K) and relative elastic part of maximum creep compliance (Je/Jmax), suggesting stable network structure formation. Total gas production (VT) of MS dough with maltose, PS dough with sucrose and SS dough with trehalose was increased from 588 to 1454 mL, 537 to 1498 mL and 637 to 1455 mL respectively. Higher weakly bound water (T22) was found in the dough with trehalose at 60 min of fermentation, suggesting more hydrogen bonds and stable network. Thus, trehalose might be a potential improver in HHP treated starch-based gluten-free products.

Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks.

Attention mechanisms, especially self-attention, have played an increasingly important role in deep feature representation for visual tasks. Self-attention updates the feature at each position by computing a weighted sum of features using pair-wise affinities across all positions to capture the long-range dependency within a single sample. However, self-attention has quadratic complexity and ignores potential correlation between different samples. This article proposes a novel attention mechanism which we call external attention, based on two external, small, learnable, shared memories, which can be implemented easily by simply using two cascaded linear layers and two normalization layers; it conveniently replaces self-attention in existing popular architectures. External attention has linear complexity and implicitly considers the correlations between all data samples. We further incorporate the multi-head mechanism into external attention to provide an all-MLP architecture, external attention MLP (EAMLP), for image classification. Extensive experiments on image classification, object detection, semantic segmentation, instance segmentation, image generation, and point cloud analysis reveal that our method provides results comparable or superior to the self-attention mechanism and some of its variants, with much lower computational and memory costs.

Recursive-NeRF: An Efficient and Dynamically Growing NeRF.

View synthesis methods using implicit continuous shape representations learned from a set of images, such as the Neural Radiance Field (NeRF) method, have gained increasing attention due to their high quality imagery and scalability to high resolution. However, the heavy computation required by its volumetric approach prevents NeRF from being useful in practice; minutes are taken to render a single image of a few megapixels. Now, an image of a scene can be rendered in a level-of-detail manner, so we posit that a complicated region of the scene should be represented by a large neural network while a small neural network is capable of encoding a simple region, enabling a balance between efficiency and quality. Recursive-NeRF is our embodiment of this idea, providing an efficient and adaptive rendering and training approach for NeRF. The core of Recursive-NeRF learns uncertainties for query coordinates, representing the quality of the predicted color and volumetric intensity at each level. Only query coordinates with high uncertainties are forwarded to the next level to a bigger neural network with a more powerful representational capability. The final rendered image is a composition of results from neural networks of all levels. Our evaluation on public datasets and a large-scale scene dataset we collected shows that Recursive-NeRF is more efficient than NeRF while providing state-of-the-art quality. The code will be available at https://github.com/Gword/Recursive-NeRF.

SceneViewer: Automating Residential Photography in Virtual Environments.

Selecting views is one of the most common but overlooked procedures in topics related to 3D scenes. Typically, existing applications and researchers manually select views through a trial-and-error process or "preset" a direction, such as the top-down views. For example, literature for scene synthesis requires views for visualizing scenes. Research on panorama and VR also require initial placements for cameras, etc. This article presents SceneViewer, an integrated system for automatic view selections. Our system is achieved by applying rules of interior photography, which guides potential views and seeks better views. Through experiments and applications, we show the potentiality and novelty of the proposed method.

A nomogram based on clinical factors for preoperative prediction of nipple involvement in breast cancer.

Background: At present, the indication for nipple-sparing mastectomy (NSM) remains inconclusive, and occult nipple involvement (NI) is one of the most important problems when carrying out NSM. Therefore, we aimed to identify the predictive factors of NI, to provide a tool for selecting suitable candidates for NSM. Methods: In this retrospective study, a total of 250 breast cancer patients who received mastectomy were recruited, and the association between NI and tumor clinicopathologic characteristics was investigated. Nipple signs, tumor size measured by ultrasound (US), and tumor location were developed as a nomogram to predict NI. Results: Among the 250 patients, 34 (12.6%) had NI, and 216 (86.4%) did not. In the training group, NI was associated with nipple signs, tumor size, tumor-nipple distance (TND), tumor location, lymph node metastasis, and HER2 overexpression. Both in the training and in the validation groups, NI showed a significant association with nipple signs, tumor size measured by ultrasound, and tumor location. Based on these three clinical factors, the preoperative model nomogram was proved to have high efficiency in predicting NI, possessing a sensitivity of 80.0% and a specificity of 86.7% in the validation group. Conclusions: We proposed a predictive model nomogram utilizing preoperative tumor characteristics, including nipple signs, tumor size measured by ultrasound, and tumor location. This predictive model could help in the planning of nipple-sparing mastectomy.

Evaluating photoneutron dose equivalents for lung cancer using PMMA phantoms undergoing 15 MV IMRT.

BACKGROUND: According to statistics of the Ministry of Health and Welfare in 2017, the second leading cause of death in Taiwan was lung cancer. OBJECTIVE: Routine treatment planning does not consider photoneutron dose equivalent (PNDE) of patient induced secondary radiation resulting from primary exposure of lung cancer. However, such treatment is potentially important for improving estimates of health risks. METHODS: This study used 10, 30, 50, 70, and 90 kg of polymethylmethacrylate (PMMA) phantoms as patient to measure PNDE varying anatomical area during lung cancer of intensity modulated radiotherapy (IMRT) treatment. Paired thermoluminescent dosimeters (TLD-600 and 700) were calibrated using university reactor neutrons. TLDs were inserted into phantom which was closely corresponded of the represented tissues or organs. RESULTS: Neutron doses (ND) of organ or tissue (N⁢DT) were determined in these phantoms using paired TLDs approach. The risks of incurring fatal secondary malignancies, maximum statistical and total errors were estimated. Evaluated PNDE ranged from 0.80 ± 0.12 to 0.56 ± 0.08 mSv/Gy for these phantoms. CONCLUSION: The estimated N⁢DT decreased with increasing distance that is from the central axis. Evaluated PNDE and N⁢D𝑠𝑘𝑖𝑛 for these phantoms were discussed. This investigation also identified secondary risks associated with PNDE relating to radiation protection.

Comparative study of thermo-mechanical, rheological, and structural properties of gluten-free model doughs from high hydrostatic pressure treated maize, potato, and sweet potato starches.

Effects of high hydrostatic pressure (HHP, 100, 300 and 500 MPa for 30 min at 25 °C) treated maize (MS), potato (PS), and sweet potato (SS) starches on thermo-mechanical, rheological, microstructural properties and water distribution of gluten-free model doughs were investigated. Significant differences were found among starch model doughs in terms of water absorption, dough development time, and dough stability at 500 MPa. Total gas production of MS, PS and SS doughs was significantly increased from 541 to 605 mL (300 MPa), 527 to 568 mL (500 MPa) and 551 to 620 mL (500 MPa) respectively as HHP increased. HHP increased storage (G') and loss (G″) modulus in terms of rheological properties suggesting, the higher viscoelastic behavior of starch model doughs. The dough after 500 MPa treatment showed lower degree of dependence of G' on frequency sweep suggesting, the formation of a stable network structure. In addition, continuous abundant water distribution and uniform microstructure were found in MS (300 MPa), PS (500 MPa) and SS (500 MPa) doughs for 60 min fermentation. Thus, the starches after HHP show great application potential in gluten-free doughs with improved characteristics.

Functionalization of sweet potato leaf polyphenols by nanostructured composite ß-lactoglobulin particles from molecular level complexations: A review.

Sweet potato leaf polyphenols (SPLPs) have shown potential health benefits in the food and pharmaceutical industries. Nowadays, consumption of SPLPs from animal feeds to foodstuff is becoming a trend worldwide. However, the application of SPLPs is limited by their low bioavailability and stability. ß-lactoglobulin (ßlg), a highly regarded whey protein, can interact with SPLPs at the molecular level to form reversible or irreversible nanocomplexes (NCs). Consequently, the functional properties and final quality of SPLPs are directly modified. In this review, the composition and structure of SPLPs and ßlg, as well as methods of molecular complexation and mechanisms of formation of SPLPsßlgNCs, are revisited. The modified functionalities of SPLPsßlgNCs, especially protein conformational structures, antioxidant activity, solubility, thermal stability, emulsifying, and gelling properties including allergenic potential, digestibility, and practical applications are discussed for SPLPs future development.

Accurate Dynamic SLAM Using CRF-Based Long-Term Consistency.

Accurate camera pose estimation is essential and challenging for real world dynamic 3D reconstruction and augmented reality applications. In this article, we present a novel RGB-D SLAM approach for accurate camera pose tracking in dynamic environments. Previous methods detect dynamic components only across a short time-span of consecutive frames. Instead, we provide a more accurate dynamic 3D landmark detection method, followed by the use of long-term consistency via conditional random fields, which leverages long-term observations from multiple frames. Specifically, we first introduce an efficient initial camera pose estimation method based on distinguishing dynamic from static points using graph-cut RANSAC. These static/dynamic labels are used as priors for the unary potential in the conditional random fields, which further improves the accuracy of dynamic 3D landmark detection. Evaluation using the TUM and Bonn RGB-D dynamic datasets shows that our approach significantly outperforms state-of-the-art methods, providing much more accurate camera trajectory estimation in a variety of highly dynamic environments. We also show that dynamic 3D reconstruction can benefit from the camera poses estimated by our RGB-D SLAM approach.

Structural, antioxidant, aroma, and sensory characteristics of Maillard reaction products from sweet potato protein hydrolysates as influenced by different ultrasound-assisted enzymatic treatments.

Effects of energy-divergent ultrasound (EDU), energy-gathered ultrasound (EGU), and energy-gathered ultrasound-microwave (EGUM) on structure, antioxidant activities, aroma, and sensory attributes of Maillard reaction products (MRPs) from sweet potato protein hydrolysates (SPPH) were investigated. EGU and EGUM markedly enhanced the Maillard reaction (MR) progress. FTIR results revealed significant peptide structure changes in MRPs as compared to their SPPHs counterparts. EGU-MRPs exhibited the highest percentages in lower MW fractions of 200-3,000 Da, and presented a significantly enhanced ORAC value of 92.10 µg TE/mL (p < 0.05). Besides, EGU-MRPs and EGUM-MRPs showed higher content and quality of aroma compounds than other MRPs, and presented increased umami, sweetness, and sourness attributes, but decreased bitterness (p < 0.05). Their stronger umami taste was highly correlated to 1-naphthalenol, dodecanoic acid, <200, 200-500, 500-1,000 and 1,000-3,000 Da. Thus, EGU and EGUM assisted enzymatic hydrolysis coupled with MR might be promising ways to produce natural flavoring with improved antioxidant activities.

Contribution of ultrasound and slightly acid electrolytic water combination on inactivating Rhizopus stolonifer in sweet potato.

Effects of ultrasound (US, 300, 400, and 500 W) and slightly acidic electrolyzed water (SAEW, 10, 30, and 50 mg/L) combination on inactivating Rhizopus stolonifer in sweet potato tuberous roots (TRs) were investigated. US at 300, 400, and 500 W simultaneous SAEW with available chlorine concentration of 50 mg/L at 40 and 55 °C for 10 min significantly inhibited colony diameters (from 90.00 to 6.00-71.62 mm) and spores germination (p < 0.05). US + SAEW treatment could destroy cell membrane integrity and lead to the leakage of nucleic acids and proteins (p < 0.05). Scanning and transmission electron microscopy results showed that US + SAEW treatment could damage ultrastructure of R. stolonifer, resulted in severe cell-wall pitting, completely disrupted into debris, apparent separation of plasma wall, massive vacuoles space, and indistinct intracellular organelles. US500 + SAEW50 treatment at 40 and 55 °C increased cell membrane permeability, and decreased mitochondrial membrane potential of R. stolonifer. In addition, US500 + SAEW50 at 40 °C and US300 + SAEW50 at 55 °C controlled R. stolonifer growth in sweet potato TRs during 20 days of storage, suggesting effective inhibition on the infection of R. stolonifer. Therefore, US + SAEW treatment could be a new efficient alternative method for storing and preserving sweet potato TRs.

Physicochemical properties, antioxidant activities, and binding behavior of 3,5-di-O-caffeoylquinic acid with beta-lactoglobulin colloidal particles.

Milk proteins and polyphenols are increasingly being studied as functional ingredients due to the epidemiologically-proved health benefits. In this study, composite ß-lactoglobulin (ß-lg) or ß-lactoglobulin nanoparticles (ß-lgNPs)-3,5-di-O-caffeoylquinic acid (3,5diCQA) with superior physicochemical and antioxidant activity (AA) were produced using ß-lg and 3,5-di-O-caffeoylquinic acid. The main interactions between ß-lg or ß-lgNPs with 3,5diCQA were hydrogen bonding and hydrophobic effects. The 3,5diCQA caused a decrease in α-helix and ß-sheet structure with a corresponding increase in unordered structure. Compared to ß-lg alone, composite ß-lg or ß-lgNPs-3,5diCQA slightly decreased the particle size but increased their negative surface potentials especially for ß-lg or ß-lgNPs at a molar ratio of 5:1. The addition of 3,5diCQA appreciably improved the AA in a dose-dependent manner. These results shed light on the structural, physicochemical, and AA of composite ß-lg or ß-lgNPs-3,5diCQA non-covalent complexes, important for application as functional ingredients in food solutions as well as in the pharmaceutical industry.