HoloCamera: Advanced Volumetric Capture for Cinematic-Quality VR Applications.

High-precision virtual environments are increasingly important for various education, simulation, training, performance, and entertainment applications. We present HoloCamera, an innovative volumetric capture instrument to rapidly acquire, process, and create cinematic-quality virtual avatars and scenarios. The HoloCamera consists of a custom-designed free-standing structure with 300 high-resolution RGB cameras mounted with uniform spacing spanning the four sides and the ceiling of a room-sized studio. The light field acquired from these cameras is streamed through a distributed array of GPUs that interleave the processing and transmission of 4K resolution images. The distributed compute infrastructure that powers these RGB cameras consists of 50 Jetson AGX Xavier boards, with each processing unit dedicated to driving and processing imagery from six cameras. A high-speed Gigabit Ethernet network fabric seamlessly interconnects all computing boards. In this systems paper, we provide an in-depth description of the steps involved and lessons learned in constructing such a cutting-edge volumetric capture facility that can be generalized to other such facilities. We delve into the techniques employed to achieve precise frame synchronization and spatial calibration of cameras, careful determination of angled camera mounts, image processing from the camera sensors, and the need for a resilient and robust network infrastructure. To advance the field of volumetric capture, we are releasing a high-fidelity static light-field dataset, which will serve as a benchmark for further research and applications of cinematic-quality volumetric light fields.

Neural Subspaces for Light Fields.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

Benchmarking AlphaFold for protein complex modeling reveals accuracy determinants.

High-resolution experimental structural determination of protein-protein interactions has led to valuable mechanistic insights, yet due to the massive number of interactions and experimental limitations there is a need for computational methods that can accurately model their structures. Here we explore the use of the recently developed deep learning method, AlphaFold, to predict structures of protein complexes from sequence. With a benchmark of 152 diverse heterodimeric protein complexes, multiple implementations and parameters of AlphaFold were tested for accuracy. Remarkably, many cases (43%) had near-native models (medium or high critical assessment of predicted interactions accuracy) generated as top-ranked predictions by AlphaFold, greatly surpassing the performance of unbound protein-protein docking (9% success rate for near-native top-ranked models), however AlphaFold modeling of antibody-antigen complexes within our set was unsuccessful. We identified sequence and structural features associated with lack of AlphaFold success, and we also investigated the impact of multiple sequence alignment input. Benchmarking of a multimer-optimized version of AlphaFold (AlphaFold-Multimer) with a set of recently released antibody-antigen structures confirmed a low rate of success for antibody-antigen complexes (11% success), and we found that T cell receptor-antigen complexes are likewise not accurately modeled by that algorithm, showing that adaptive immune recognition poses a challenge for the current AlphaFold algorithm and model. Overall, our study demonstrates that end-to-end deep learning can accurately model many transient protein complexes, and highlights areas of improvement for future developments to reliably model any protein-protein interaction of interest.

Evaluation Challenges for the Application of Extended Reality Devices in Medicine.

Augmented and virtual reality devices are being actively investigated and implemented for a wide range of medical uses. However, significant gaps in the evaluation of these medical devices and applications hinder their regulatory evaluation. Addressing these gaps is critical to demonstrating the devices' safety and effectiveness. We outline the key technical and clinical evaluation challenges discussed during the US Food and Drug Administration's public workshop, "Medical Extended Reality: Toward Best Evaluation Practices for Virtual and Augmented Reality in Medicine" and future directions for evaluation method development. Evaluation challenges were categorized into several key technical and clinical areas. Finally, we highlight current efforts in the standards communities and illustrate connections between the evaluation challenges and the intended uses of the medical extended reality (MXR) devices. Participants concluded that additional research is needed to assess the safety and effectiveness of MXR devices across the use cases.

Visual Attention to Health Warning Labels on Waterpipe Venue Menus in Immersive Virtual Reality.

INTRODUCTION: This study examined how health warning labels (HWL) on a waterpipe venue menu captured and held the attention of consumers and influenced waterpipe tobacco smoking (WTS) attitudes, beliefs, and behaviors. AIMS AND METHODS: A randomized experiment (N = 96) of young adult waterpipe smokers was conducted in an immersive virtual reality laboratory. Participants viewed one of two virtual reality scenarios, a menu with an HWL and nicotine concentration or menu without an HWL and nicotine concentration. Eye-tracking metrics were collected, and participants completed posttest questionnaires on demographics, tobacco use history, and WTS attitudes, beliefs, and behaviors. T-tests were used to assess group differences, and a mediation analysis conducted to examine the relationship between the HWL and intention to quit WTS. RESULTS: Participants in the HWL group demonstrated greater visual attention to the warning and nicotine areas and less visual attention to the flavor and ingredients areas of the menu compared to the control group. The HWL group demonstrated greater negative attitudes toward WTS (p = .002), greater perceived risk of decreased lung function (p = .026), and greater intention to quit WTS (p = 0.003). The mediation model indicated the relationship between the HWL on a menu and intention to quit WTS was mediated by an increase in negative attitudes toward WTS. CONCLUSIONS: The HWLs on a menu captured and held the attention of consumers and increased negative attitudes, perceptions of health risk, and intention to quit WTS indicating potential benefit of including a warning label or nicotine concentration on menus to correct misperceptions of WTS. IMPLICATIONS: The study contributes to the broader literature on communicating the harms and risks of WTS. The findings suggest that HWL and nicotine concentration on waterpipe venue menus attract attention from consumers in environments comparable to the real world and the strategy warrants further exploration as a targeted policy intervention to educate the public and reduce the health burden of WTS.

Saliency Computation for Virtual Cinematography in 360° Videos.

Recent advances in virtual reality cameras have contributed to a phenomenal growth of 360$^{\circ }$∘ videos. Estimating regions likely to attract user attention is critical for efficiently streaming and rendering 360$^{\circ }$∘ videos. In this article, we present a simple, novel, GPU-driven pipeline for saliency computation and virtual cinematography in 360$^{\circ }$∘ videos using spherical harmonics (SH). We efficiently compute the 360$^{\circ }$∘ video saliency through the spectral residual of the SH coefficients between multiple bands at over 60FPS for 4K resolution videos. Further, our interactive computation of spherical saliency can be used for saliency-guided virtual cinematography in 360$^{\circ }$∘ videos.

A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming.

With the rapidly increasing resolutions of 360° cameras, head-mounted displays, and live-streaming services, streaming high-resolution panoramic videos over limited-bandwidth networks is becoming a critical challenge. Foveated video streaming can address this rising challenge in the context of eye-tracking-equipped virtual reality head-mounted displays. However, conventional log-polar foveated rendering suffers from a number of visual artifacts such as aliasing and flickering. In this paper, we introduce a new log-rectilinear transformation that incorporates summed-area table filtering and off-the-shelf video codecs to enable foveated streaming of 360° videos suitable for VR headsets with built-in eye-tracking. To validate our approach, we build a client-server system prototype for streaming 360° videos which leverages parallel algorithms over real-time video transcoding. We conduct quantitative experiments on an existing 360° video dataset and observe that the log-rectilinear transformation paired with summed-area table filtering heavily reduces flickering compared to log-polar subsampling while also yielding an additional 10% reduction in bandwidth usage.

3D-Kernel Foveated Rendering for Light Fields.

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47×-7.28× speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

Correcting the Proximity Effect in Nanophotonic Phased Arrays.

Thermally modulated Nanophotonic Phased Arrays (NPAs) can be used as phase-only holographic displays. Compared to the holographic displays based on Liquid Crystal on Silicon Spatial Light Modulators (LCoS SLMs), NPAs have the advantage of integrated light source and high refresh rate. However, the formation of the desired wavefront requires accurate modulation of the phase which is distorted by the thermal proximity effect. This problem has been largely overlooked and existing approaches to similar problems are either slow or do not provide a good result in the setting of NPAs. We propose two new algorithms based on the iterative phase retrieval algorithm and the proximal algorithm to address this challenge. We have carried out computational simulations to compare and contrast various algorithms in terms of image quality and computational efficiency. This work is going to benefit the research on NPAs and enable the use of large-scale NPAs as holographic displays.

Virtual reality: physiological and behavioral mechanisms to increase individual pain tolerance limits.

ABSTRACT: Immersive virtual reality (VR) consists of immersion in artificial environments through the use of real-time render technologies and the latest generation devices. The users feel just as immersed as they would feel in an everyday life situation, and this sense of presence seems to have therapeutic potentials. However, the VR mechanisms remain only partially known. This study is novel in that, for the first time in VR research, appropriate controls for VR contexts, immersive characteristics (ie, control VR), and multifaceted objective and subjective outcomes were included in a within-subject study design conducted on healthy participants. Participants received heat thermal stimulations to determine how VR can increase individual heat-pain tolerance limits (primary outcome) measured in degrees Celsius and seconds while recording concurrent autonomic responses. We also assessed changes in pain unpleasantness, mood, situational anxiety, and level of enjoyment (secondary outcomes). The VR induced a net gain in heat-pain tolerance limits that was paralleled by an increase of the parasympathetic responses. VR improved mood, situational anxiety, and pain unpleasantness when participants perceived the context as enjoyable, but these changes did not influence the increases in pain tolerance limits. Distraction increased pain tolerance limits but did not induce such mood and physiological changes. Immersive VR has been anecdotally applied to improve acute symptoms in contexts such as battlefield, emergency, and operating rooms. This study provides a mechanistic framework for VR as a low-risk, nonpharmacological intervention, which regulates autonomic, affective (mood and situational anxiety), and evaluative (subjective pain and enjoyment ratings) responses associated with acute pain.

Eye-dominance-guided Foveated Rendering.

Optimizing rendering performance is critical for a wide variety of virtual reality (VR) applications. Foveated rendering is emerging as an indispensable technique for reconciling interactive frame rates with ever-higher head-mounted display resolutions. Here, we present a simple yet effective technique for further reducing the cost of foveated rendering by leveraging ocular dominance - the tendency of the human visual system to prefer scene perception from one eye over the other. Our new approach, eye-dominance-guided foveated rendering (EFR), renders the scene at a lower foveation level (with higher detail) for the dominant eye than the non-dominant eye. Compared with traditional foveated rendering, EFR can be expected to provide superior rendering performance while preserving the same level of perceived visual quality.

Deep-Learning-Assisted Volume Visualization.

Designing volume visualizations showing various structures of interest is critical to the exploratory analysis of volumetric data. The last few years have witnessed dramatic advances in the use of convolutional neural networks for identification of objects in large image collections. Whereas such machine learning methods have shown superior performance in a number of applications, their direct use in volume visualization has not yet been explored. In this paper, we present a deep-learning-assisted volume visualization to depict complex structures, which are otherwise challenging for conventional approaches. A significant challenge in designing volume visualizations based on the high-dimensional deep features lies in efficiently handling the immense amount of information that deep-learning methods provide. In this paper, we present a new technique that uses spectral methods to facilitate user interactions with high-dimensional features. We also present a new deep-learning-assisted technique for hierarchically exploring a volumetric dataset. We have validated our approach on two electron microscopy volumes and one magnetic resonance imaging dataset.

Tracking fluctuation hotspots on the yeast ribosome through the elongation cycle.

Chemical modification was used to quantitatively determine the flexibility of nearly the entire rRNA component of the yeast ribosome through 8 discrete stages of translational elongation, revealing novel observations at the gross and fine-scales. These include (i) the bulk transfer of energy through the intersubunit bridges from the large to the small subunit after peptidyltransfer, (ii) differences in the interaction of the sarcin ricin loop with the two elongation factors and (iii) networked information exchange pathways that may functionally facilitate intra- and intersubunit coordination, including the 5.8S rRNA. These analyses reveal hot spots of fluctuations that set the stage for large-scale conformational changes essential for translocation and enable the first molecular dynamics simulation of an 80S complex. Comprehensive datasets of rRNA base flexibilities provide a unique resource to the structural biology community that can be computationally mined to complement ongoing research toward the goal of understanding the dynamic ribosome.

Investigation of Multiple Frequency Ranges Using Discrete Wavelet Decomposition of Resting-State Functional Connectivity in Mild Traumatic Brain Injury Patients.

The aim of this study was to investigate if discrete wavelet decomposition provides additional insight into resting-state processes through the analysis of functional connectivity within specific frequency ranges within the default mode network (DMN) that may be affected by mild traumatic brain injury (mTBI). Participants included 32 mTBI patients (15 with postconcussive syndrome [PCS+] and 17 without [PCS-]). mTBI patients received resting-state functional magnetic resonance imaging (rs-fMRI) at acute (within 10 days of injury) and chronic (6 months postinjury) time points and were compared with 31 controls (healthy control [HC]). The wavelet decomposition divides the time series into multiple frequency ranges based on four scaling factors (SF1: 0.125-0.250 Hz, SF2: 0.060-0.125 Hz, SF3: 0.030-0.060 Hz, SF4: 0.015-0.030 Hz). Within each SF, wavelet connectivity matrices for nodes of the DMN were created for each group (HC, PCS+, PCS-), and bivariate measures of strength and diversity were calculated. The results demonstrate reduced strength of connectivity in PCS+ patients compared with PCS- patients within SF1 during both the acute and chronic stages of injury, as well as recovery of connectivity within SF1 across the two time points. Furthermore, the PCS- group demonstrated greater network strength compared with controls at both time points, suggesting a potential compensatory or protective mechanism in these patients. These findings stress the importance of investigating resting-state connectivity within multiple frequency ranges; however, many of our findings are within SF1, which may overlap with frequencies associated with cardiac and respiratory activities.

Visualization of Brain Microstructure Through Spherical Harmonics Illumination of High Fidelity Spatio-Angular Fields.

Diffusion kurtosis imaging (DKI) is gaining rapid adoption in the medical imaging community due to its ability to measure the non-Gaussian property of water diffusion in biological tissues. Compared to traditional diffusion tensor imaging (DTI), DKI can provide additional details about the underlying microstructural characteristics of the neural tissues. It has shown promising results in studies on changes in gray matter and mild traumatic brain injury where DTI is often found to be inadequate. The DKI dataset, which has high-fidelity spatio-angular fields, is difficult to visualize. Glyph-based visualization techniques are commonly used for visualization of DTI datasets; however, due to the rapid changes in orientation, lighting, and occlusion, visually analyzing the much more higher fidelity DKI data is a challenge. In this paper, we provide a systematic way to manage, analyze, and visualize high-fidelity spatio-angular fields from DKI datasets, by using spherical harmonics lighting functions to facilitate insights into the brain microstructure.

Diagnosis and management of nonsyndromic hereditary gingival fibromatosis in a 13 year old girl: Report of a rare case.

Hereditary gingival fibromatosis is a rare condition characterized by various degree of gingival overgrowth. It usually develops as an isolated disorder but can manifest with multisystem syndrome. We are here presenting a case of a 13-year-old girl who presented with severe enlargement of gingiva covering all most the entire crown involving both maxillary and mandibular arches. Differential diagnosis includes drug-induced and idiopathic gingival enlargement. Excess gingival tissue was removed by full mouth gingivectomy and sent for histopathological examination. Postoperative course was uneventful and patient's esthetics improved significantly. A 12 month postoperative period shows no recurrence.

Saliency-assisted navigation of very large landscape images.

The field of visualization has addressed navigation of very large datasets, usually meshes and volumes. Significantly less attention has been devoted to the issues surrounding navigation of very large images. In the last few years the explosive growth in the resolution of camera sensors and robotic image acquisition techniques has widened the gap between the display and image resolutions to three orders of magnitude or more. This paper presents the first steps towards navigation of very large images, particularly landscape images, from an interactive visualization perspective. The grand challenge in navigation of very large images is identifying regions of potential interest. In this paper we outline a three-step approach. In the first step we use multi-scale saliency to narrow down the potential areas of interest. In the second step we outline a method based on statistical signatures to further cull out regions of high conformity. In the final step we allow a user to interactively identify the exceptional regions of high interest that merit further attention. We show that our approach of progressive elicitation is fast and allows rapid identification of regions of interest. Unlike previous work in this area, our approach is scalable and computationally reasonable on very large images. We validate the results of our approach by comparing them to user-tagged regions of interest on several very large landscape images from the Internet.

Persuading visual attention through geometry.

Artists, illustrators, photographers, and cinematographers have long used the principles of contrast and composition to guide visual attention. In this paper we introduce geometry modification as a tool to persuasively direct visual attention. We build upon recent advances in mesh saliency to develop techniques to alter geometry to elicit greater visual attention. Eye-tracking-based user studies show that our approach successfully guides user attention in a statistically significant manner. Our approach operates directly on geometry, and therefore, produces view-independent results that can be used with existing view-dependent techniques of visual persuasion.

Parallel, stochastic measurement of molecular surface area.

Biochemists often wish to compute surface areas of proteins. A variety of algorithms have been developed for this task, but they are designed for traditional single-processor architectures. The current trend in computer hardware is towards increasingly parallel architectures for which these algorithms are not well suited. We describe a parallel, stochastic algorithm for molecular surface area computation that maps well to the emerging multi-core architectures. Our algorithm is also progressive, providing a rough estimate of surface area immediately and refining this estimate as time goes on. Furthermore, the algorithm generates points on the molecular surface which can be used for point-based rendering. We demonstrate a GPU implementation of our algorithm and show that it compares favorably with several existing molecular surface computation programs, giving fast estimates of the molecular surface area with good accuracy.