Fast Non-Rigid Radiance Fields from Monocularized Data.

The reconstruction and novel view synthesis of dynamic scenes recently gained increased attention. As reconstruction from large-scale multi-view data involves immense memory and computational requirements, recent benchmark datasets provide collections of single monocular views per timestamp sampled from multiple (virtual) cameras. We refer to this form of inputs as monocularized data. Existing work shows impressive results for synthetic setups and forward-facing real-world data, but is often limited in the training speed and angular range for generating novel views. This paper addresses these limitations and proposes a new method for full 360° inward-facing novel view synthesis of non-rigidly deforming scenes. At the core of our method are: 1) An efficient deformation module that decouples the processing of spatial and temporal information for accelerated training and inference; and 2) A static module representing the canonical scene as a fast hash-encoded neural radiance field. In addition to existing synthetic monocularized data, we systematically analyze the performance on real-world inward-facing scenes using a newly recorded challenging dataset sampled from a synchronized large-scale multi-view rig. In both cases, our method is significantly faster than previous methods, converging in less than 7 minutes and achieving real-time framerates at 1K resolution, while obtaining a higher visual accuracy for generated novel views. Our code and dataset are available online: https://github.com/MoritzKappel/MoNeRF.

Wavelet-Based Fast Decoding of 360 ° Videos.

In this paper, we propose a wavelet-based video codec specifically designed for VR displays that enables real-time playback of high-resolution 360° videos. Our codec exploits the fact that only a fraction of the full 360° video frame is visible on the display at any time. To load and decode the video viewport-dependently in real time, we make use of the wavelet transform for intra- as well as inter-frame coding. Thereby, the relevant content is directly streamed from the drive, without the need to hold the entire frames in memory. With an average of 193 frames per second at 8192 × 8192 -pixel full-frame resolution, the conducted evaluation demonstrates that our codec's decoding performance is up to 272% higher than that of the state-of-the-art video codecs H.265 and AV1 for typical VR displays. By means of a perceptual study, we further illustrate the necessity of high frame rates for a better VR experience. Finally, we demonstrate how our wavelet-based codec can also directly be used in conjunction with foveation for further performance increase.

Omnidirectional Galvanic Vestibular Stimulation in Virtual Reality.

In this paper we propose omnidirectional galvanic vestibular stimulation (GVS) to mitigate cybersickness in virtual reality applications. One of the most accepted theories indicates that Cybersickness is caused by the visually induced impression of ego motion while physically remaining at rest. As a result of this sensory mismatch, people associate negative symptoms with VR and sometimes avoid the technology altogether. To reconcile the two contradicting sensory perceptions, we investigate GVS to stimulate the vestibular canals behind our ears with low-current electrical signals that are specifically attuned to the visually displayed camera motion. We describe how to calibrate and generate the appropriate GVS signals in real-time for pre-recorded omnidirectional videos exhibiting ego-motion in all three spatial directions. For validation, we conduct an experiment presenting real-world 360° videos shot from a moving first-person perspective in a VR head-mounted display. Our findings indicate that GVS is able to significantly reduce discomfort for cybersickness-susceptible VR users, creating a deeper and more enjoyable immersive experience for many people.

Visual Analytics for Development and Evaluation of Order Selection Criteria for Autoregressive Processes.

Order selection of autoregressive processes is an active research topic in time series analysis, and the development and evaluation of automatic order selection criteria remains a challenging task for domain experts. We propose a visual analytics approach, to guide the analysis and development of such criteria. A flexible synthetic model generator-combined with specialized responsive visualizations-allows comprehensive interactive evaluation. Our fast framework allows feedback-driven development and fine-tuning of new order selection criteria in real-time. We demonstrate the applicability of our approach in three use-cases for two general as well as a real-world example.

Temporal Video Filtering and Exposure Control for Perceptual Motion Blur.

We propose the computation of a perceptual motion blur in videos. Our technique takes the predicted eye motion into account when watching the video. Compared to traditional motion blur recorded by a video camera our approach results in a perceptual blur that is closer to reality. This postprocess can also be used to simulate different shutter effects or for other artistic purposes. It handles real and artificial video input, is easy to compute and has a low additional cost for rendered content. We illustrate its advantages in a user study using eye tracking.

An approach toward fast gradient-based image segmentation.

In this paper, we present and investigate an approach to fast multilabel color image segmentation using convex optimization techniques. The presented model is in some ways related to the well-known Mumford-Shah model, but deviates in certain important aspects. The optimization problem has been designed with two goals in mind. The objective function should represent fundamental concepts of image segmentation, such as incorporation of weighted curve length and variation of intensity in the segmented regions, while allowing transformation into a convex concave saddle point problem that is computationally inexpensive to solve. This paper introduces such a model, the nontrivial transformation of this model into a convex-concave saddle point problem, and the numerical treatment of the problem. We evaluate our approach by applying our algorithm to various images and show that our results are competitive in terms of quality at unprecedentedly low computation times. Our algorithm allows high-quality segmentation of megapixel images in a few seconds and achieves interactive performance for low resolution images.

ElectroEncephaloGraphics: Making waves in computer graphics research.

Electroencephalography (EEG) is a novel modality for investigating perceptual graphics problems. Until recently, EEG has predominantly been used for clinical diagnosis, in psychology, and by the brain-computer-interface community. Researchers are extending it to help understand the perception of visual output from graphics applications and to create approaches based on direct neural feedback. Researchers have applied EEG to graphics to determine perceived image and video quality by detecting typical rendering artifacts, to evaluate visualization effectiveness by calculating the cognitive load, and to automatically optimize rendering parameters for images and videos on the basis of implicit neural feedback.

Optimizing apparent display resolution enhancement for arbitrary videos.

Display resolution is frequently exceeded by available image resolution. Recently, apparent display resolution enhancement (ADRE) techniques show how characteristics of the human visual system can be exploited to provide super-resolution on high refresh rate displays. In this paper, we address the problem of generalizing the ADRE technique to conventional videos of arbitrary content. We propose an optimization-based approach to continuously translate the video frames in such a way that the added motion enables apparent resolution enhancement for the salient image region. The optimization considers the optimal velocity, smoothness, and similarity to compute an appropriate trajectory. In addition, we provide an intuitive user interface that allows to guide the algorithm interactively and preserves important compositions within the video. We present a user study evaluating apparent rendering quality and show versatility of our method on a variety of general test scenes.

Modeling and verifying the polarizing reflectance of real-world metallic surfaces.

Using measurements of real-world samples of metals, the proposed approach verifies predictions of bidirectional reflectance distribution function (BRDF) models. It employs ellipsometry to verify both the actual polarizing effect and the overall reflectance behavior of the metallic surfaces.

Synthetic generation of high-dimensional datasets.

Generation of synthetic datasets is a common practice in many research areas. Such data is often generated to meet specific needs or certain conditions that may not be easily found in the original, real data. The nature of the data varies according to the application area and includes text, graphs, social or weather data, among many others. The common process to create such synthetic datasets is to implement small scripts or programs, restricted to small problems or to a specific application. In this paper we propose a framework designed to generate high dimensional datasets. Users can interactively create and navigate through multi dimensional datasets using a suitable graphical user-interface. The data creation is driven by statistical distributions based on a few user-defined parameters. First, a grounding dataset is created according to given inputs, and then structures and trends are included in selected dimensions and orthogonal projection planes. Furthermore, our framework supports the creation of complex non-orthogonal trends and classified datasets. It can successfully be used to create synthetic datasets simulating important trends as multidimensional clusters, correlations and outliers.

Shape: A 3D Modeling Tool for Astrophysics.

We present a flexible interactive 3D morpho-kinematical modeling application for astrophysics. Compared to other systems, our application reduces the restrictions on the physical assumptions, data type, and amount that is required for a reconstruction of an object's morphology. It is one of the first publicly available tools to apply interactive graphics to astrophysical modeling. The tool allows astrophysicists to provide a priori knowledge about the object by interactively defining 3D structural elements. By direct comparison of model prediction with observational data, model parameters can then be automatically optimized to fit the observation. The tool has already been successfully used in a number of astrophysical research projects.

Motion Field Estimation from Alternate Exposure Images.

Traditional optical flow algorithms rely on consecutive short-exposed images. In this work, we make use of an additional long-exposed image for motion field estimation. Long-exposed images integrate motion information directly in the form of motion-blur. With this additional information, more robust and accurate motion fields can be estimated. In addition, the moment of occlusion can be determined. Considering the basic signal-theoretical problem in motion field estimation, we exploit the fact that long-exposed images integrate motion information to prevent temporal aliasing. A suitable image formation model relates the long-exposed image to preceding and succeeding short-exposed images in terms of dense 2D motion and per-pixel occlusion/disocclusion timings. Based on our image formation model, we describe a practical variational algorithm to estimate the motion field not only for visible image regions but also for regions getting occluded. Results for synthetic as well as real-world scenes demonstrate the validity of the approach.

Automated Analytical Methods to Support Visual Exploration of High-Dimensional Data.

Visual exploration of multivariate data typically requires projection onto lower dimensional representations. The number of possible representations grows rapidly with the number of dimensions, and manual exploration quickly becomes ineffective or even unfeasible. This paper proposes automatic analysis methods to extract potentially relevant visual structures from a set of candidate visualizations. Based on features, the visualizations are ranked in accordance with a specified user task. The user is provided with a manageable number of potentially useful candidate visualizations, which can be used as a starting point for interactive data analysis. This can effectively ease the task of finding truly useful visualizations and potentially speed up the data exploration task. In this paper, we present ranking measures for class-based as well as non-class-based scatterplots and parallel coordinates visualizations. The proposed analysis methods are evaluated on different data sets.

Computer generated holograms from three dimensional meshes using an analytic light transport model.

We present a method to analytically compute the light distribution of triangles directly in frequency space. This allows for fast evaluation, shading, and propagation of light from 3D mesh objects using angular spectrum methods. The algorithm complexity is only dependent on the hologram resolution and the polygon count of the 3D model. In contrast to other polygon based computer generated holography methods we do not need to perform a Fourier transform per surface. The theory behind the approach is derived, and a suitable algorithm to compute a digital hologram from a general triangle mesh is presented. We review some first results rendered on a spatial-light-modulator-based display by our proof-of-concept software.

Weighted minimal hypersurface reconstruction.

Many problems in computer vision can be formulated as a minimization problem for an energy functional. If this functional is given as an integral of a scalar-valued weight function over an unknown hypersurface, then the sought-after minimal surface can be determined as a solution of the functional's Euler-Lagrange equation. This paper deals with a general class of weight functions that may depend on surface point coordinates as well as surface orientation. We derive the Euler-Lagrange equation in arbitrary dimensional space without the need for any surface parameterization, generalizing existing proofs. Our work opens up the possibility of solving problems involving minimal hypersurfaces in a dimension higher than three, which were previously impossible to solve in practice. We also introduce two applications of our new framework: We show how to reconstruct temporally coherent geometry from multiple video streams, and we use the same framework for the volumetric reconstruction of refractive and transparent natural phenomena, here bodies of flowing water.

Seeing people in different light--joint shape, motion, and reflectance capture.

By means of passive optical motion capture, real people can be authentically animated and photo-realistically textured. To import real-world characters into virtual environments, however, surface reflectance properties must also be known. We describe a video-based modeling approach that captures human shape and motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to recover spatially varying surface reflectance properties of clothes from multiview video footage. The resulting model description enables us to realistically reproduce the appearance of animated virtual actors under different lighting conditions, as well as to interchange surface attributes among different people, e.g., for virtual dressing. Our contribution can be used to create 3D renditions of real-world people under arbitrary novel lighting conditions on standard graphics hardware.

Computer generated holography using parallel commodity graphics hardware.

This paper presents a novel method for using programmable graphics hardware to generate fringe patterns for SLM-based holographic displays. The algorithm is designed to take the programming constraints imposed by the graphics hardware pipeline model into consideration, and scales linearly with the number of object points. In contrast to previous methods we do not have to use the Fresnel approximation. The technique can also be used on several graphics processors in parallel for further optimization. We achieve real-time frame rates for objects consisting of a few hundred points at a resolution of 960x600 pixels and over 10 frames per second for 1000 points.

Reconstruction and visualization of planetary nebulae.

From our terrestrially confined viewpoint, the actual three-dimensional shape of distant astronomical objects is, in general, very challenging to determine. For one class of astronomical objects, however, spatial structure can be recovered from conventional 2D images alone. So-called planetary nebulae (PNe) exhibit pronounced symmetry characteristics that come about due to fundamental physical processes. Making use of this symmetry constraint, we present a technique to automatically recover the axisymmetric structure of many planetary nebulae from photographs. With GPU-based volume rendering driving a nonlinear optimization, we estimate the nebula's local emission density as a function of its radial and axial coordinates and we recover the orientation of the nebula relative to Earth. The optimization refines the nebula model and its orientation by minimizing the differences between the rendered image and the original astronomical image. The resulting model allows creating realistic 3D visualizations of these nebulae, for example, for planetarium shows and other educational purposes. In addition, the recovered spatial distribution of the emissive gas can help astrophysicists gain deeper insight into the formation processes of planetary nebulae.