Shadowless Projection Mapping using Retrotransmissive Optics.

This paper presents a shadowless projection mapping system for interactive applications in which a target surface is frequently occluded from a projector with a user's body. We propose a delay-free optical solution for this critical problem. Specifically, as the primary technical contribution, we apply a large format retrotransmissive plate to project images onto the target surface from wide viewing angles. We also tackle technical issues unique to the proposed shadowless principle. First, the retrotransmissive optics inevitably suffer from stray light, which leads to significant contrast degradation of the projected result. We propose to block the stray light by covering the retrotransmissive plate with a spatial mask. Because the mask reduces not only the stray light but the achievable luminance of the projected result, we develop a computational algorithm that determines the shape of the mask to balance the image quality. Second, we propose a touch sensing technique by leveraging the optically bidirectional property of the retrotransmissive plate to support interaction between the user and the projected contents on the target object. We implement a proof-of-concept prototype and validate the above-mentioned techniques through experiments.

Multifocal Stereoscopic Projection Mapping.

Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence-accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the "virtual image" (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: (1) The 3D CG object is displayed on non-planar and even moving surfaces; (2) the physical surfaces need to be shown without the focus modulation; (3) the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the "lens breathing" artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task.

Directionally Decomposing Structured Light for Projector Calibration.

Intrinsic projector calibration is essential in projection mapping (PM) applications, especially in dynamic PM. However, due to the shallow depth-of-field (DOF) of a projector, more work is needed to ensure accurate calibration. We aim to estimate the intrinsic parameters of a projector while avoiding the limitation of shallow DOF. As the core of our technique, we present a practical calibration device that requires a minimal working volume directly in front of the projector lens regardless of the projector's focusing distance and aperture size. The device consists of a flat-bed scanner and pinhole-array masks. For calibration, a projector projects a series of structured light patterns in the device. The pinholes directionally decompose the structured light, and only the projected rays that pass through the pinholes hit the scanner plane. For each pinhole, we extract a ray passing through the optical center of the projector. Consequently, we regard the projector as a pinhole projector that projects the extracted rays only, and we calibrate the projector by applying the standard camera calibration technique, which assumes a pinhole camera model. Using a proof-of-concept prototype, we demonstrate that our technique can calibrate projectors with different focusing distances and aperture sizes at the same accuracy as a conventional method. Finally, we confirm that our technique can provide intrinsic parameters accurate enough for a dynamic PM application, even when a projector is placed too far from a projection target for a conventional method to calibrate the projector using a fiducial object of reasonable size.

FleXeen: Visually Manipulating Perceived Fabric Bending Stiffness in Spatial Augmented Reality.

The appearance of fabric motion is suggested to affect the human perception of bending stiffness. This study presents a novel spatial augmented reality, or projection mapping, approach that can visually manipulate the perceived bending stiffness of a fabric. Particularly, we proposed a flow enhancement method that can change the apparent fabric motion by using a simple optical flow analysis technique rather than complex physical simulations for interactive applications. Through a psychophysical experiment, we investigated the relationship between the magnification factor of our flow enhancement and the perceived bending stiffness of a fabric. Furthermore, we constructed a prototype application system that allows users to control the stiffness of a fabric without changing the actual physical fabric. By evaluating the prototype, we confirmed that the proposed technique can manipulate the perceived stiffness of various materials (i.e., cotton, polyester, and mixed cotton and linen) at an average accuracy of 90.3 percent.

Extended LazyNav: Virtual 3D Ground Navigation for Large Displays and Head-Mounted Displays.

This paper presents the extended work on LazyNav, a head-free, eyes-free and hands-free mid-air ground navigation control model presented at the IEEE 3D User Interfaces (3DUI) 2015, in particular with a new application to the head-mounted display (HMD). Our mid-air interaction metaphor makes use of only a single pair of the remaining tracked body elements to tailor the navigation. Therefore, the user can navigate in the scene while still being able to perform other interactions with her hands and head, e.g., carrying a bag, grasping a cup of coffee, or observing the content by moving her eyes and locally rotating her head. We design several body motions for navigation by considering the use of non-critical body parts and develop assumptions about ground navigation techniques. Through the user studies, we investigate the motions that are easy to discover, easy to control, socially acceptable, accurate and not tiring. Finally, we evaluate the desired ground navigation features with a prototype application in both a large display (LD) and a HMD navigation scenarios. We highlight several recommendations for designing a particular mid-air ground navigation technique for a LD and a HMD.

SoftAR: visually manipulating haptic softness perception in spatial augmented reality.

We present SoftAR, a novel spatial augmented reality (AR) technique based on a pseudo-haptics mechanism that visually manipulates the sense of softness perceived by a user pushing a soft physical object. Considering the limitations of projection-based approaches that change only the surface appearance of a physical object, we propose two projection visual effects, i.e., surface deformation effect (SDE) and body appearance effect (BAE), on the basis of the observations of humans pushing physical objects. The SDE visualizes a two-dimensional deformation of the object surface with a controlled softness parameter, and BAE changes the color of the pushing hand. Through psychophysical experiments, we confirm that the SDE can manipulate softness perception such that the participant perceives significantly greater softness than the actual softness. Furthermore, fBAE, in which BAE is applied only for the finger area, significantly enhances manipulation of the perception of softness. We create a computational model that estimates perceived softness when SDE+fBAE is applied. We construct a prototype SoftAR system in which two application frameworks are implemented. The softness adjustment allows a user to adjust the softness parameter of a physical object, and the softness transfer allows the user to replace the softness with that of another object.