Musical Metaverse: vision, opportunities, and challenges.

The so-called metaverse relates to a vision of a virtual, digital world which is parallel to the real, physical world, where each user owns and interact through his/her own avatar. Music is one of the possible activities that can be conducted in such a space. The "Musical Metaverse" (MM), the metaverse part which is dedicated to musical activities, is currently in its infancy, although is a concept that is constantly evolving and is progressing at a steady pace. However, to the best of the author's knowledge, as of today an investigation about the opportunities and challenges posed by the MM has not been conducted yet. In this paper, we provide a vision for the MM and discuss what are the opportunities for musical stakeholders offered by current implementations of the MM, as well as we envision those that are likely to occur as the metaverse emerges. We also identify the technical, artistic, ethical, sustainability, and regulatory issues that need to be addressed so for the MM to be created and utilized in efficient, creative, and responsible ways. Given the importance and timeliness of the MM, we believe that a discussion on the related opportunities and concerns is useful to provide developers with guidelines for creating better virtual environments and musical interactions between stakeholders.

On the relation between the fields of Networked Music Performances, Ubiquitous Music, and Internet of Musical Things.

In the past two decades, we have witnessed the diffusion of an increasing number of technologies, products, and applications at the intersection of music and networking. As a result of the growing attention devoted by academy and industry to this area, three main research fields have emerged and progressively consolidated: the Networked Music Performances, Ubiquitous Music, and the Internet of Musical Things. Based on the review of the most relevant works in these fields, this paper attempts to delineate their differences and commonalities. The aim of this inquiry is helping avoid confusion between such fields and achieve a correct use of the terminology. A trend towards the convergence between such fields has already been identified, and it is plausible to expect that in the future their evolution will lead to a progressive blurring of the boundaries identified today.

To Hear or Not to Hear: Sound Availability Modulates Sensory-Motor Integration.

When we walk in place with our eyes closed after a few minutes of walking on a treadmill, we experience an unintentional forward body displacement (drift), called the sensory-motor aftereffect. Initially, this effect was thought to be due to the mismatch experienced during treadmill walking between the visual (absence of optic flow signaling body steadiness) and proprioceptive (muscle spindles firing signaling body displacement) information. Recently, the persistence of this effect has been shown even in the absence of vision, suggesting that other information, such as the sound of steps, could play a role. To test this hypothesis, six cochlear-implanted individuals were recruited and their forward drift was measured before (Control phase) and after (Post Exercise phase) walking on a treadmill while having their cochlear system turned on and turned off. The relevance in testing cochlear-implanted individuals was that when their system is turned off, they perceive total silence, even eliminating the sounds normally obtained from bone conduction. Results showed the absence of the aftereffect when the system was turned off, underlining the fundamental role played by sounds in the control of action and breaking new ground in the use of interactive sound feedback in motor learning and motor development.

Interactive footstep sounds modulate the perceptual-motor aftereffect of treadmill walking.

In this study, we investigated the role of interactive auditory feedback in modulating the inadvertent forward drift experienced while attempting to walk in place with closed eyes following a few minutes of treadmill walking. Simulations of footstep sounds upon surface materials such as concrete and snow were provided by means of a system composed of headphones and shoes augmented with sensors. In a control condition, participants could hear their actual footstep sounds. Results showed an overall enhancement of the forward drift after treadmill walking independent of the sound perceived, while the strength of the aftereffect, measured as the proportional increase (posttest/pretest) in forward drift, was higher under the influence of snow compared to both concrete and actual sound. In addition, a higher knee angle flexion was found during the snow sound condition both before and after treadmill walking. Behavioral results confirmed those of a perceptual questionnaire, which showed that the snow sound was effective in producing strong pseudo-haptic illusions. Our results provide evidence that the walking in place aftereffect results from a recalibration of haptic, visuo-motor but also sound-motor control systems. Self-motion perception is multimodal.

Haptic feedback for enhancing realism of walking simulations.

In this paper, we describe several experiments whose goal is to evaluate the role of plantar vibrotactile feedback in enhancing the realism of walking experiences in multimodal virtual environments. To achieve this goal we built an interactive and a noninteractive multimodal feedback system. While during the use of the interactive system subjects physically walked, during the use of the noninteractive system the locomotion was simulated while subjects were sitting on a chair. In both the configurations subjects were exposed to auditory and audio-visual stimuli presented with and without the haptic feedback. Results of the experiments provide a clear preference toward the simulations enhanced with haptic feedback showing that the haptic channel can lead to more realistic experiences in both interactive and noninteractive configurations. The majority of subjects clearly appreciated the added feedback. However, some subjects found the added feedback unpleasant. This might be due, on one hand, to the limits of the haptic simulation and, on the other hand, to the different individual desire to be involved in the simulations. Our findings can be applied to the context of physical navigation in multimodal virtual environments as well as to enhance the user experience of watching a movie or playing a video game.

Sound synthesis and evaluation of interactive footsteps and environmental sounds rendering for virtual reality applications.

We propose a system that affords real-time sound synthesis of footsteps on different materials. The system is based on microphones, which detect real footstep sounds from subjects, from which the ground reaction force (GRF) is estimated. Such GRF is used to control a sound synthesis engine based on physical models. Two experiments were conducted. In the first experiment, the ability of subjects to recognize the surface they were exposed to was assessed. In the second experiment, the sound synthesis engine was enhanced with environmental sounds. Results show that, in some conditions, adding a soundscape significantly improves the recognition of the simulated environment.