Effect of sonification types in upper-limb movement: a quantitative and qualitative study in hemiparetic and healthy participants.

BACKGROUND: Movement sonification, the use of real-time auditory feedback linked to movement parameters, have been proposed to support rehabilitation. Nevertheless, if promising results have been reported, the effect of the type of sound used has not been studied systematically. The aim of this study was to investigate in a single session the effect of different types of sonification both quantitatively and qualitatively on patients with acquired brain lesions and healthy participants. METHODS: An experimental setup enabling arm sonification was developed using three different categories of sonification (direct sound modulation, musical interaction, and soundscape). Simple moving forward movements performed while sliding on a table with both arms were investigated with all participants. Quantitative analysis on the movement timing were performed considering various parameters (sound condition, affected arm and dominance, sonification categories). Qualitative analysis of semi-structured interviews were also conducted, as well as neuropsychological evaluation of music perception. RESULTS: For both the patient and healthy groups (15 participants each), average duration for performing the arm movement is significantly longer with sonification compared to the no-sound condition (p < 0.001). Qualitative analysis of semi-structured interviews revealed different aspects of motivational and affective aspects of sonification. Most participants of both groups preferred to complete the task with sound (29 of 30 participants), and described the experience as playful (22 of 30 participants). More precisely, the soundscape (nature sounds) was the most constantly preferred (selected first by 14 of 30 participants). CONCLUSION: Overall, our results confirm that the sonification has an effect on the temporal execution of the movement during a single-session. Globally, sonification is welcomed by the participants, and we found convergent and differentiated appreciations of the different sonification types.

Describing movement learning using metric learning.

Analysing movement learning can rely on human evaluation, e.g. annotating video recordings, or on computing means in applying metrics on behavioural data. However, it remains challenging to relate human perception of movement similarity to computational measures that aim at modelling such similarity. In this paper, we propose a metric learning method bridging the gap between human ratings of movement similarity in a motor learning task and computational metric evaluation on the same task. It applies metric learning on a Dynamic Time Warping algorithm to derive an optimal set of movement features that best explain human ratings. We evaluated this method on an existing movement dataset, which comprises videos of participants practising a complex gesture sequence toward a target template, as well as the collected data that describes the movements. We show that it is possible to establish a linear relationship between human ratings and our learned computational metric. This learned metric can be used to describe the most salient temporal moments implicitly used by annotators, as well as movement parameters that correlate with motor improvements in the dataset. We conclude with possibilities to generalise this method for designing computational tools dedicated to movement annotation and evaluation of skill learning.

Effects of pitch and musical sounds on body-representations when moving with sound.

The effects of music on bodily movement and feelings, such as when people are dancing or engaged in physical activity, are well-documented-people may move in response to the sound cues, feel powerful, less tired. How sounds and bodily movements relate to create such effects? Here we deconstruct the problem and investigate how different auditory features affect people's body-representation and feelings even when paired with the same movement. In three experiments, participants executed a simple arm raise synchronised with changing pitch in simple tones (Experiment 1), rich musical sounds (Experiment 2) and within different frequency ranges (Experiment 3), while we recorded indirect and direct measures on their movement, body-representations and feelings. Changes in pitch influenced people's general emotional state as well as the various bodily dimensions investigated-movement, proprioceptive awareness and feelings about one's body and movement. Adding harmonic content amplified the differences between ascending and descending sounds, while shifting the absolute frequency range had a general effect on movement amplitude, bodily feelings and emotional state. These results provide new insights in the role of auditory and musical features in dance and exercise, and have implications for the design of sound-based applications supporting movement expression, physical activity, or rehabilitation.

Modulation of ellipses drawing by sonification.

Most studies on the regulation of speed and trajectory during ellipse drawing have used visual feedback. We used online auditory feedback (sonification) to induce implicit movement changes independently from vision. The sound was produced by filtering a pink noise with a band-pass filter proportional to movement speed. The first experiment was performed in 2D. Healthy participants were asked to repetitively draw ellipses during 45 s trials whilst maintaining a constant sonification pattern (involving pitch variations during the cycle). Perturbations were produced by modifying the slope of the mapping without informing the participants. All participants adapted spontaneously their speed: they went faster if the slope decreased and slower if it increased. Higher velocities were achieved by increasing both the frequency of the movements and the perimeter of the ellipses, but slower velocities were achieved mainly by decreasing the perimeter of the ellipses. The shape and the orientation of the ellipses were not significantly altered. The analysis of the speed-curvature power law parameters showed consistent modulations of the speed gain factor, while the exponent remained stable. The second experiment was performed in 3D and showed similar results, except that the main orientation of the ellipse also varied with the changes in speed. In conclusion, this study demonstrated implicit modulation of movement speed by sonification and robust stability of the ellipse geometry. Participants appeared to limit the decrease in movement frequency during slowing down to maintain a rhythmic and not discrete motor regimen.

Sensus Communis: Some Perspectives on the Origins of Non-synchronous Cross-Sensory Associations.

Adults readily make associations between stimuli perceived consecutively through different sense modalities, such as shapes and sounds. Researchers have only recently begun to investigate such correspondences in infants but only a handful of studies have focused on infants less than a year old. Are infants able to make cross-sensory correspondences from birth? Do certain correspondences require extensive real-world experience? Some studies have shown that newborns are able to match stimuli perceived in different sense modalities. Yet, the origins and mechanisms underlying these abilities are unclear. The present paper explores these questions and reviews some hypotheses on the emergence and early development of cross-sensory associations and their possible links with language development. Indeed, if infants can perceive cross-sensory correspondences between events that share certain features but are not strictly contingent or co-located, one may posit that they are using a "sixth sense" in Aristotle's sense of the term. And a likely candidate for explaining this mechanism, as Aristotle suggested, is movement.

Dissociable effects of practice variability on learning motor and timing skills.

Motor skill acquisition inherently depends on the way one practices the motor task. The amount of motor task variability during practice has been shown to foster transfer of the learned skill to other similar motor tasks. In addition, variability in a learning schedule, in which a task and its variations are interweaved during practice, has been shown to help the transfer of learning in motor skill acquisition. However, there is little evidence on how motor task variations and variability schedules during practice act on the acquisition of complex motor skills such as music performance, in which a performer learns both the right movements (motor skill) and the right time to perform them (timing skill). This study investigated the impact of rate (tempo) variability and the schedule of tempo change during practice on timing and motor skill acquisition. Complete novices, with no musical training, practiced a simple musical sequence on a piano keyboard at different rates. Each novice was assigned to one of four learning conditions designed to manipulate the amount of tempo variability across trials (large or small tempo set) and the schedule of tempo change (randomized or non-randomized order) during practice. At test, the novices performed the same musical sequence at a familiar tempo and at novel tempi (testing tempo transfer), as well as two novel (but related) sequences at a familiar tempo (testing spatial transfer). We found that practice conditions had little effect on learning and transfer performance of timing skill. Interestingly, practice conditions influenced motor skill learning (reduction of movement variability): lower temporal variability during practice facilitated transfer to new tempi and new sequences; non-randomized learning schedule improved transfer to new tempi and new sequences. Tempo (rate) and the sequence difficulty (spatial manipulation) affected performance variability in both timing and movement. These findings suggest that there is a dissociable effect of practice variability on learning complex skills that involve both motor and timing constraints.

Rising tones and rustling noises: Metaphors in gestural depictions of sounds.

Communicating an auditory experience with words is a difficult task and, in consequence, people often rely on imitative non-verbal vocalizations and gestures. This work explored the combination of such vocalizations and gestures to communicate auditory sensations and representations elicited by non-vocal everyday sounds. Whereas our previous studies have analyzed vocal imitations, the present research focused on gestural depictions of sounds. To this end, two studies investigated the combination of gestures and non-verbal vocalizations. A first, observational study examined a set of vocal and gestural imitations of recordings of sounds representative of a typical everyday environment (ecological sounds) with manual annotations. A second, experimental study used non-ecological sounds whose parameters had been specifically designed to elicit the behaviors highlighted in the observational study, and used quantitative measures and inferential statistics. The results showed that these depicting gestures are based on systematic analogies between a referent sound, as interpreted by a receiver, and the visual aspects of the gestures: auditory-visual metaphors. The results also suggested a different role for vocalizations and gestures. Whereas the vocalizations reproduce all features of the referent sounds as faithfully as vocally possible, the gestures focus on one salient feature with metaphors based on auditory-visual correspondences. Both studies highlighted two metaphors consistently shared across participants: the spatial metaphor of pitch (mapping different pitches to different positions on the vertical dimension), and the rustling metaphor of random fluctuations (rapidly shaking of hands and fingers). We interpret these metaphors as the result of two kinds of representations elicited by sounds: auditory sensations (pitch and loudness) mapped to spatial position, and causal representations of the sound sources (e.g. rain drops, rustling leaves) pantomimed and embodied by the participants' gestures.

Continuous Auditory Feedback of Eye Movements: An Exploratory Study toward Improving Oculomotor Control.

As eye movements are mostly automatic and overtly generated to attain visual goals, individuals have a poor metacognitive knowledge of their own eye movements. We present an exploratory study on the effects of real-time continuous auditory feedback generated by eye movements. We considered both a tracking task and a production task where smooth pursuit eye movements (SPEM) can be endogenously generated. In particular, we used a visual paradigm which enables to generate and control SPEM in the absence of a moving visual target. We investigated whether real-time auditory feedback of eye movement dynamics might improve learning in both tasks, through a training protocol over 8 days. The results indicate that real-time sonification of eye movements can actually modify the oculomotor behavior, and reinforce intrinsic oculomotor perception. Nevertheless, large inter-individual differences were observed preventing us from reaching a strong conclusion on sensorimotor learning improvements.

Investigating three types of continuous auditory feedback in visuo-manual tracking.

The use of continuous auditory feedback for motor control and learning is still understudied and deserves more attention regarding fundamental mechanisms and applications. This paper presents the results of three experiments studying the contribution of task-, error-, and user-related sonification to visuo-manual tracking and assessing its benefits on sensorimotor learning. First results show that sonification can help decreasing the tracking error, as well as increasing the energy in participant's movement. In the second experiment, when alternating feedback presence, the user-related sonification did not show feedback dependency effects, contrary to the error and task-related feedback. In the third experiment, a reduced exposure of 50% diminished the positive effect of sonification on performance, whereas the increase of the average energy with sound was still significant. In a retention test performed on the next day without auditory feedback, movement energy was still superior for the groups previously trained with the feedback. Although performance was not affected by sound, a learning effect was measurable in both sessions and the user-related group improved its performance also in the retention test. These results confirm that a continuous auditory feedback can be beneficial for movement training and also show an interesting effect of sonification on movement energy. User-related sonification can prevent feedback dependency and increase retention. Consequently, sonification of the user's own motion appears as a promising solution to support movement learning with interactive feedback.

Sensori-Motor Learning with Movement Sonification: Perspectives from Recent Interdisciplinary Studies.

This article reports on an interdisciplinary research project on movement sonification for sensori-motor learning. First, we describe different research fields which have contributed to movement sonification, from music technology including gesture-controlled sound synthesis, sonic interaction design, to research on sensori-motor learning with auditory-feedback. In particular, we propose to distinguish between sound-oriented tasks and movement-oriented tasks in experiments involving interactive sound feedback. We describe several research questions and recently published results on movement control, learning and perception. In particular, we studied the effect of the auditory feedback on movements considering several cases: from experiments on pointing and visuo-motor tracking to more complex tasks where interactive sound feedback can guide movements, or cases of sensory substitution where the auditory feedback can inform on object shapes. We also developed specific methodologies and technologies for designing the sonic feedback and movement sonification. We conclude with a discussion on key future research challenges in sensori-motor learning with movement sonification. We also point out toward promising applications such as rehabilitation, sport training or product design.

From ear to hand: the role of the auditory-motor loop in pointing to an auditory source.

Studies of the nature of the neural mechanisms involved in goal-directed movements tend to concentrate on the role of vision. We present here an attempt to address the mechanisms whereby an auditory input is transformed into a motor command. The spatial and temporal organization of hand movements were studied in normal human subjects as they pointed toward unseen auditory targets located in a horizontal plane in front of them. Positions and movements of the hand were measured by a six infrared camera tracking system. In one condition, we assessed the role of auditory information about target position in correcting the trajectory of the hand. To accomplish this, the duration of the target presentation was varied. In another condition, subjects received continuous auditory feedback of their hand movement while pointing to the auditory targets. Online auditory control of the direction of pointing movements was assessed by evaluating how subjects reacted to shifts in heard hand position. Localization errors were exacerbated by short duration of target presentation but not modified by auditory feedback of hand position. Long duration of target presentation gave rise to a higher level of accuracy and was accompanied by early automatic head orienting movements consistently related to target direction. These results highlight the efficiency of auditory feedback processing in online motor control and suggest that the auditory system takes advantages of dynamic changes of the acoustic cues due to changes in head orientation in order to process online motor control. How to design an informative acoustic feedback needs to be carefully studied to demonstrate that auditory feedback of the hand could assist the monitoring of movements directed at objects in auditory space.

Quantitation and mapping of tissue optical properties using modulated imaging.

We describe the development of a rapid, noncontact imaging method, modulated imaging (MI), for quantitative, wide-field characterization of optical absorption and scattering properties of turbid media. MI utilizes principles of frequency-domain sampling and model-based analysis of the spatial modulation transfer function (s-MTF). We present and compare analytic diffusion and probabilistic Monte Carlo models of diffuse reflectance in the spatial frequency domain. Next, we perform MI measurements on tissue-simulating phantoms exhibiting a wide range of l values (0.5 mm to 3 mm) and (micro(s) (')micro(a)) ratios (8 to 500), reporting an overall accuracy of approximately 6% and 3% in absorption and reduced scattering parameters, respectively. Sampling of only two spatial frequencies, achieved with only three camera images, is found to be sufficient for accurate determination of the optical properties. We then perform MI measurements in an in vivo tissue system, demonstrating spatial mapping of the absorption and scattering optical contrast in a human forearm and dynamic measurements of a forearm during venous occlusion. Last, metrics of spatial resolution are assessed through both simulations and measurements of spatially heterogeneous phantoms.

Modulated imaging: quantitative analysis and tomography of turbid media in the spatial-frequency domain.

Experiments performed on turbid phantoms demonstrate that spatially modulated illumination facilitates quantitative wide-field optical property mapping and tomographic imaging in turbid media.

Use of the delta-P1 approximation for recovery of optical absorption, scattering, and asymmetry coefficients in turbid media.

We introduce a robust method to recover optical absorption, reduced scattering, and single-scattering asymmetry coefficients (microa, micro's, g1) of infinite turbid media over a range of (micro's/microa) spanning 3 orders of magnitude. This is accomplished through the spatially resolved measurement of irradiance at source-detector separations spanning 0.25-8 transport mean free paths (l*). These measurements are rapidly processed by a multistaged nonlinear optimization algorithm in which the measured irradiances are compared with predictions given by the delta-P1 variant of the diffusion approximation to the Boltzmann transport equation. The ability of the delta-P1 model to accurately describe radiative transport within media of arbitrary albedo and on spatial scales comparable to l* is the key element enabling the separation of g1 from micro's.

Sampling tissue volumes using frequency-domain photon migration.

We demonstrate the use of Monte Carlo simulations to generate photon scattering density functions (PSDFs) that represent the tissue volume sampled by steady-state and frequency-domain photon migration. We use these results to illustrate how scaling laws can be developed to determine the mean sampling depth of the multiply scattered photons detected by photon migration methods that remain valid outside the bounds of the standard diffusion approximation, i.e., at small source-detector separations and in media where the optical absorption is significant relative to scattering. Using both the PSDF computation and the newly formulated scaling laws, we focus on a comprehensive description of the effects of source modulation frequency, optical absorption, and source-detector separation on the depth of the sampled tissue volume as well as the sensitivity of frequency-domain photon migration measurements to the presence of a localized absorption heterogeneity.

Monitoring neoadjuvant chemotherapy in breast cancer using quantitative diffuse optical spectroscopy: a case study.

Presurgical chemotherapy is widely used in the treatment of locally advanced breast cancer. Monitoring the response to therapy can improve survival and reduce morbidity. We employ a noninvasive, near-infrared method based on diffuse optical spectroscopy (DOS) to quantitatively monitor tumor response to neoadjuvant chemotherapy. DOS was used to monitor tumor response in one patient with locally advanced breast cancer throughout the course of her therapy. Measurements were performed prior to doxorubicin-cyclophosphamide therapy and at several time points over the course of three treatment cycles (68 days). Our results show strong tumor to normal (T/N) tissue contrast in total hemoglobin concentration (T/N=2.4), water fraction (T/N=6.9), tissue hemoglobin oxygen saturation, S(t)O(2) (T/N=0.9), and lipid fraction (T/N=0.7) prior to treatment. Over a 10-week period, the peak total hemoglobin and water dropped 56 and 67%, respectively. Lipid content nearly returned to baseline (T/N =0.9) while S(t)O(2) exceeded pretreatment levels (T/N =1.5). Approximately half of the hemoglobin and water changes occurred within 5 days of treatment (26 and 37%, respectively). These data suggest that noninvasive, quantitative optical methods that characterize tumor physiology may be useful in assessing and optimizing individual response to neoadjuvant chemotherapy.

In vivo endoscopic tissue diagnostics based on spectroscopic absorption, scattering, and phase function properties.

A fast spectroscopic system for superficial and local determination of the absorption and scattering properties of tissue (480 to 950 nm) is described. The probe can be used in the working channel of an endoscope. The scattering properties include the reduced scattering coefficient and a parameter of the phase function called gamma, which depends on its first two moments. The inverse problem algorithm is based on the fit of absolute reflectance measurements to cubic B-spline functions derived from the interpolation of a set of Monte Carlo simulations. The algorithm's robustness was tested with simulations altered with various amounts of noise. The method was also assessed on tissue phantoms of known optical properties. Finally, clinical measurements performed endoscopically in vivo in the stomach of human subjects are presented. The absorption and scattering properties were found to be significantly different in the antrum and in the fundus and are correlated with histopathologic observations. The method and the instrument show promise for noninvasive tissue diagnostics of various epithelia.

In vivo quantification of optical contrast agent dynamics in rat tumors by use of diffuse optical spectroscopy with magnetic resonance imaging coregistration.

We present a study of the dynamics of optical contrast agents indocyanine green (ICG) and methylene blue (MB) in an adenocarcinoma rat tumor model. Measurements are conducted with a combined frequency-domain and steady-state optical technique that facilitates rapid measurement of tissue absorption in the 650-1000-nm spectral region. Tumors were also imaged by use of contrast-enhanced magnetic resonance imaging (MRI) and coregistered with the location of the optical probe. The absolute concentrations of contrast agent, oxyhemoglobin, deoxyhemoglobin, and water are measured simultaneously each second for approximately 10 min. The differing tissue uptake kinetics of ICG and MB in these late-stage tumors arise from differences in their effective molecular weights. ICG, because of its binding to plasma proteins, behaves as a macromolecular contrast agent with a low vascular permeability. A compartmental model describing ICG dynamics is used to quantify physiologic parameters related to capillary permeability. In contrast, MB behaves as a small-molecular-weight contrast agent that leaks rapidly from the vasculature into the extravascular, extracellular space, and is sensitive to blood flow and the arterial input function.

Coregistration of diffuse optical spectroscopy and magnetic resonance imaging in a rat tumor model.

We report coregistration of near-infrared diffuse optical spectroscopy (DOS) and magnetic resonance imaging (MRI) for the study of animal model tumors. A combined broadband steady-state and frequency-domain apparatus was used to determine tissue oxyhemoglobin, deoxyhemoglobin, and water concentration locally in tumors. Simultaneous MRI coregistration provided structural (T2-weighted) and contrast-enhanced images of the tumor that were correlated with the optical measurements. By use of Monte Carlo simulations, the optically sampled volume was superimposed on the MR images, showing precisely which tissue structure was probed optically. DOS and MRI coregistration measurements were performed on seven rats over 20 days and were separated into three tumor tissue classifications: viable, edematous, and necrotic. A ratio of water concentration to total hemoglobin concentration, as measured optically, was performed for each tissue type and showed values for edematous tissue to be greater than viable tissue (1.2 +/- 0.49 M/microM versus 0.48 +/- 0.15 M/microM). Tissue hemoglobin oxygen saturation (StO2) also showed a large variation between tissue types: viable tissue had an optically measured StO2 value of 61 +/- 5%, whereas StO2 determined for necrotic tissue was 43 +/- 6%.

A new optical method for the non-invasive detection of minimal tissue alterations.

Histological analysis, which is used to detect and diagnose most tissue alterations, requires an invasive biopsy procedure and a time-consuming tissue treatment, which limit its efficiency in providing rapid, cost-effective diagnosis and hinder the longitudinal study of tissue alteration. To address these limitations, we have developed a novel procedure, using the features of elastic-scattering spectroscopy, for a real-time, non-invasive analysis of tissues. We have tested whether this approach can detect in vivo changes in mouse skin induced by a single exposure to either complete Freund's adjuvant or 12-O-tetradecanoylphorbol-13-acetate, two drugs known to induce discrete alterations of epidermis and dermis, without obvious changes on the skin surface. Here we report that the evaluation of localized absorption and reduced scattering coefficients permitted the detection of changes in skin regions that showed histological alterations, but not in regions which failed to be modified by the drugs. Results show that the optical in vivo analysis of small regions has sufficient specificity and sensitivity to detect minimal alterations of superficial tissues. In view of the prominent involvement of mucosal alterations in most human diseases, including carcinomas, the method provides a useful complement to standard biopsy, notably for the in vivo screening of early in situ epithelial alterations.