Representation of negative numbers: point estimation tasks using multi-reference sonification mappings.

In this study, we examine different approaches to the presentation of Y coordinates in mobile auditory graphs, including the representation of negative numbers. These studies involved both normally sighted and visually impaired users, as there are applications where normally sighted users might employ auditory graphs, such as the unseen monitoring of stocks, or fuel consumption in a car. Multi-reference sonification schemes are investigated as a means of improving the performance of mobile non-visual point estimation tasks. The results demonstrated that both populations are able to carry out point estimation tasks with a good level of performance when presented with auditory graphs using multiple reference tones. Additionally, visually impaired participants performed better on graphs represented in this format than normally sighted participants. This work also implements the component representation approach for negative numbers to represent the mapping by using the same positive mapping reference for the digit and adding a sign before the digit which leads to a better accuracy of the polarity sign. This work contributes to the areas of the design process of mobile auditory devices in human-computer interaction and proposed a methodological framework related to improving auditory graph performance in graph reproduction.

SONOICE! a Sonar-Voice dynamic user interface for assisting individuals with blindness and visual impairment in pinpointing elements in 2D tactile readers.

Pinpointing elements on large tactile surfaces is challenging for individuals with blindness and visual impairment (BVI) seeking to access two-dimensional (2D) information. This is particularly evident when using 2D tactile readers, devices designed to provide 2D information using static tactile representations with audio explanations. Traditional pinpointing methods, such as sighted assistance and trial-and-error, are limited and inefficient, while alternative pinpointing user interfaces (UI) are still emerging and need advancement. To address these limitations, we develop three distinct navigation UIs using a user-centred design approach: Sonar (proximity-radar sonification), Voice (direct clock-system speech instructions), and Sonoice, a new method that combines elements of both. The navigation UIs were incorporated into the Tactonom Reader device to conduct a trial study with ten BVI participants. Our UIs exhibited superior performance and higher user satisfaction than the conventional trial-and-error approach, showcasing scalability to varied assistive technology and their effectiveness regardless of graphic complexity. The innovative Sonoice approach achieved the highest efficiency in pinpointing elements, but user satisfaction was highest with the Sonar approach. Surprisingly, participant preferences varied and did not always align with their most effective strategy, underscoring the importance of accommodating individual user preferences and contextual factors when choosing between the three UIs. While more extensive training may reveal further differences between these UIs, our results emphasise the significance of offering diverse options to meet user needs. Altogether, the results provide valuable insights for improving the functionality of 2D tactile readers, thereby contributing to the future development of accessible technology.

A Wearable Personalised Sonification and Biofeedback Device to Enhance Movement Awareness.

Movement sonification has emerged as a promising approach for rehabilitation and motion control. Despite significant advancements in sensor technologies, challenges remain in developing cost-effective, user-friendly, and reliable systems for gait detection and sonification. This study introduces a novel wearable personalised sonification and biofeedback device to enhance movement awareness for individuals with irregular gait and posture. Through the integration of inertial measurement units (IMUs), MATLAB, and sophisticated audio feedback mechanisms, the device offers real-time, intuitive cues to facilitate gait correction and improve functional mobility. Utilising a single wearable sensor attached to the L4 vertebrae, the system captures kinematic parameters to generate auditory feedback through discrete and continuous tones corresponding to heel strike events and sagittal plane rotations. A preliminary test that involved 20 participants under various audio feedback conditions was conducted to assess the system's accuracy, reliability, and user synchronisation. The results indicate a promising improvement in movement awareness facilitated by auditory cues. This suggests a potential for enhancing gait and balance, particularly beneficial for individuals with compromised gait or those undergoing a rehabilitation process. This paper details the development process, experimental setup, and initial findings, discussing the integration challenges and future research directions. It also presents a novel approach to providing real-time feedback to participants about their balance, potentially enabling them to make immediate adjustments to their posture and movement. Future research should evaluate this method in varied real-world settings and populations, including the elderly and individuals with Parkinson's disease.

Online reach adjustments induced by real-time movement sonification.

Movement sonification can improve motor control in both healthy subjects (e.g., learning or refining a sport skill) and those with sensorimotor deficits (e.g., stroke patients and deafferented individuals). It is not known whether improved motor control and learning from movement sonification are driven by feedback-based real-time ("online") trajectory adjustments, adjustments to internal models over multiple trials, or both. We searched for evidence of online trajectory adjustments (muscle twitches) in response to movement sonification feedback by comparing the kinematics and error of reaches made with online (i.e., real-time) and terminal sonification feedback. We found that reaches made with online feedback were significantly more jerky than reaches made with terminal feedback, indicating increased muscle twitching (i.e., online trajectory adjustment). Using a between-subject design, we found that online feedback was associated with improved motor learning of a reach path and target over terminal feedback; however, using a within-subjects design, we found that switching participants who had learned with online sonification feedback to terminal feedback was associated with a decrease in error. Thus, our results suggest that, with our task and sonification, movement sonification leads to online trajectory adjustments which improve internal models over multiple trials, but which themselves are not helpful online corrections.

Musical Neurofeedback Advancements, Feedback Modalities, and Applications: A Systematic Review.

The field of EEG-Neurofeedback (EEG-NF) training has showcased significant promise in treating various mental disorders, while also emerging as a cognitive enhancer across diverse applications. The core principle of EEG-NF involves consciously guiding the brain in desired directions, necessitating active engagement in neurofeedback (NF) tasks over an extended period. Music listening tasks have proven to be effective stimuli for such training, influencing emotions, mood, and brainwave patterns. This has spurred the development of musical NF systems and training protocols. Despite these advancements, there exists a gap in systematic literature that comprehensively explores and discusses the various modalities of feedback mechanisms, its benefits, and the emerging applications. Addressing this gap, our review article presents a thorough literature survey encompassing studies on musical NF conducted over the past decade. This review highlights the several benefits and applications ranging from neurorehabilitation to therapeutic interventions, stress management, diagnostics of neurological disorders, and sports performance enhancement. While acknowledged for advantages and popularity of musical NF, there is an opportunity for growth in the literature in terms of the need for systematic randomized controlled trials to compare its effectiveness with other modalities across different tasks. Addressing this gap will involve developing standardized methodologies for studying protocols and optimizing parameters, presenting an exciting prospect for advancing the field.

Quantifying Changes in Dexterity as a Result of Piano Training in People with Parkinson's Disease.

People with Parkinson's disease often show deficits in dexterity, which, in turn, can lead to limitations in performing activities of daily life. Previous studies have suggested that training in playing the piano may improve or prevent a decline in dexterity in this population. In this pilot study, we tested three participants on a six-week, custom, piano-based training protocol, and quantified dexterity before and after the intervention using a sensor-enabled version of the nine-hole peg test, the box and block test, a test of finger synergies using unidimensional force sensors, and the Quantitative Digitography test using a digital piano, as well as selected relevant items from the motor parts of the MDS-Unified Parkinson's Disease Rating Scale (MDS-UPDRS) and the Parkinson's Disease Questionnaire (PDQ-39) quality of life questionnaire. The participants showed improved dexterity following the training program in several of the measures used. This pilot study proposes measures that can track changes in dexterity as a result of practice in people with Parkinson's disease and describes a potential protocol that needs to be tested in a larger cohort.

The Effects of Motionless Interventions Based on Visual-Auditory Instructions With Sonification on Learning a Rhythmic Motor Skill.

Our aim in this study was to investigate the effects of motionless interventions, based on visual-auditory integration with a sonification technique, on the learning a complex rhythmic motor skill. We recruited 22 male participants with high physical fitness and provided them four acquisition sessions in which to practice hurdle running, based on a visual-auditory instructional pattern. Next, we divided participants into three groups: visual-auditory, auditory, and control. In six sessions of motionless interventions, with no physical practice, participants in the visual-auditory group received a visual-auditory pattern similar to their experience during the acquisition period. The auditory group only listened to the sound of sonified movements of an expert hurdler, and the control group received no instructional interventions. Finally, participants in all three groups underwent post-intervention and transfer tests to determine their errors in the spatial and relative timing of their leading leg's knee angular displacement. Both visual-auditory and auditory groups had significantly less spatial error than the control group. However, there were no significant group differences in relative timing in any test phase. These results indicate that the use of the sonification technique in the form of visual-auditory instruction adapted to the athletes' needs benefitted perception-sensory capacities to improve motor skill learning.

Enhancing Agency in Posttraumatic Stress Disorder Therapies Through Sensorimotor Technologies.

Posttraumatic stress disorder (PTSD) is a significant public health concern, with only a third of patients recovering within a year of treatment. While PTSD often disrupts the sense of body ownership and sense of agency (SA), attention to the SA in trauma has been lacking. This perspective paper explores the loss of the SA in PTSD and its relevance in the development of symptoms. Trauma is viewed as a breakdown of the SA, related to a freeze response, with peritraumatic dissociation increasing the risk of PTSD. Drawing from embodied cognition, we propose an enactive perspective of PTSD, suggesting therapies that restore the SA through direct engagement with the body and environment. We discuss the potential of agency-based therapies and innovative technologies such as gesture sonification, which translates body movements into sounds to enhance the SA. Gesture sonification offers a screen-free, noninvasive approach that could complement existing trauma-focused therapies. We emphasize the need for interdisciplinary collaboration and clinical research to further explore these approaches in preventing and treating PTSD.

Inclusive multisensory science and immunology books for blind, low-vision and diverse-needs audiences.

This paper presents interdisciplinary research exploring the development of inclusive multisensory science books, communicating immunology data for blind, low-vision and diverse-needs audiences. The research adopted an inductive theory-building approach, practice-based art methods and music and design methods, leveraging the lived experience of a legally blind artist. The research also involved designers and scientists in a cocreation process, producing books that incorporate tactile artworks, Braille-inspired protein models, image sonification and interaction. Two multisensory book titles, "The Heroes Within You: A Multisensory Exploration of Infection and Immunity" and "My Goodness: A Multisensory Exploration of Nutrition and Immunity", were developed for the Monash Sensory Science 2023 Exhibition Day. The books offer an innovative way to make science and art more accessible and engaging, addressing the limitations of traditional museum methods. Feedback from audiences has been positive, emphasizing the fascination, sensory engagement and ease of understanding. This paper highlights the potential for an interdisciplinary and inclusive approach to science and art, demonstrating the value of multisensory books as tools for science communication. The findings highlight the positive reception of this novel approach and suggest its potential for broader applications, promoting inclusivity and accessibility.

Listening to life: Sonification for enhancing discovery in biological research.

Sonification, or the practice of generating sound from data, is a promising alternative or complement to data visualization for exploring research questions in the life sciences. Expressing or communicating data in the form of sound rather than graphs, tables, or renderings can provide a secondary information source for multitasking or remote monitoring purposes or make data accessible when visualizations cannot be used. While popular in astronomy, neuroscience, and geophysics as a technique for data exploration and communication, its potential in the biological and biotechnological sciences has not been fully explored. In this review, we introduce sonification as a concept, some examples of how sonification has been used to address areas of interest in biology, and the history of the technique. We then highlight a selection of biology-related publications that involve sonifications of DNA datasets and protein datasets, sonifications for data collection and interpretation, and sonifications aimed to improve science communication and accessibility. Through this review, we aim to show how sonification has been used both as a discovery tool and a communication tool and to inspire more life-science researchers to incorporate sonification into their own studies.

A review of sonification solutions in assistive systems for visually impaired people.

PURPOSE: Visually impaired people (VIP) find it challenging to understand and gain awareness of their surroundings. Most activities require the use of the auditory or tactile senses. As such, assistive systems which are capable of aiding visually impaired people to understand, navigate and form a mental representation of their environment are extensively being studied and developed. The aim of this paper is to provide insight regarding the characteristics, as well as the advantages and drawbacks of different types of sonification strategies in assistive systems, to assess their suitability for certain use-cases. MATERIALS AND METHODS: To this end, we reviewed a sizeable number of assistive solutions for VIP which provide a form of auditory feedback to the user, encountered in different scientific databases (Scopus, IEEE Xplore, ACM and Google Scholar) through direct searches and cross-referencing. RESULTS: We classified these solutions based on the aural information they provide to the VIP - alerts, guidance and information about their environment, be it spatial or semantic. Our intention is not to provide an exhaustive review, but to select representative implementations from recent literature that highlight the particularities of each sonification approach. CONCLUSIONS: Thus, anyone who is intent on developing an assistive solution will be able to choose the desired sonification class, being aware of the advantages/disadvantages and at the same time having a fairly wide selection of articles from the representative class.

The motivation behind this paper is to provide an overview of sonification strategies in the context of assistive systems for the visually impaired people.Whilst surveys and reviews which provide in-depth insights into assistive technologies and sonification exist, papers which provide a combined view of these topics are rather lacking.The analysis of the selected papers provides insight regarding the characteristics of different types of sonification strategies in assistive systems for visually impaired people and their suitability for certain use-cases.

Application of Convolutional Neural Network for Decoding of 12-Lead Electrocardiogram from a Frequency-Modulated Audio Stream (Sonified ECG).

Research of novel biosignal modalities with application to remote patient monitoring is a subject of state-of-the-art developments. This study is focused on sonified ECG modality, which can be transmitted as an acoustic wave and received by GSM (Global System for Mobile Communications) microphones. Thus, the wireless connection between the patient module and the cloud server can be provided over an audio channel, such as a standard telephone call or audio message. Patients, especially the elderly or visually impaired, can benefit from ECG sonification because the wireless interface is readily available, facilitating the communication and transmission of secure ECG data from the patient monitoring device to the remote server. The aim of this study is to develop an AI-driven algorithm for 12-lead ECG sonification to support diagnostic reliability in the signal processing chain of the audio ECG stream. Our methods present the design of two algorithms: (1) a transformer (ECG-to-Audio) based on the frequency modulation (FM) of eight independent ECG leads in the very low frequency band (300-2700 Hz); and (2) a transformer (Audio-to-ECG) based on a four-layer 1D convolutional neural network (CNN) to decode the audio ECG stream (10 s @ 11 kHz) to the original eight-lead ECG (10 s @ 250 Hz). The CNN model is trained in unsupervised regression mode, searching for the minimum error between the transformed and original ECG signals. The results are reported using the PTB-XL 12-lead ECG database (21,837 recordings), split 50:50 for training and test. The quality of FM-modulated ECG audio is monitored by short-time Fourier transform, and examples are illustrated in this paper and supplementary audio files. The errors of the reconstructed ECG are estimated by a popular ECG diagnostic toolbox. They are substantially low in all ECG leads: amplitude error (quartile range RMSE = 3-7 µV, PRD = 2-5.2%), QRS detector (Se, PPV > 99.7%), P-QRS-T fiducial points' time deviation (<2 ms). Low errors generalized across diverse patients and arrhythmias are a testament to the efficacy of the developments. They support 12-lead ECG sonification as a wireless interface to provide reliable data for diagnostic measurements by automated tools or medical experts.

Concurrent Supra-Postural Auditory-Hand Coordination Task Affects Postural Control: Using Sonification to Explore Environmental Unpredictability in Factors Affecting Fall Risk.

Clinical screening tests for balance and mobility often fall short of predicting fall risk. Cognitive distractors and unpredictable external stimuli, common in busy natural environments, contribute to this risk, especially in older adults. Less is known about the effects of upper sensory-motor coordination, such as coordinating one's hand with an external stimulus. We combined movement sonification and affordable inertial motion sensors to develop a task for the precise measurement and manipulation of full-body interaction with stimuli in the environment. In a double-task design, we studied how a supra-postural activity affected quiet stance. The supra-postural task consisted of rhythmic synchronization with a repetitive auditory stimulus. The stimulus was attentionally demanding because it was being modulated continuously. The participant's hand movement was sonified in real time, and their goal was to synchronize their hand movement with the stimulus. In the unpredictable condition, the tempo changed at random points in the trial. A separate sensor recorded postural fluctuations. Young healthy adults were compared to older adult (OA) participants without known risk of falling. The results supported the hypothesis that supra-postural coordination would entrain postural control. The effect was stronger in OAs, supporting the idea that diminished reserve capacities reduce the ability to isolate postural control from sensory-motor and cognitive activity.

uB-VisioGeoloc: An image sequences dataset of pedestrian navigation including geolocalised-inertial information and spatial sound rendering of the urban environment's obstacles.

The dataset proposed is a collection of pedestrian navigation data sequences combining visual and spatial information. The pedestrian navigation sequences are situations encountered by a pedestrian walking in an urban outdoor environment, such as moving on the sidewalk, navigating through a crowd, or crossing a street when the pedestrian light traffic is green. The acquired data are timestamped provided RGB-D images and are associated with GPS, and inertial data (acceleration, rotation). These recordings were acquired by separate processes, avoiding delays during their capture to guarantee a synchronisation between the moment of acquisition by the sensor and the moment of recording on the system. The acquisition was made in the city of Dijon, France, including narrow streets, wide avenues, and parks. Annotations of the RGB-D are also provided by bounding boxes indicating the position of relevant static or dynamic objects present in a pedestrian area, such as a tree, bench, or person. This pedestrian navigation dataset aims to support the development of smart mobile systems to assist visually impaired people in their daily movements in an outdoor environment. In this context, the visual data and localisation sequences we provide can be used to elaborate the appropriate visual processing methods to extract relevant information about the obstacles and their current positions on the path. Alongside the dataset, a visual-to-auditory substitution method has been employed to convert each image sequence into corresponding stereophonic sound files, allowing for comparison and evaluation. Synthetic sequences associated with the same information set are also provided based on the recordings of a displacement within the 3D model of a real place in Dijon.

Enhancing Perceptual-Motor Skills in Sports: The Role of Ecological Sounds.

Starting approximately from the beginning of the new millennium, a series of studies highlighted that auditory information deriving from biological motion can significantly influence the behavioral, cognitive and neurophysiological processes involved in the perception and execution of complex movements. In particular, it was observed that an appropriate use of sounds deriving from one's own movement promotes improvements in the movement execution itself. Two main approaches can be used, namely the sonification one or the ecological sound one; the former is based on the conversion of physiological and/or physical movement data into sound, while the latter is based on the use of auditory recordings of movement sounds as models. In the present article, some of the main applications of both approaches-especially the latter-to the domains of sport and motor rehabilitation are reviewed, with the aim of addressing two questions: Is it possible to consider rhythm as a Gestalt of human movement? If so, is it possible to build up cognitive strategies to improve/standardize movement performance from this Gestalt? As with most topics in science, a definitive answer is not possible, yet the evidence leads us to lean toward a positive answer to both questions.

The sound of silence: Quantification of typical absence seizures by sonifying EEG signals from a custom-built wearable device.

OBJECTIVE: To develop and validate a method for long-term (24-h) objective quantification of absence seizures in the EEG of patients with childhood absence epilepsy (CAE) in their real home environment using a wearable device (waEEG), comparing automatic detection methods with auditory recognition after seizure sonification. METHODS: The waEEG recording was acquired with two scalp electrodes. Automatic analysis was performed using previously validated software (Persyst® 14) and then fully reviewed by an experienced clinical neurophysiologist. The EEG data were converted into an audio file in waveform format with a 60-fold time compression factor. The sonified EEG was listened to by three inexperienced observers and the number of seizures and the processing time required for each data set were recorded blind to other data. Quantification of seizures from the patient diary was also assessed. RESULTS: Eleven waEEG recordings from seven CAE patients with an average age of 8.18 ± 1.60 years were included. No differences in the number of seizures were found between the recordings using automated methods and expert audio assessment, with significant correlations between methods (ρ > .89, p < .001) and between observers (ρ > .96, p < .001). For the entire data set, the audio assessment yielded a sensitivity of .830 and a precision of .841, resulting in an F1 score of .835. SIGNIFICANCE: Auditory waEEG seizure detection by lay medical personnel provided similar accuracy to post-processed automatic detection by an experienced clinical neurophysiologist, but in a less time-consuming procedure and without the need for specialized resources. Sonification of long-term EEG recordings in CAE provides a user-friendly and cost-effective clinical workflow for quantifying seizures in clinical practice, minimizing human and technical constraints.

Evaluation of different feedback designs for target guidance in human controlled robotic cranes: A comparison between high and low performance groups.

Labour shortages and costly operator training are driving the need for digital on-board robotic crane operator support in forestry and construction. This simulator study investigated the effects of sonification (auditory, pitch/loudness) and continuous visual (brightness/size) feedback on aiming movements with a robotic crane for low and high performers. The feedback was designed non-linear and linear. Thirty-six participants controlled a robotic crane bimanually using joysticks across 320 movements. Performance and skill indicators (movement time, accuracy, trajectory, smoothness) as well as satisfaction, and usefulness were assessed. Low-performing participants showed higher movement accuracy, particularly with non-linear pitch feedback compared to visual feedback. High performers exhibited no significant performance improvement in movement time, accuracy, or smoothness. There was no effect of linear or non-linear mapping of the feedback. Additionally, perceived satisfaction was lower with auditory than visual feedback. These results suggest that real-time auditory feedback can enhance operator accuracy whereas acceptance remains challenging.

(Pre)analytical considerations concerning the analysis of synovial calprotectin.

OBJECTIVES: Several studies have demonstrated that synovial calprotectin is a highly accurate biomarker in diagnosing periprosthetic joint infections (PJI). Assuring reliability is of great importance and coincides with adequate preanalytical handling. This study focuses on potentially interfering factors. METHODS: To assess the stability of synovial calprotectin, the effect of time, storage temperature, EDTA, freeze-thaw cycles, viscosity, and blood and lipid contamination was investigated. In the blood and lipid contamination experiments, hemolyzed and non-hemolyzed blood, homogenized adipose tissue, intralipid and chylomicrons were added. The effect of viscosity was investigated using freeze-thaw cycles, enzymatic pretreatment and sonification. RESULTS: No effect on synovial calprotectin levels was observed in synovial samples kept at room temperature compared to samples kept at 4 °C for up to seven days of storage. Freeze-thaw cycles did not result in significantly different calprotectin levels, although samples without EDTA resulted in higher recoveries after 1 and 2 freeze-thaw cycles. Blood and lipid contamination did not interfere with accurate synovial calprotectin analysis. Sample pretreatment to reduce sample viscosity by pretreating samples with DNAse and/or hyaluronidase did not influence calprotectin analysis. Sonification, however, resulted in increased calprotectin values. CONCLUSIONS: Synovial calprotectin is a stable biomarker and its analysis is not easily influenced by potential interfering factors.

Movement Sonification Techniques to Improve Balance in Parkinson's Disease: A Pilot Randomized Controlled Trial.

BACKGROUND: Movement sonification has been recently introduced into the field of neuromotor rehabilitation alongside Neurologic Music Therapy and music-based interventions. This study introduces the use of musical auditory cues encompassing the melodic-harmonic aspect of music. METHODS: Nineteen patients with Parkinson's disease were randomly assigned to the experimental (n = 10) and control (n = 9) groups and underwent thrice-weekly sessions of the same gait training program, with or without sonification. Functional and motor parameters, as well as fatigue, quality of life, and the impact of intervention on patients' well-being, were assessed at baseline (PRE), the end of treatment (POST), and at follow-up (FU). Between-group differences were assessed for each outcome measure using linear mixed-effects models. The outcome measure was entered as the dependent variable, group and time as fixed effects, and time by group as the interaction effect. RESULTS: Mini BESTest and Dynamic Gait Index scores significantly improved in the experimental group (p = 0.01 and p = 0.03, respectively) from PRE to FU, demonstrating a significant impact of the sonification treatment on balance. No other significant differences were observed in the outcome measures. CONCLUSIONS: Larger sample sizes are needed to confirm the effectiveness of sonification approaches in Parkinson's disease, as well as in other neurological disorders.

Towards a unified terminology for sonification and visualization.

Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data differ. Over the past 30 years, the sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times. A design theory of audio-visual analysis would be a relevant step in this direction. An indispensable foundation for this endeavor is a terminology describing the combined design space. To build a bridge between the domains, we adopt three of the established theoretical constructs from visualization theory for the field of sonification. The three constructs are the spatial substrate, the visual mark, and the visual channel. In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. Auditory channels are encoded into auditory marks to convey information. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.