Climate data sonification and visualization: An analysis of topics, aesthetics, and characteristics in 32 recent projects.

Introduction: It has proven a hard challenge to stimulate climate action with climate data. While scientists communicate through words, numbers, and diagrams, artists use movement, images, and sound. Sonification, the translation of data into sound, and visualization, offer techniques for representing climate data with often innovative and exciting results. The concept of sonification was initially defined in terms of engineering, and while this view remains dominant, researchers increasingly make use of knowledge from electroacoustic music (EAM) to make sonifications more convincing. Methods: The Aesthetic Perspective Space (APS) is a two-dimensional model that bridges utilitarian-oriented sonification and music. We started with a review of 395 sonification projects, from which a corpus of 32 that target climate change was chosen; a subset of 18 also integrate visualization of the data. To clarify relationships with climate data sources, we determined topics and subtopics in a hierarchical classification. Media duration and lexical diversity in descriptions were determined. We developed a protocol to span the APS dimensions, Intentionality and Indexicality, and evaluated its circumplexity. Results: We constructed 25 scales to cover a range of qualitative characteristics applicable to sonification and sonification-visualization projects, and through exploratory factor analysis, identified five essential aspects of the project descriptions, labeled Action, Technical, Context, Perspective, and Visualization. Through linear regression modeling, we investigated the prediction of aesthetic perspective from essential aspects, media duration, and lexical diversity. Significant regressions across the corpus were identified for Perspective (ß = 0.41***) and lexical diversity (ß = -0.23*) on Intentionality, and for Perspective (ß = 0.36***) and Duration (logarithmic; ß = -0.25*) on Indexicality. Discussion: We discuss how these relationships play out in specific projects, also within the corpus subset that integrated data visualization, as well as broader implications of aesthetics on design techniques for multimodal representations aimed at conveying scientific data. Our approach is informed by the ongoing discussion in sound design and auditory perception research communities on the relationship between sonification and EAM. Through its analysis of topics, qualitative characteristics, and aesthetics across a range of projects, our study contributes to the development of empirically founded design techniques, applicable to climate science communication and other fields.

The Multi-hub Academic Conference: Global, Inclusive, Culturally Diverse, Creative, Sustainable.

New conference formats are emerging in response to COVID-19 and climate change. Virtual conferences are sustainable and inclusive regardless of participant mobility (financial means, caring commitments, disability), but lack face-to-face contact. Hybrid conferences (physical meetings with additional virtual presentations) tend to discriminate against non-fliers and encourage unsustainable flying. Multi-hub conferences mix real and virtual interactions during talks and social breaks and are distributed across nominally equal hubs. We propose a global multi-hub solution in which all hubs interact daily in real time with all other hubs in parallel sessions by internet videoconferencing. Conference sessions are confined to three equally-spaced 4-h UTC timeslots. Local programs comprise morning and afternoon/evening sessions (recordings from night sessions can be watched later). Three reference hubs are located exactly 8 h apart; additional hubs are within 2 h and their programs are aligned with the closest reference hub. The conference experience at each hub depends on the number of local participants and the time difference to the nearest reference. Participants are motivated to travel to the nearest hub. Mobility-based discrimination is minimized. Lower costs facilitate diversity, equity, and inclusion. Academic quality, creativity, enjoyment, and low-carbon sustainability are simultaneously promoted.

Soundscape in Times of Change: Case Study of a City Neighbourhood During the COVID-19 Lockdown.

The coronavirus disease 2019 (COVID-19) lockdown meant a greatly reduced social and economic activity. Sound is of major importance to people's perception of the environment, and some remarked that the soundscape was changing for the better. But are these anecdotal reports based in truth? Has traffic noise from cars and airplanes really gone down, so that more birdsong can be heard? Have socially distanced people quietened down? This article presents a case study of the human perception of environmental sounds in an urban neighborhood in the Basque Country between 15 March and 25 May 2020. The social restrictions imposed through national legislation divided the 69-day period into three phases. We collected observations, field audio recordings, photography, and diary notes on 50 days. Experts in soundscape and architecture were presented with the recordings, in randomized order, and made two separate perceptual analyses. One group (N = 11) rated the recordings for pleasantness and eventfulness using an adapted version of the Swedish Soundscape Quality Protocol, and a partly overlapping group (N = 12) annotated perceived sound events with free-form semantic labels. The labels were systematically classified into a four-level Taxonomy of Sound Sources, allowing an estimation of the relative amounts of Natural, Human, and Technological sounds. Loudness and three descriptors developed for bioacoustics were extracted computationally. Analysis showed that Eventfulness, Acoustic Complexity, and Acoustic Richness increased significantly over the time period, while the amount of Technological sounds decreased. These observations were interpreted as reflecting changes in people's outdoor activities and behavior over the whole 69-day period, evidenced in an increased presence of Human sounds of voices and walking, and a significant shift from motorized vehicles toward personal mobility devices, again evidenced by perceived sounds. Quantitative results provided a backdrop against which qualitative analyses of diary notes and observations were interpreted in relation to the restrictions and the architectural specifics of the site. An integrated analysis of all sources pointed at the temporary suspension of human outdoor activity as the main reason for such a change. In the third phase, the progressive return of street life and the usage of personal mobility vehicles seemed to be responsible for a clear increase in Eventfulness and Loudness even in the context of an overall decrease of Technological sounds. Indoor human activity shared through open windows and an increased presence of birdsong emerge as a novel characteristic element of the local urban soundscape. We discuss how such changes in the acoustic environment of the site, in acoustic measurements and as perceived by humans, point toward the soundscape being a crucial component of a comprehensive urban design strategy that aims to improve health and quality of life for increasingly large and dense populations in the future.

Real and Imagined Smellscapes.

The smellscape is the olfactory environment as perceived and understood, consisting of odours and scents from multiple smell sources. To what extent can audiovisual information evoke the smells of a real, complex, and multimodal environment? To investigate smellscape imagination, we compared results from two studies. In the first, onsite participants (N = 15) made a sensory walk through seven locations of an open-air market. In the second, online participants (N = 53) made a virtual walk through the same locations reproduced with audio and video recordings. Responses in the form of free-form verbal annotations, ratings with semantic scales, and a 'smell wheel', were analysed for environmental quality, smell source type and strength, and hedonic tone. The degree of association between real and imagined smellscapes was measured through canonical correlation analysis. Hedonic tone, as expressed through frequency counts of keywords in free-form annotations was significantly associated, suggesting that smell sources might generally be correctly inferred from audiovisual information, when such imagination is required. On the other hand, onsite ratings of olfactory quality were not significantly associated with online ratings of audiovisual reproductions, when participants were not specifically asked to imagine smells. We discuss findings in the light of cross-modal association, categorisation, and memory recall of smells.

Cross-Modal Perception of Noise-in-Music: Audiences Generate Spiky Shapes in Response to Auditory Roughness in a Novel Electroacoustic Concert Setting.

Noise has become integral to electroacoustic music aesthetics. In this paper, we define noise as sound that is high in auditory roughness, and examine its effect on cross-modal mapping between sound and visual shape in participants. In order to preserve the ecological validity of contemporary music aesthetics, we developed Rama, a novel interface, for presenting experimentally controlled blocks of electronically generated sounds that varied systematically in roughness, and actively collected data from audience interaction. These sounds were then embedded as musical drones within the overall sound design of a multimedia performance with live musicians, Audience members listened to these sounds, and collectively voted to create the shape of a visual graphic, presented as part of the audio-visual performance. The results of the concert setting were replicated in a controlled laboratory environment to corroborate the findings. Results show a consistent effect of auditory roughness on shape design, with rougher sounds corresponding to spikier shapes. We discuss the implications, as well as evaluate the audience interface.

Colour Association with Music Is Mediated by Emotion: Evidence from an Experiment Using a CIE Lab Interface and Interviews.

Crossmodal associations may arise at neurological, perceptual, cognitive, or emotional levels of brain processing. Higher-level modal correspondences between musical timbre and visual colour have been previously investigated, though with limited sets of colour. We developed a novel response method that employs a tablet interface to navigate the CIE Lab colour space. The method was used in an experiment where 27 film music excerpts were presented to participants (n = 22) who continuously manipulated the colour and size of an on-screen patch to match the music. Analysis of the data replicated and extended earlier research, for example, that happy music was associated with yellow, music expressing anger with large red colour patches, and sad music with smaller patches towards dark blue. Correlation analysis suggested patterns of relationships between audio features and colour patch parameters. Using partial least squares regression, we tested models for predicting colour patch responses from audio features and ratings of perceived emotion in the music. Parsimonious models that included emotion robustly explained between 60% and 75% of the variation in each of the colour patch parameters, as measured by cross-validated R2. To illuminate the quantitative findings, we performed a content analysis of structured spoken interviews with the participants. This provided further evidence of a significant emotion mediation mechanism, whereby people tended to match colour association with the perceived emotion in the music. The mixed method approach of our study gives strong evidence that emotion can mediate crossmodal association between music and visual colour. The CIE Lab interface promises to be a useful tool in perceptual ratings of music and other sounds.