BCI Toolbox: An open-source python package for the Bayesian causal inference model.

Psychological and neuroscientific research over the past two decades has shown that the Bayesian causal inference (BCI) is a potential unifying theory that can account for a wide range of perceptual and sensorimotor processes in humans. Therefore, we introduce the BCI Toolbox, a statistical and analytical tool in Python, enabling researchers to conveniently perform quantitative modeling and analysis of behavioral data. Additionally, we describe the algorithm of the BCI model and test its stability and reliability via parameter recovery. The present BCI toolbox offers a robust platform for BCI model implementation as well as a hands-on tool for learning and understanding the model, facilitating its widespread use and enabling researchers to delve into the data to uncover underlying cognitive mechanisms.

The overlooked role of unisensory precision in multisensory research.

Zhu et al. present an alternative explanation for the weaker multisensory illusions in football goalkeepers compared with outfielders and non-athletes, showing that better unisensory precision in goalkeepers can also account for this effect.

The effects of color and saturation on the enjoyment of real-life images.

This study investigated the effects of color presence and saturation on the affective judgment of real-life images, as functions of the image's affective valence. In the first two experiments, participants observed and rated original color photos and their grayscale versions, presented in an interleaved order across two separate experimental sessions. Color photos were rated as more pleasant than grayscale photos when image valence was positive, and more unpleasant when image valence was negative. The third experiment consisted of the same original images and their versions with saturation reduced by 50%. Original photos were rated as more pleasant than saturation-reduced photos when image valence was positive, yet less unpleasant when image valence was negative, implying potentially separable mechanisms for processing color presence and saturation. Significant interactions were found between color or saturation mode and valence on affective judgment in all three experiments. The effects persisted after controlling for colorfulness and luminance between the color and grayscale (or de-saturated) conditions.

Big number, big body: Jersey numbers alter body size perception.

Vision has been shown to be an active process that can be shaped by top-down influences. Here, we add to this area of research by showing a surprising example of how visual perception can be affected by cognition (i.e., cognitive penetration). Observers were presented, on each trial, with a picture of a computer-generated football player and asked to rate the slenderness of the player on an analog scale. The results of two experiments showed that observers perceived athletes wearing small jersey numbers as more slender than those with high numbers. This finding suggests that the cognition of numbers quantitatively alters body size perception. We conjecture that this effect is the result of previously learned associations (i.e., prior expectations) affecting perceptual inference. Such associations are likely the result of implicit learning of the statistical regularities of number and size attributes co-occurrences by the nervous system. We discuss how these results are consistent with previous research on statistical learning and how they fit into the Bayesian framework of perception. The current finding supports the notion of top-down influences of cognition on perception.

The box-circle illusion.

A novel geometrical optical illusion is reported in this article: the horizontal distances of the contextual structures distort the perceived vertical positions of observed objects. Specifically, the illusion manifests in the form of connected boxes of varying widths but equal heights, each containing a circle at the center. Despite identical vertical positioning of the circles, they appear misaligned. The illusion diminishes when the boxes are removed. Potential underlying mechanisms are discussed.

Crossmodal interactions in human learning and memory.

Most studies of memory and perceptual learning in humans have employed unisensory settings to simplify the study paradigm. However, in daily life we are often surrounded by complex and cluttered scenes made up of many objects and sources of sensory stimulation. Our experiences are, therefore, highly multisensory both when passively observing the world and when acting and navigating. We argue that human learning and memory systems are evolved to operate under these multisensory and dynamic conditions. The nervous system exploits the rich array of sensory inputs in this process, is sensitive to the relationship between the sensory inputs, and continuously updates sensory representations, and encodes memory traces based on the relationship between the senses. We review some recent findings that demonstrate a range of human learning and memory phenomena in which the interactions between visual and auditory modalities play an important role, and suggest possible neural mechanisms that can underlie some surprising recent findings. We outline open questions as well as directions of future research to unravel human perceptual learning and memory.

Improving memory for unusual events with wakeful reactivation.

Memory consists of multiple processes, from encoding information, consolidating it into short- and long- term memory, and later retrieving relevant information. Targeted memory reactivation is an experimental method during which sensory components of a multisensory representation (such as sounds or odors) are 'reactivated', facilitating the later retrieval of unisensory attributes. We examined whether novel and unpredicted events benefit from reactivation to a greater degree than normal stimuli. We presented participants with everyday objects, and 'tagged' these objects with sounds (e.g., animals and their matching sounds) at different screen locations. 'Oddballs' were created by presenting unusual objects and sounds (e.g., a unicorn with a heartbeat sound). During a short reactivation phase, participants listened to a replay of normal and oddball sounds. Participants were then tested on their memory for visual and spatial information in the absence of sounds. Participants were better at remembering the oddball objects compared to normal ones. Importantly, participants were also better at recalling the locations of oddball objects whose sounds were reactivated, compared to objects whose sounds that were not presented again. These results suggest that episodic memory benefits from associating objects with unusual cues, and that reactivating those cues strengthen the entire multisensory representation, resulting in enhanced memory for unisensory attributes.

Changing the Tendency to Integrate the Senses.

Integration of sensory signals that emanate from the same source, such as the visual of lip articulations and the sound of the voice of a speaking individual, can improve perception of the source signal (e.g., speech). Because momentary sensory inputs are typically corrupted with internal and external noise, there is almost always a discrepancy between the inputs, facing the perceptual system with the problem of determining whether the two signals were caused by the same source or different sources. Thus, whether or not multisensory stimuli are integrated and the degree to which they are bound is influenced by factors such as the prior expectation of a common source. We refer to this factor as the tendency to bind stimuli, or for short, binding tendency. In theory, the tendency to bind sensory stimuli can be learned by experience through the acquisition of the probabilities of the co-occurrence of the stimuli. It can also be influenced by cognitive knowledge of the environment. The binding tendency varies across individuals and can also vary within an individual over time. Here, we review the studies that have investigated the plasticity of binding tendency. We discuss the protocols that have been reported to produce changes in binding tendency, the candidate learning mechanisms involved in this process, the possible neural correlates of binding tendency, and outstanding questions pertaining to binding tendency and its plasticity. We conclude by proposing directions for future research and argue that understanding mechanisms and recipes for increasing binding tendency can have important clinical and translational applications for populations or individuals with a deficiency in multisensory integration.

Subliminal audio-visual temporal congruency in music videos enhances perceptual pleasure.

Human perception is inherently multisensory, with cross-modal integration playing a critical role in generating a coherent perceptual experience. To understand the causes of pleasurable experiences, we must understand whether and how the relationship between separate sensory modalities influences our experience of pleasure. We investigated the effect of congruency between vision and audition in the form of temporal alignment between the cuts in a video and the beats in an accompanying soundtrack. Despite the subliminal nature of this manipulation, a higher perceptual pleasure was found for temporal congruency compared with incongruency. These results suggest that the temporal aspect of the interaction between the visual and auditory modalities plays a critical role in shaping our perceptual pleasure, even when such interaction is not accessible to conscious awareness.

Bayesian causal inference: A unifying neuroscience theory.

Understanding of the brain and the principles governing neural processing requires theories that are parsimonious, can account for a diverse set of phenomena, and can make testable predictions. Here, we review the theory of Bayesian causal inference, which has been tested, refined, and extended in a variety of tasks in humans and other primates by several research groups. Bayesian causal inference is normative and has explained human behavior in a vast number of tasks including unisensory and multisensory perceptual tasks, sensorimotor, and motor tasks, and has accounted for counter-intuitive findings. The theory has made novel predictions that have been tested and confirmed empirically, and recent studies have started to map its algorithms and neural implementation in the human brain. The parsimony, the diversity of the phenomena that the theory has explained, and its illuminating brain function at all three of Marr's levels of analysis make Bayesian causal inference a strong neuroscience theory. This also highlights the importance of collaborative and multi-disciplinary research for the development of new theories in neuroscience.

Hierarchical drift diffusion modeling uncovers multisensory benefit in numerosity discrimination tasks.

Studies of accuracy and reaction time in decision making often observe a speed-accuracy tradeoff, where either accuracy or reaction time is sacrificed for the other. While this effect may mask certain multisensory benefits in performance when accuracy and reaction time are separately measured, drift diffusion models (DDMs) are able to consider both simultaneously. However, drift diffusion models are often limited by large sample size requirements for reliable parameter estimation. One solution to this restriction is the use of hierarchical Bayesian estimation for DDM parameters. Here, we utilize hierarchical drift diffusion models (HDDMs) to reveal a multisensory advantage in auditory-visual numerosity discrimination tasks. By fitting this model with a modestly sized dataset, we also demonstrate that large sample sizes are not necessary for reliable parameter estimation.

Disturbed Eating and Body Dysmorphic Symptoms in a Young Adult Sample Are Separable Constructs That Each Show a Mixture of Distributions.

Eating disorder and body dysmorphic disorder symptoms overlap and frequently co-occur clinically, yet whether they represent one or more underlying constructs in the general population is unknown. We examined relationships between these symptoms on underlying factor structures and dimensional distributions in a young adult sample of 328 students using the Eating Disorder Examination-Questionnaire (EDE-Q 6.0) and the Dysmorphic Concern Questionnaire (DCQ). We performed factor and hierarchical cluster analyses on pooled items and Gaussian mixture modeling on score distributions. EDE-Q 6.0 and DCQ total scores were correlated (r = 0.53, p < .001). Pooled items demonstrated a three-factor solution; DCQ items separating from two EDE-Q 6.0 factors. Hierarchical clustering yielded a two-cluster solution that separated the two scales. Mixture modeling demonstrated that more than one underlying distribution best fit the data for each scale. These results suggest that the EDE-Q 6.0 and DCQ measure different sets of psychopathological features, despite their tendency to track together. Moreover, eating disorder and body dysmorphic phenotypes each show nonuniform variation from normal to abnormal. This argues against using linear dimensional applications of these scales to assess individuals ranging from mild to severe in symptom severity. Separate scales may be necessary to characterize lower and higher ranges of clinical severity.

What you see is what you hear: Twenty years of research using the Sound-Induced Flash Illusion.

In the Sound-Induced Flash Illusion (SIFI) sound dramatically alters visual perception, as presenting a single flash with two beeps results in the perception of two flashes. In this comprehensive review, we synthesise 20 years of research using the SIFI, from over 100 studies. We discuss the neural and computational principles governing this illusion and examine the influence of perceptual experience, development, ageing and clinical conditions. Convergent findings show that the SIFI results from optimal integration and probabilistic inference and directly reflects crossmodal interactions in the temporal domain. Its neural basis lies in early modulation of visual cortex by auditory and multisensory regions. The SIFI shows increasingly strong potential as an efficient tool for measuring multisensory processing. Greater harmonisation across studies is now required to maximise this potential. We therefore propose considerations for researchers relating to choice of stimulus parameters and signpost directions for future research.

Explaining the Effect of Likelihood Manipulation and Prior Through a Neural Network of the Audiovisual Perception of Space.

Results in the recent literature suggest that multisensory integration in the brain follows the rules of Bayesian inference. However, how neural circuits can realize such inference and how it can be learned from experience is still the subject of active research. The aim of this work is to use a recent neurocomputational model to investigate how the likelihood and prior can be encoded in synapses, and how they affect audio-visual perception, in a variety of conditions characterized by different experience, different cue reliabilities and temporal asynchrony. The model considers two unisensory networks (auditory and visual) with plastic receptive fields and plastic crossmodal synapses, trained during a learning period. During training visual and auditory stimuli are more frequent and more tuned close to the fovea. Model simulations after training have been performed in crossmodal conditions to assess the auditory and visual perception bias: visual stimuli were positioned at different azimuth (±10° from the fovea) coupled with an auditory stimulus at various audio-visual distances (±20°). The cue reliability has been altered by using visual stimuli with two different contrast levels. Model predictions are compared with behavioral data. Results show that model predictions agree with behavioral data, in a variety of conditions characterized by a different role of prior and likelihood. Finally, the effect of a different unimodal or crossmodal prior, re-learning, temporal correlation among input stimuli, and visual damage (hemianopia) are tested, to reveal the possible use of the model in the clarification of important multisensory problems.

Perceived Electrical Injury: Misleading Symptomology Due to Multisensory Stimuli.

BACKGROUND: An electrical accident victim's recollection is often distorted by Bayesian inference in multisensory integration. For example, hearing the sound and seeing the bright flash of an electrical arc can create the false impression that someone had experienced an electrical shock. These subjects will often present to an emergency department seeking either treatment or reassurance. CASE REPORTS: We present seven cases in which the subjects were startled by an electrical shock (real or perceived) and injury was reported. Calculations of the current and path were used to allocate causality between the shock and a history of chronic disease or previous trauma. In all seven cases, our analysis suggests that no current was passed through the body. WHY SHOULD AN EMERGENCY PHYSICIAN BE AWARE OF THIS?: Symptomology seen as corroborating may actually be confounding. Witness and survivor descriptions of electrical shocks are fraught with subjectivity and misunderstanding. Available current is usually irrelevant and overemphasized, such as stress on a 100-ampere welding source, which is orders of magnitude beyond lethal limits. History can also be biased for a number of reasons. Bayesian inference in multisensory perception can lead to a subject sincerely believing they had experienced an electrical shock. Determination of the current pathway and calculations of the amplitude and duration of the shock can be critical for understanding the limits and potential causation of electrical injury.

Prior expectation of objects in space is dependent on the direction of gaze.

Many studies of multisensory spatial localization have shown that observers' responses are well-characterized by Bayesian inference, as localization judgments are influenced not only by the reliability of sensory encoding, but expectations about where things occur in space. Here, we investigate the frame of reference for the prior expectation of objects in space. Using an audiovisual localization task, we systematically manipulate fixation position and evaluate whether this prior is encoded in an eye-centered, head-centered, or hybrid frame of reference. Results show that in a majority of participants, this prior is encoded in an eye-centered frame of reference.

The material-weight illusion is a Bayes-optimal percept under competing density priors.

The material-weight illusion (MWI) is one example in a class of weight perception illusions that seem to defy principled explanation. In this illusion, when an observer lifts two objects of the same size and mass, but that appear to be made of different materials, the denser-looking (e.g., metal-look) object is perceived as lighter than the less-dense-looking (e.g., polystyrene-look) object. Like the size-weight illusion (SWI), this perceptual illusion occurs in the opposite direction of predictions from an optimal Bayesian inference process, which predicts that the denser-looking object should be perceived as heavier, not lighter. The presence of this class of illusions challenges the often-tacit assumption that Bayesian inference holds universal explanatory power to describe human perception across (nearly) all domains: If an entire class of perceptual illusions cannot be captured by the Bayesian framework, how could it be argued that human perception truly follows optimal inference? However, we recently showed that the SWI can be explained by an optimal hierarchical Bayesian causal inference process (Peters, Ma & Shams, 2016) in which the observer uses haptic information to arbitrate among competing hypotheses about objects' possible density relationship. Here we extend the model to demonstrate that it can readily explain the MWI as well. That hierarchical Bayesian inference can explain both illusions strongly suggests that even puzzling percepts arise from optimal inference processes.

Recalibrating the body: visuotactile ventriloquism aftereffect.

Visuotactile ventriloquism is a recently reported effect showing that somatotopic tactile representations (namely, representation of location along the surface of one's arm) can be biased by simultaneous presentation of a visual stimulus in a spatial localization task along the surface of the skin. Here we investigated whether the exposure to discrepancy between tactile and visual stimuli on the skin can induce lasting changes in the somatotopic representations of space. We conducted an experiment investigating this question by asking participants to perform a localization task that included unisensory and bisensory trials, before and after exposure to spatially discrepant visuotactile stimuli. Participants localized brief flashes of light and brief vibrations that were presented along the surface of their forearms, and were presented either individually (unisensory conditions) or were presented simultaneously at the same location or different locations. We then compared the localization of tactile stimuli in unisensory tactile conditions before and after the exposure to discrepant bisensory stimuli. After exposure, participants exhibited a shift in their tactile localizations in the direction of the visual stimulus that was presented during the exposure block. These results demonstrate that the somatotopic spatial representations are capable of rapidly recalibrating after a very brief exposure to visually discrepant stimuli.

A biologically inspired neurocomputational model for audiovisual integration and causal inference.

Recently, experimental and theoretical research has focused on the brain's abilities to extract information from a noisy sensory environment and how cross-modal inputs are processed to solve the causal inference problem to provide the best estimate of external events. Despite the empirical evidence suggesting that the nervous system uses a statistically optimal and probabilistic approach in addressing these problems, little is known about the brain's architecture needed to implement these computations. The aim of this work was to realize a mathematical model, based on physiologically plausible hypotheses, to analyze the neural mechanisms underlying multisensory perception and causal inference. The model consists of three layers topologically organized: two encode auditory and visual stimuli, separately, and are reciprocally connected via excitatory synapses and send excitatory connections to the third downstream layer. This synaptic organization realizes two mechanisms of cross-modal interactions: the first is responsible for the sensory representation of the external stimuli, while the second solves the causal inference problem. We tested the network by comparing its results to behavioral data reported in the literature. Among others, the network can account for the ventriloquism illusion, the pattern of sensory bias and the percept of unity as a function of the spatial auditory-visual distance, and the dependence of the auditory error on the causal inference. Finally, simulations results are consistent with probability matching as the perceptual strategy used in auditory-visual spatial localization tasks, agreeing with the behavioral data. The model makes untested predictions that can be investigated in future behavioral experiments.