Hearing what you see: Distinct excitatory and disinhibitory mechanisms contribute to visually-evoked auditory sensations.

Visual motion or flashing lights can evoke auditory sensations in some people. This large-scale internet study aimed to validate a combined subjective/objective test of the genuineness of this putative form of synaesthesia (visually-evoked auditory response, vEAR). Correlations were measured between each individual's ratings of the vividness of auditory sensations evoked by a series of looping videos, and measurement of the videos' physical low-level motion energy, calculated using Adelson and Bergen's (1985) computational model of low-level visual motion processing. The strength of this association for each individual provided a test of how strongly subjective vEAR was driven by objective motion energy ('ME-sensitivity'). A second aim was to infer whether vEAR depends on cortical excitation and/or disinhibition of early visual and/or auditory brain areas. To achieve this, correlations were measured between the above vEAR measures and visual contrast surround-suppression, which is thought to index lateral inhibition in the early visual system. As predicted by a disinhibition account of vEAR, video ratings were overall higher in individuals showing weaker surround-suppression. Interestingly, surround-suppression and ME-sensitivity did not correlate. Additionally, both surround-suppression and ME-sensitivity each independently predicted different clusters of trait measures selected for their possible association with cortical excitability and/or disinhibition: Surround-suppression was associated with vEAR self-ratings and auditory-evoked visual phosphenes, while ME-sensitivity was independently associated with ratings of other traits including susceptibility to migraine and pattern glare. Altogether, these results suggest there are two independent mechanisms underlying vEAR and its associated traits, based putatively on cortical disinhibition versus excitability.

Correlation of individual differences in audiovisual asynchrony across stimuli and tasks: New constraints on temporal renormalization theory.

Sight and sound are out of synch in different people by different amounts for different tasks. But surprisingly, different concurrent measures of perceptual asynchrony correlate negatively (Freeman et al., 2013). Thus, if vision subjectively leads audition in one individual, the same individual might show a visual lag in other measures of audiovisual integration (e.g., McGurk illusion, Stream-Bounce illusion). This curious negative correlation was first observed between explicit temporal order judgments and implicit phoneme identification tasks, performed concurrently as a dual task, using incongruent McGurk stimuli. Here we used a new set of explicit and implicit tasks and congruent stimuli, to test whether this negative correlation persists across testing sessions, and whether it might be an artifact of using specific incongruent stimuli. None of these manipulations eliminated the negative correlation between explicit and implicit measures. This supports the generalizability and validity of the phenomenon, and offers new theoretical insights into its explanation. Our previously proposed "temporal renormalization" theory assumes that the timings of sensory events registered within the brain's different multimodal subnetworks are each perceived relative to a representation of the typical average timing of such events across the wider network. Our new data suggest that this representation is stable and generic, rather than dependent on specific stimuli or task contexts, and that it may be acquired through experience with a variety of simultaneous stimuli. Our results also add further evidence that speech comprehension may be improved in some individuals by artificially delaying voices relative to lip-movements. (PsycINFO Database Record

Sounds from seeing silent motion: Who hears them, and what looks loudest?

Some people hear what they see: car indicator lights, flashing neon shop signs, and people's movements as they walk may all trigger an auditory sensation, which we call the visual-evoked auditory response (vEAR or 'visual ear'). We have conducted the first large-scale online survey (N > 4000) of this little-known phenomenon. We analysed the prevalence of vEAR, what induces it, and what other traits are associated with it. We assessed prevalence by asking whether respondents had previously experienced vEAR. Participants then rated silent videos for vividness of evoked auditory sensations, and answered additional trait questions. Prevalence appeared higher relative to other typical synaesthesias. Prior awareness and video ratings were associated with greater frequency of other synaesthesias, including flashes evoked by sounds, and musical imagery. Higher-rated videos often depicted meaningful events that predicted sounds (e.g., collisions). However, even videos containing abstract flickering or moving patterns could also elicit higher ratings, despite having no predictable association with sounds. Such videos had higher levels of raw 'motion energy' (ME), which we quantified using a simple computational model of motion processing in early visual cortex. Critically, only respondents reporting prior awareness of vEAR tended to show a positive correlation between video ratings and ME. This specific sensitivity to ME suggests that in vEAR, signals from visual motion processing may affect audition relatively directly without requiring higher-level interpretative processes. Our other findings challenge the popular assumption that individuals with synaesthesia are rare and have ideosyncratic patterns of brain hyper-connectivity. Instead, our findings of apparently high prevalence and broad associations with other synaesthesias and traits are jointly consistent with a common dependence on normal variations in physiological mechanisms of disinhibition or excitability of sensory brain areas and their functional connectivity. The prevalence of vEAR makes it easier to test such hypotheses further, and makes the results more relevant to understanding not only synaesthetic anomalies but also normal perception.

Sight and sound persistently out of synch: stable individual differences in audiovisual synchronisation revealed by implicit measures of lip-voice integration.

Are sight and sound out of synch? Signs that they are have been dismissed for over two centuries as an artefact of attentional and response bias, to which traditional subjective methods are prone. To avoid such biases, we measured performance on objective tasks that depend implicitly on achieving good lip-synch. We measured the McGurk effect (in which incongruent lip-voice pairs evoke illusory phonemes), and also identification of degraded speech, while manipulating audiovisual asynchrony. Peak performance was found at an average auditory lag of ~100 ms, but this varied widely between individuals. Participants' individual optimal asynchronies showed trait-like stability when the same task was re-tested one week later, but measures based on different tasks did not correlate. This discounts the possible influence of common biasing factors, suggesting instead that our different tasks probe different brain networks, each subject to their own intrinsic auditory and visual processing latencies. Our findings call for renewed interest in the biological causes and cognitive consequences of individual sensory asynchronies, leading potentially to fresh insights into the neural representation of sensory timing. A concrete implication is that speech comprehension might be enhanced, by first measuring each individual's optimal asynchrony and then applying a compensatory auditory delay.

fMRI correlates of object-based attentional facilitation vs. suppression of irrelevant stimuli, dependent on global grouping and endogenous cueing.

Theories of object-based attention often make two assumptions: that attentional resources are facilitatory, and that they spread automatically within grouped objects. Consistent with this, ignored visual stimuli can be easier to process, or more distracting, when perceptually grouped with an attended target stimulus. But in past studies, the ignored stimuli often shared potentially relevant features or locations with the target. In this fMRI study, we measured the effects of attention and grouping on Blood Oxygenation Level Dependent (BOLD) responses in the human brain to entirely task-irrelevant events. Two checkerboards were displayed each in opposite hemifields, while participants responded to check-size changes in one pre-cued hemifield, which varied between blocks. Grouping (or segmentation) between hemifields was manipulated between blocks, using common (vs. distinct) motion cues. Task-irrelevant transient events were introduced by randomly changing the color of either checkerboard, attended or ignored, at unpredictable intervals. The above assumptions predict heightened BOLD signals for irrelevant events in attended vs. ignored hemifields for ungrouped contexts, but less such attentional modulation under grouping, due to automatic spreading of facilitation across hemifields. We found the opposite pattern, in primary visual cortex. For ungrouped stimuli, BOLD signals associated with task-irrelevant changes were lower, not higher, in the attended vs. ignored hemifield; furthermore, attentional modulation was not reduced but actually inverted under grouping, with higher signals for events in the attended vs. ignored hemifield. These results challenge two popular assumptions underlying object-based attention. We consider a broader biased-competition framework: task-irrelevant stimuli are suppressed according to how strongly they compete with task-relevant stimuli, with intensified competition when the irrelevant features or locations comprise the same object.

Commonalities for numerical and continuous quantity skills at temporo-parietal junction.

How do our abilities to process number and other continuous quantities such as time and space relate to each other? Recent evidence suggests that these abilities share common magnitude processing and neural resources, although other findings also highlight the role of dimension-specific processes. To further characterize the relation between number, time, and space, we first examined them in a population with a developmental numerical dysfunction (developmental dyscalculia) and then assessed the extent to which these abilities correlated both behaviorally and anatomically in numerically normal participants. We found that (1) participants with dyscalculia showed preserved continuous quantity processing and (2) in numerically normal adults, numerical and continuous quantity abilities were at least partially dissociated both behaviorally and anatomically. Specifically, gray matter volume correlated with both measures of numerical and continuous quantity processing in the right TPJ; in contrast, individual differences in number proficiency were associated with gray matter volume in number-specific cortical regions in the right parietal lobe. Together, our new converging evidence of selective numerical impairment and of number-specific brain areas at least partially distinct from common magnitude areas suggests that the human brain is equipped with different ways of quantifying the outside world.

Sight and sound out of synch: fragmentation and renormalisation of audiovisual integration and subjective timing.

The sight and sound of a person speaking or a ball bouncing may seem simultaneous, but their corresponding neural signals are spread out over time as they arrive at different multisensory brain sites. How subjective timing relates to such neural timing remains a fundamental neuroscientific and philosophical puzzle. A dominant assumption is that temporal coherence is achieved by sensory resynchronisation or recalibration across asynchronous brain events. This assumption is easily confirmed by estimating subjective audiovisual timing for groups of subjects, which is on average similar across different measures and stimuli, and approximately veridical. But few studies have examined normal and pathological individual differences in such measures. Case PH, with lesions in pons and basal ganglia, hears people speak before seeing their lips move. Temporal order judgements (TOJs) confirmed this: voices had to lag lip-movements (by ∼200 msec) to seem synchronous to PH. Curiously, voices had to lead lips (also by ∼200 msec) to maximise the McGurk illusion (a measure of audiovisual speech integration). On average across these measures, PH's timing was therefore still veridical. Age-matched control participants showed similar discrepancies. Indeed, normal individual differences in TOJ and McGurk timing correlated negatively: subjects needing an auditory lag for subjective simultaneity needed an auditory lead for maximal McGurk, and vice versa. This generalised to the Stream-Bounce illusion. Such surprising antagonism seems opposed to good sensory resynchronisation, yet average timing across tasks was still near-veridical. Our findings reveal remarkable disunity of audiovisual timing within and between subjects. To explain this we propose that the timing of audiovisual signals within different brain mechanisms is perceived relative to the average timing across mechanisms. Such renormalisation fully explains the curious antagonistic relationship between disparate timing estimates in PH and healthy participants, and how they can still perceive the timing of external events correctly, on average.

fMRI correlates of subjective reversals in ambiguous structure-from-motion.

We used fMRI to examine the neural correlates of subjective reversals for bistable structure-from-motion. We compared transparent random-dot kinematograms depicting either a cylinder rotating in depth or two flat surfaces translating in opposite directions at apparently different depths. For both such stimuli, the motion of dots on the different apparent depth planes typically appears to reverse direction periodically on prolonged viewing. Yet for cylindrical but not flat stimuli, such subjective reversals also coincide with apparent reversal of 3D rotation direction. We hypothesized that the lateral occipital complex (region LOC), sensitive to 3D form, might show greater event-related activity for subjective reversals of cylindrical than flat stimuli; conversely, motion-sensitive hMT+/V5 should respond in common to subjective reversals for either type of stimuli, as both are perceived as changes in planar motion. We obtained an event-related measure of neural activity associated with subjective reversals after first factoring out block-related differences between cylindrical versus flat stimuli (and thereby the associated low-level blocked stimulus differences). In support of our hypothesis, only the cylindrical stimuli produced reversal-related activity in contralateral human LOC. In contrast, the hMT+/V5 complex was activated alike by subjective reversals for both cylindrical and flat stimuli. Intriguingly, V1 also showed (contralateral) specificity for rotational reversals, suggesting a possible feedback influence from LOC. These results reveal specific neural correlates for subjective switches of 3D rotation versus translation, as distinct from subjective reversals in general.

Time processing in dyscalculia.

To test whether atypical number development may affect other types of quantity processing, we investigated temporal discrimination in adults with developmental dyscalculia (DD). This also allowed us to test whether number and time may be sub-served by a common quantity system or decision mechanisms: if they do, both should be impaired in dyscalculia, but if number and time are distinct they should dissociate. Participants judged which of two successively presented horizontal lines was longer in duration, the first line being preceded by either a small or a large number prime ("1" or "9") or by a neutral symbol ("#"), or in a third task participants decided which of two Arabic numbers (either "1," "5," "9") lasted longer. Results showed that (i) DD's temporal discriminability was normal as long as numbers were not part of the experimental design, even as task-irrelevant stimuli; however (ii) task-irrelevant numbers dramatically disrupted DD's temporal discriminability the more their salience increased, though the actual magnitude of the numbers had no effect; in contrast (iii) controls' time perception was robust to the presence of numbers but modulated by numerical quantity: therefore small number primes or numerical stimuli seemed to make durations appear shorter than veridical, but longer for larger numerical prime or numerical stimuli. This study is the first to show spared temporal discrimination - a dimension of continuous quantity - in a population with a congenital number impairment. Our data reinforce the idea of a partially shared quantity system across numerical and temporal dimensions, which supports both dissociations and interactions among dimensions; however, they suggest that impaired number in DD is unlikely to originate from systems initially dedicated to continuous quantity processing like time.

Numbers and time doubly dissociate.

The magnitude dimensions of number, time and space have been suggested to share some common magnitude processing, which may imply symmetric interaction among dimensions. Here we challenge these suggestions by presenting a double dissociation between two neuropsychological patients with left (JT) and right (CB) parietal lesions and selective impairment of number and time processing respectively. Both patients showed an influence of task-irrelevant number stimuli on time but not space processing. In JT otherwise preserved time processing was severely impaired in the mere presence of task-irrelevant numbers, which themselves could not be processed accurately. In CB, impaired temporal estimation was influenced by preserved number processing: small numbers made (already grossly underestimated) time intervals appear even shorter relative to large numbers. However, numerical estimation was not influenced by time in healthy controls and in both patients. This new double dissociation between number and time processing and the asymmetric interaction of number on time: (1) provides further support to the hypothesis of a partly shared magnitude system among dimensions, instead of the proposal of a single, fully shared system or of independent magnitude systems which would not explain dissociations or interactions among dimensions; (2) may be explained in terms of a stable hierarchy of dimensions, with numbers being the strongest.

The role of right and left parietal lobes in the conceptual processing of numbers.

Neuropsychological and functional imaging studies have associated the conceptual processing of numbers with bilateral parietal regions (including intraparietal sulcus). However, the processes driving these effects remain unclear because both left and right posterior parietal regions are activated by many other conceptual, perceptual, attention, and response-selection processes. To dissociate parietal activation that is number-selective from parietal activation related to other stimulus or response-selection processes, we used fMRI to compare numbers and object names during exactly the same conceptual and perceptual tasks while factoring out activations correlating with response times. We found that right parietal activation was higher for conceptual decisions on numbers relative to the same tasks on object names, even when response time effects were fully factored out. In contrast, left parietal activation for numbers was equally involved in conceptual processing of object names. We suggest that left parietal activation for numbers reflects a range of processes, including the retrieval of learnt facts that are also involved in conceptual decisions on object names. In contrast, number selectivity in right parietal cortex reflects processes that are more involved in conceptual decisions on numbers than object names. Our results generate a new set of hypotheses that have implications for the design of future behavioral and functional imaging studies of patients with left and right parietal damage.

Dissociations and interactions between time, numerosity and space processing.

This study investigated time, numerosity and space processing in a patient (CB) with a right hemisphere lesion. We tested whether these magnitude dimensions share a common magnitude system or whether they are processed by dimension-specific magnitude systems. Five experimental tasks were used: Tasks 1-3 assessed time and numerosity independently and time and numerosity jointly. Tasks 4 and 5 investigated space processing independently and space and numbers jointly. Patient CB was impaired at estimating time and at discriminating between temporal intervals, his errors being underestimations. In contrast, his ability to process numbers and space was normal. A unidirectional interaction between numbers and time was found in both the patient and the control subjects. Strikingly, small numbers were perceived as lasting shorter and large numbers as lasting longer. In contrast, number processing was not affected by time, i.e. short durations did not result in perceiving fewer numbers and long durations in perceiving more numbers. Numbers and space also interacted, with small numbers answered faster when presented on the left side of space, and the reverse for large numbers. Our results demonstrate that time processing can be selectively impaired. This suggests that mechanisms specific for time processing may be partially independent from those involved in processing numbers and space. However, the interaction between numbers and time and between numbers and space also suggests that although independent, there maybe some overlap between time, numbers and space. These data suggest a partly shared mechanism between time, numbers and space which may be involved in magnitude processing or may be recruited to perform cognitive operations on magnitude dimensions.

The middle house or the middle floor: bisecting horizontal and vertical mental number lines in neglect.

This study explores the processing of mental number lines and physical lines in five patients with left unilateral neglect. Three tasks were used: mental number bisection ('report the middle number between two numbers'), physical line bisection ('mark the middle of a line'), and a landmark task ('is the mark on the line to the left/right or higher/lower than the middle of the line?'). We manipulated the number line orientation purely by task instruction: neglect patients were told that the number-pairs represented either houses on a street (horizontal condition) or floors in a building (vertical condition). We also manipulated physical line orientation for comparison. All five neglect patients showed a rightward bias for horizontally oriented physical and number lines (e.g. saying 'five' is the middle house number between 'two' and 'six'). Only three of these patients also showed an upward bias for vertically oriented number lines. The remaining two patients did not show any bias in processing vertical lines. Our results suggest that: (1) horizontal and vertical neglect can associate or dissociate among different patients; (2) bisecting number lines operates on internal horizontal and vertical representations possibly analogous to horizontal and vertical physical lines; (3) at least partially independent mechanisms may be involved in processing horizontal and vertical number lines.

Subjective appearance of ambiguous structure-from-motion can be driven by objective switches of a separate less ambiguous context.

Two ambiguous transparent structure-from-motion (SFM) stimuli often appear to co-rotate. Grossmann & Dobbins (2003) reported breakdown of such perceptual coupling when one stimulus was made unambiguous (by rendering it opaque), leading them to propose that coupling depends generally on differential stimulus ambiguity. In contrast, we demonstrate robust stimulus-driven coupling even when one SFM stimulus is relatively disambiguated, by using relative-luminance and/or binocular-disparity cues. Such context stimuli could induce stimulus-driven coupling by disambiguating the transparent stimulus, though critically only when the context was clearly non-opaque and coaxial with the ambiguous stimulus. This demonstrates long-range information-sharing between separate stimulus representations, subject to specific constraints.