Crossmodal Associations and Working Memory in the Brain.

Crossmodal associations between stimuli from different sensory modalities could emerge in non-synesthetic people and be stored in working memory to guide goal-directed behaviors. This chapter reviews a plethora of studies in this field to summarize where, when, and how crossmodal associations and working memory are processed. It has been found that in those brain regions that are traditionally considered as unimodal primary sensory areas, neural activity could be influenced by crossmodal sensory signals at temporally very early stage of information processing. This phenomenon could not be due to feedback projections from higher level associative areas. Sequentially, neural processes would then occur in associative cortical areas including the posterior parietal cortex and prefrontal cortex. Neural oscillations in multiple frequency bands may reflect brain activity in crossmodal associations, and it is likely that neural synchrony is related to potential neural mechanisms underlying these processes. Primary sensory areas and associative areas coordinate together through neural synchrony to fulfil crossmodal associations and to guide working memory performance.

Mixed Selectivity Coding of Content-Temporal Detail by Dorsomedial Posterior Parietal Neurons.

The dorsomedial posterior parietal cortex (dmPPC) is part of a higher-cognition network implicated in elaborate processes underpinning memory formation, recollection, episode reconstruction, and temporal information processing. Neural coding for complex episodic processing is however under-documented. Here, we recorded extracellular neural activities from three male rhesus macaques (Macaca mulatta) and revealed a set of neural codes of "neuroethogram" in the primate parietal cortex. Analyzing neural responses in macaque dmPPC to naturalistic videos, we discovered several groups of neurons that are sensitive to different categories of ethogram items, low-level sensory features, and saccadic eye movement. We also discovered that the processing of category and feature information by these neurons is sustained by the accumulation of temporal information over a long timescale of up to 30 s, corroborating its reported long temporal receptive windows. We performed an additional behavioral experiment with additional two male rhesus macaques and found that saccade-related activities could not account for the mixed neuronal responses elicited by the video stimuli. We further observed monkeys' scan paths and gaze consistency are modulated by video content. Taken altogether, these neural findings explain how dmPPC weaves fabrics of ongoing experiences together in real time. The high dimensionality of neural representations should motivate us to shift the focus of attention from pure selectivity neurons to mixed selectivity neurons, especially in increasingly complex naturalistic task designs.

Neural mechanisms of musical structure and tonality, and the effect of musicianship.

Introduction: The neural basis for the processing of musical syntax has previously been examined almost exclusively in classical tonal music, which is characterized by a strictly organized hierarchical structure. Musical syntax may differ in different music genres caused by tonality varieties. Methods: The present study investigated the neural mechanisms for processing musical syntax across genres varying in tonality - classical, impressionist, and atonal music - and, in addition, examined how musicianship modulates such processing. Results: Results showed that, first, the dorsal stream, including the bilateral inferior frontal gyrus and superior temporal gyrus, plays a key role in the perception of tonality. Second, right frontotemporal regions were crucial in allowing musicians to outperform non-musicians in musical syntactic processing; musicians also benefit from a cortical-subcortical network including pallidum and cerebellum, suggesting more auditory-motor interaction in musicians than in non-musicians. Third, left pars triangularis carries out online computations independently of tonality and musicianship, whereas right pars triangularis is sensitive to tonality and partly dependent on musicianship. Finally, unlike tonal music, the processing of atonal music could not be differentiated from that of scrambled notes, both behaviorally and neurally, even among musicians. Discussion: The present study highlights the importance of studying varying music genres and experience levels and provides a better understanding of musical syntax and tonality processing and how such processing is modulated by music experience.

Fallacious reversal of event-order during recall reveals memory reconstruction in rhesus monkeys.

Whether nonhuman primate species can construct, still less reconstruct, order of past events remains controversial. Here we show that rhesus macaques are capable of reconstructing the temporal order of memory traces of dynamic videos. We made use of 2000 unseen naturalistic videos of wildlife content for encoding, and then probed monkeys' recollection of temporal-order of events with a temporal-order judgement (TOJ) test. This encoding-TOJ procedure was repeated at three different time points (day 1, day 2, and day 32+). We specifically tested for differential TOJ memory performance for videos that were displayed in a reverse sequence versus videos that were displayed in a normal sequence at these different time points. We observed that during TOJ monkeys committed more errors for video content that were shown in reverse but only upon re-exposures (i.e., day 2 and day 32+). Moreover, this memory distortion effect is significantly accentuated by social relevance of the video content. We interpret that the monkeys reversed the out-of-order events in accordance to their knowledge priors; such fallaciously re-ordered memory traces then led to higher rate of errors. Demonstrating in macaque monkeys a form of errors in temporal-order memory for reverse videos carries implications for studying memory retrospection in the primates.

Behavioral evidence for memory replay of video episodes in the macaque.

Humans recall the past by replaying fragments of events temporally. Here, we demonstrate a similar effect in macaques. We trained six rhesus monkeys with a temporal-order judgement (TOJ) task and collected 5000 TOJ trials. In each trial, the monkeys watched a naturalistic video of about 10 s comprising two across-context clips, and after a 2 s delay, performed TOJ between two frames from the video. The data are suggestive of a non-linear, time-compressed forward memory replay mechanism in the macaque. In contrast with humans, such compression of replay is, however, not sophisticated enough to allow these monkeys to skip over irrelevant information by compressing the encoded video globally. We also reveal that the monkeys detect event contextual boundaries, and that such detection facilitates recall by increasing the rate of information accumulation. Demonstration of a time-compressed, forward replay-like pattern in the macaque provides insights into the evolution of episodic memory in our lineage.

Guilty by association: How group-based (collective) guilt arises in the brain.

People do not only feel guilty for transgressions that they are causally responsible for (i.e., personal guilt); they also feel guilty for transgressions committed by those they identify as in-group members (i.e., collective or group-based guilt). Decades of research using scenario-based imagination tasks and self-reported measures has shown that when reminded of transgressions committed by in-group members, people express guilt and are willing to make amends, even when they are not causally responsible for the transgressions. However, it remains elusive whether people genuinely experience guilt or simply display remorseful gestures deemed appropriate in those contexts. To resolve this puzzle, it is critical to closely examine the neurocognitive basis of group-based guilt and its relationship with personal guilt, a goal that self-reported measures alone cannot satisfactorily achieve. Here, we combined functional MRI with an interaction-based minimal group paradigm in which participants either directly caused harm to a group of victims (i.e., personal guilt), or observed in-group members cause harm to the victims (i.e., group-based guilt). In three experiments (N â€‹= â€‹90), we demonstrated and replicated that the perceived responsibility one shared with in-group members in transgression predicted both behavioral and neural manifestations of group-based guilt. Multivariate pattern analysis (MVPA) of the functional MRI data showed that group-based guilt recruited patterns of neural responses in anterior middle cingulate cortex that resembled personal guilt. These results have broadened our understanding of how group membership is integrated into the neurocognitive processes underlying social emotions.

Attenuation of Pain Perception Induced by the Rubber Hand Illusion.

Adaptive behavior usually requires accurate representations of body positions and ownership, which rely on integration of multiple sources of sensory information. The rubber hand illusion (RHI) presents a compelling example demonstrating that the combination of visual and tactile signals strongly influences the subjective experience of body ownership. However, it still remains unclear how the perception of body ownership in turn alters other aspects of sensory processing, such as pain perception. In the present study, we examined whether the RHI could modulate the subjective experience of pain. We set three conditions corresponding to different levels of ownership of the rubber hand: the synchronous condition in which the rubber and the real hand were simultaneously stroked; the asynchronous condition in which the two hands were asynchronously stroked; the own-hand-only condition in which only the real hand was stroked. Results from the screening experiment indicated that subjects experienced the stronger RHI in the synchronous condition, compared with the strength of RHI in the other two conditions. In the main experiment, subjects were requested to report the intensity and unpleasantness of pain evoked by laser stimuli under the three stroking conditions. Results showed that pain ratings were significantly lower under the synchronous condition than those under the other two conditions, suggesting the RHI could induce a significant analgesic effect. Furthermore, the correlation analysis showed that the degree of the analgesic effect was positively correlated with the RHI strength across individuals. Taken together, these results suggest an analgesic effect of the RHI and support the potential usage of visual illusions in future translational research on pain.

A Simple and Compact MR-Compatible Electromagnetic Vibrotactile Stimulator.

We have developed a low-cost electromagnetic vibrotactile stimulator that uses the magnetic field of an MR scanner as a permanent magnet to power a vibrating motor. A simple variable current power supply is controlled by software using a USB data acquisition controller. In our study, the function of our novel stimulator was verified in a vibration frequency discrimination working memory task, in which various ranges of frequencies and amplitudes are delivered in MRI scanner. Furthermore, our functional MRI study revealed activations of the primary and secondary somatosensory cortices during the perception of tactile stimulation. Therefore, the new designed electromagnetic vibrotactile stimulator is capable of generating various frequencies of tactile stimuli and represents a powerful and useful tool for studying somatosensory functions with functional MRI.

Neural Correlates of Feedback Processing in Visuo-Tactile Crossmodal Paired-Associate Learning.

Previous studies have examined the neural correlates for crossmodal paired-associate (PA) memory and the temporal dynamics of its formation. However, the neural dynamics for feedback processing of crossmodal PA learning remain unclear. To examine this process, we recorded event-related scalp electrical potentials for PA learning of unimodal visual-visual pairs and crossmodal visual-tactile pairs when participants performed unimodal and crossmodal tasks. We examined event-related potentials (ERPs) after the onset of feedback in the tasks for three effects: feedback type (positive feedback vs. negative feedback), learning (as the learning progressed) and the task modality (crossmodal vs. unimodal). The results were as follows: (1) feedback type: the amplitude of P300 decreased with incorrect trials and the P400/N400 complex was only present in incorrect trials; (2) learning: progressive positive voltage shifts in frontal recording sites and negative voltage shifts in central and posterior recording sites were identified as learning proceeded; and (3) task modality: compared with the unimodal PA learning task, positive voltage shifts in frontal sites and negative voltage shifts in posterior sites were found in the crossmodal PA learning task. To sum up, these results shed light on cortical excitability related to feedback processing of crossmodal PA learning.

Language and Sensory Neural Plasticity in the Superior Temporal Cortex of the Deaf.

Visual stimuli are known to activate the auditory cortex of deaf people, presenting evidence of cross-modal plasticity. However, the mechanisms underlying such plasticity are poorly understood. In this functional MRI study, we presented two types of visual stimuli, language stimuli (words, sign language, and lip-reading) and a general stimulus (checkerboard) to investigate neural reorganization in the superior temporal cortex (STC) of deaf subjects and hearing controls. We found that only in the deaf subjects, all visual stimuli activated the STC. The cross-modal activation induced by the checkerboard was mainly due to a sensory component via a feed-forward pathway from the thalamus and primary visual cortex, positively correlated with duration of deafness, indicating a consequence of pure sensory deprivation. In contrast, the STC activity evoked by language stimuli was functionally connected to both the visual cortex and the frontotemporal areas, which were highly correlated with the learning of sign language, suggesting a strong language component via a possible feedback modulation. While the sensory component exhibited specificity to features of a visual stimulus (e.g., selective to the form of words, bodies, or faces) and the language (semantic) component appeared to recruit a common frontotemporal neural network, the two components converged to the STC and caused plasticity with different multivoxel activity patterns. In summary, the present study showed plausible neural pathways for auditory reorganization and correlations of activations of the reorganized cortical areas with developmental factors and provided unique evidence towards the understanding of neural circuits involved in cross-modal plasticity.

Plastic change of prefrontal cortex mediates anxiety-like behaviors associated with chronic pain in neuropathic rats.

Clinical studies show that anxiety and chronic pain are concomitant. The neural basis for the comorbidity is unclear. The prefrontal cortex (PFC) has been recognized as a critical area for affective disorders and chronic pain modulation. In this study, we examined the role of the PFC in the pathogenesis of anxiety associated with chronic pain in a rat model of neuropathic pain with spare nerve injury (SNI). The SNI rats showed apparent anxiety-like behaviors in both open field (OF) test and elevated-plus maze (EPM) test eight weeks after surgery. Thus, the number of entries to the central area in the OF decreased to 45% (±5%, n = 15) of sham control (n = 17), while the overall motor activity (i.e., total distance) was unaffected. In the EPM, the percentage of entries into the open arms significantly (p < 0.001) decreased in SNI rats (SNI: 12.58 ± 2.7%, n = 15; sham: 30.75 ± 2.82%, n = 17), so did the time spent in the open arms (SNI: 4.35 ± 1.45%, n = 15; Sham: 11.65 ± 2.18%, n = 17). To explore the neural basis for the association between anxiety and chronic pain, local field potentials (LFPs) were recorded from the medial PFC (mPFC) and ventral hippocampus. In SNI rats, there were significantly greater increases in both theta-frequency power in the mPFC and theta-frequency synchronization between the mPFC and ventral hippocampus, when animals were displaying elevated anxiety-like behaviors in avoiding anxiogenic regions in EPM and OF chamber. Western blot analyses showed a significant elevation of serotonin transporter expression in the anxious SNI rats. Inhibition of serotonin transporter effectively alleviated anxiety-like behaviors following sub-chronic (15 days) treatment with systemic citalopram (10 mg/kg/day, intraperitoneally). Moreover, the anxiety-like behaviors in the SNI rats were also suppressed by direct mPFC application of serotonin. Taken together, we conclude that the plasticity of serotonin transmission in the mPFC likely contribute to the promotion of anxiety state associated with neuropathic pain.

The Causal Role of the Prefrontal Cortex and Somatosensory Cortex in Tactile Working Memory.

In the present study, we searched for causal evidence linking activity in the bilateral primary somatosensory cortex (SI), posterior parietal cortex (PPC), and prefrontal cortex (PFC) with behavioral performance in vibrotactile working memory. Participants performed a vibrotactile delayed matching-to-sample task, while single-pulse transcranial magnetic stimulation (sp-TMS) was applied over these cortical areas at 100, 200, 300, 600, 1600, and 1900 ms after the onset of vibrotactile stimulation (200 ms duration). In our experiments, sp-TMS over the contralateral SI at the early delay (100 and 200 ms) deteriorated the accuracy of task performance, and over the ipsilateral SI at the late delay (1600 and 1900 ms) also induced such deteriorating effects. Furthermore, deteriorating effects caused by sp-TMS over the contralateral DLPFC at the same maintenance stage (1600 ms) were correlated with the effects caused by sp-TMS over the ipsilateral SI, indicating that information retained in the ipsilateral SI during the late delay may be associated with the DLPFC. Taken together, these results suggest that both the contralateral and ipsilateral SIs are involved in tactile WM, and the contralateral DLPFC bridges the contralateral SI and ipsilateral SI for goal-directed action.

Neural correlates of visuo-tactile crossmodal paired-associate learning and memory in humans.

Studies have indicated that a cortical sensory system is capable of processing information from different sensory modalities. However, it still remains unclear when and how a cortical system integrates and retains information across sensory modalities during learning. Here we investigated the neural dynamics underlying crossmodal associations and memory by recording event-related potentials (ERPs) when human participants performed visuo-tactile (crossmodal) and visuo-visual (unimodal) paired-associate (PA) learning tasks. In a trial of the tasks, the participants were required to explore and learn the relationship (paired or non-paired) between two successive stimuli. EEG recordings revealed dynamic ERP changes during participants' learning of paired-associations. Specifically, (1) the frontal N400 component showed learning-related changes in both unimodal and crossmodal tasks but did not show any significant difference between these two tasks, while the central P400 displayed both learning changes and task differences; (2) a late posterior negative slow wave (LPN) showed the learning effect only in the crossmodal task; (3) alpha-band oscillations appeared to be involved in crossmodal working memory. Additional behavioral experiments suggested that these ERP components were not relevant to the participants' familiarity with stimuli per se. Further, by shortening the delay length (from 1300ms to 400ms or 200 ms) between the first and second stimulus in the crossmodal task, declines in participants' task performance were observed accordingly. Taken together, these results provide insights into the cortical plasticity (induced by PA learning) of neural networks involved in crossmodal associations in working memory.

Neural correlates of heat-evoked pain memory in humans.

The neural processes underlying pain memory are not well understood. To explore these processes, contact heat-evoked potentials (CHEPs) were recorded in humans with electroencephalography (EEG) technique during a delayed matching-to-sample task, a working memory task involving presentations of two successive painful heat stimuli (S-1 and S-2) with different intensities separated by a 2-s interval (the memorization period). At the end of the task, the subject was required to discriminate the stimuli by indicating which (S-1 or S-2) induced more pain. A control task was used, in which no active discrimination was required between stimuli. All event-related potential (ERP) analysis was aligned to the onset of S-1. EEG activity exhibited two successive CHEPs: an N2-P2 complex (∼400 ms after onset of S-1) and an ultralate component (ULC, ∼900 ms). The amplitude of the N2-P2 at vertex, but not the ULC, was significantly correlated with stimulus intensity in these two tasks, suggesting that the N2-P2 represents neural coding of pain intensity. A late negative component (LNC) in the frontal recording region was observed only in the memory task during a 500-ms period before onset of S-2. LNC amplitude differed between stimulus intensities and exhibited significant correlations with the N2-P2 complex. These indicate that the frontal LNC is involved in maintenance of intensity of pain in working memory. Furthermore, alpha-band oscillations observed in parietal recording regions during the late delay displayed significant power differences between tasks. This study provides in the temporal domain previously unidentified neural evidence showing the neural processes involved in working memory of painful stimuli.

Mice with Shank3 Mutations Associated with ASD and Schizophrenia Display Both Shared and Distinct Defects.

Genetic studies have revealed significant overlaps of risk genes among psychiatric disorders. However, it is not clear how different mutations of the same gene contribute to different disorders. We characterized two lines of mutant mice with Shank3 mutations linked to ASD and schizophrenia. We found both shared and distinct synaptic and behavioral phenotypes. Mice with the ASD-linked InsG3680 mutation manifest striatal synaptic transmission defects before weaning age and impaired juvenile social interaction, coinciding with the early onset of ASD symptoms. On the other hand, adult mice carrying the schizophrenia-linked R1117X mutation show profound synaptic defects in prefrontal cortex and social dominance behavior. Furthermore, we found differential Shank3 mRNA stability and SHANK1/2 upregulation in these two lines. These data demonstrate that different alleles of the same gene may have distinct phenotypes at molecular, synaptic, and circuit levels in mice, which may inform exploration of these relationships in human patients.

Cooperative processing in primary somatosensory cortex and posterior parietal cortex during tactile working memory.

In the present study, causal roles of both the primary somatosensory cortex (SI) and the posterior parietal cortex (PPC) were investigated in a tactile unimodal working memory (WM) task. Individual magnetic resonance imaging-based single-pulse transcranial magnetic stimulation (spTMS) was applied, respectively, to the left SI (ipsilateral to tactile stimuli), right SI (contralateral to tactile stimuli) and right PPC (contralateral to tactile stimuli), while human participants were performing a tactile-tactile unimodal delayed matching-to-sample task. The time points of spTMS were 300, 600 and 900 ms after the onset of the tactile sample stimulus (duration: 200 ms). Compared with ipsilateral SI, application of spTMS over either contralateral SI or contralateral PPC at those time points significantly impaired the accuracy of task performance. Meanwhile, the deterioration in accuracy did not vary with the stimulating time points. Together, these results indicate that the tactile information is processed cooperatively by SI and PPC in the same hemisphere, starting from the early delay of the tactile unimodal WM task. This pattern of processing of tactile information is different from the pattern in tactile-visual cross-modal WM. In a tactile-visual cross-modal WM task, SI and PPC contribute to the processing sequentially, suggesting a process of sensory information transfer during the early delay between modalities.

Prefrontal cortex and sensory cortices during working memory: quantity and quality.

The activity in sensory cortices and the prefrontal cortex (PFC) throughout the delay interval of working memory (WM) tasks reflect two aspects of WM-quality and quantity, respectively. The delay activity in sensory cortices is fine-tuned to sensory information and forms the neural basis of the precision of WM storage, while the delay activity in the PFC appears to represent behavioral goals and filters out irrelevant distractions, forming the neural basis of the quantity of task-relevant information in WM. The PFC and sensory cortices interact through different frequency bands of neuronal oscillation (theta, alpha, and gamma) to fulfill goal-directed behaviors.

Sequential roles of primary somatosensory cortex and posterior parietal cortex in tactile-visual cross-modal working memory: a single-pulse transcranial magnetic stimulation (spTMS) study.

BACKGROUND: Both monkey neurophysiological and human EEG studies have shown that association cortices, as well as primary sensory cortical areas, play an essential role in sequential neural processes underlying cross-modal working memory. OBJECTIVE: The present study aims to further examine causal and sequential roles of the primary sensory cortex and association cortex in cross-modal working memory. METHODS: Individual MRI-based single-pulse transcranial magnetic stimulation (spTMS) was applied to bilateral primary somatosensory cortices (SI) and the contralateral posterior parietal cortex (PPC), while participants were performing a tactile-visual cross-modal delayed matching-to-sample task. Time points of spTMS were 300 ms, 600 ms, 900 ms after the onset of the tactile sample stimulus in the task. RESULTS: The accuracy of task performance and reaction time were significantly impaired when spTMS was applied to the contralateral SI at 300 ms. Significant impairment on performance accuracy was also observed when the contralateral PPC was stimulated at 600 ms. CONCLUSION: SI and PPC play sequential and distinct roles in neural processes of cross-modal associations and working memory.

Differential roles of delay-period neural activity in the monkey dorsolateral prefrontal cortex in visual-haptic crossmodal working memory.

Previous studies have shown that neurons of monkey dorsolateral prefrontal cortex (DLPFC) integrate information across modalities and maintain it throughout the delay period of working-memory (WM) tasks. However, the mechanisms of this temporal integration in the DLPFC are still poorly understood. In the present study, to further elucidate the role of the DLPFC in crossmodal WM, we trained monkeys to perform visuo-haptic (VH) crossmodal and haptic-haptic (HH) unimodal WM tasks. The neuronal activity recorded in the DLPFC in the delay period of both tasks indicates that the early-delay differential activity probably is related to the encoding of sample information with different strengths depending on task modality, that the late-delay differential activity reflects the associated (modality-independent) action component of haptic choice in both tasks (that is, the anticipation of the behavioral choice and/or active recall and maintenance of sample information for subsequent action), and that the sustained whole-delay differential activity likely bridges and integrates the sensory and action components. In addition, the VH late-delay differential activity was significantly diminished when the haptic choice was not required. Taken together, the results show that, in addition to the whole-delay differential activity, DLPFC neurons also show early- and late-delay differential activities. These previously unidentified findings indicate that DLPFC is capable of (i) holding the coded sample information (e.g., visual or tactile information) in the early-delay activity, (ii) retrieving the abstract information (orientations) of the sample (whether the sample has been haptic or visual) and holding it in the late-delay activity, and (iii) preparing for behavioral choice acting on that abstract information.

A fast schema for parameter estimation in diffusion kurtosis imaging.

Diffusion kurtosis imaging (DKI) is a new model in magnetic resonance imaging (MRI) characterizing restricted diffusion of water molecules in living tissues. We propose a method for fast estimation of the DKI parameters. These parameters - apparent diffusion coefficient (ADC) and apparent kurtosis coefficient (AKC) - are evaluated using an alternative iteration schema (AIS). This schema first roughly estimates a pair of ADC and AKC values from a subset of the DKI data acquired at 3 b-values. It then iteratively and alternately updates the ADC and AKC until they are converged. This approach employs the technique of linear least square fitting to minimize estimation error in each iteration. In addition to the common physical and biological constrains that set the upper and lower boundaries of the ADC and AKC values, we use a smoothing procedure to ensure that estimation is robust. Quantitative comparisons between our AIS methods and the conventional methods of unconstrained nonlinear least square (UNLS) using both synthetic and real data showed that our unconstrained AIS method can significantly accelerate the estimation procedure without compromising its accuracy, with the computational time for a DKI dataset successfully reduced to only 1 or 2min. Moreover, the incorporation of the smoothing procedure using one of our AIS methods can significantly enhance the contrast of AKC maps and greatly improve the visibility of details in fine structures.