Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension.

Current large language models (LLMs) rely on word prediction as their backbone pretraining task. Although word prediction is an important mechanism underlying language processing, human language comprehension occurs at multiple levels, involving the integration of words and sentences to achieve a full understanding of discourse. This study models language comprehension by using the next sentence prediction (NSP) task to investigate mechanisms of discourse-level comprehension. We show that NSP pretraining enhanced a model's alignment with brain data especially in the right hemisphere and in the multiple demand network, highlighting the contributions of nonclassical language regions to high-level language understanding. Our results also suggest that NSP can enable the model to better capture human comprehension performance and to better encode contextual information. Our study demonstrates that the inclusion of diverse learning objectives in a model leads to more human-like representations, and investigating the neurocognitive plausibility of pretraining tasks in LLMs can shed light on outstanding questions in language neuroscience.

Onscreen presence of instructors in video lectures affects learners' neural synchrony and visual attention during multimedia learning.

COVID-19 forced students to rely on online learning using multimedia tools, and multimedia learning continues to impact education beyond the pandemic. In this study, we combined behavioral, eye-tracking, and neuroimaging paradigms to identify multimedia learning processes and outcomes. College students viewed four video lectures including slides with either an onscreen human instructor, an animated instructor, or no onscreen instructor. Brain activity was recorded via fMRI, visual attention was recorded via eye-tracking, and learning outcome was assessed via post-tests. Onscreen presence of instructor, compared with no instructor presence, resulted in superior post-test performance, less visual attention on the slide, more synchronized eye movements during learning, and higher neural synchronization in cortical networks associated with socio-emotional processing and working memory. Individual variation in cognitive and socio-emotional abilities and intersubject neural synchronization revealed different levels of cognitive and socio-emotional processing in different learning conditions. The instructor-present condition evoked increased synchronization, likely reflecting extra processing demands in attentional control, working memory engagement, and socio-emotional processing. Although human instructors and animated instructors led to comparable learning outcomes, the effects were due to the dynamic interplay of information processing vs. attentional distraction. These findings reflect a benefit-cost trade-off where multimedia learning outcome is enhanced only when the cognitive benefits motivated by the social presence of onscreen instructor outweigh the cognitive costs brought about by concurrent attentional distraction unrelated to learning.

From eye movements to scanpath networks: A method for studying individual differences in expository text reading.

Eye movements have been examined as an index of attention and comprehension during reading in the literature for over 30 years. Although eye-movement measurements are acknowledged as reliable indicators of readers' comprehension skill, few studies have analyzed eye-movement patterns using network science. In this study, we offer a new approach to analyze eye-movement data. Specifically, we recorded visual scanpaths when participants were reading expository science text, and used these to construct scanpath networks that reflect readers' processing of the text. Results showed that low ability and high ability readers' scanpath networks exhibited distinctive properties, which are reflected in different network metrics including density, centrality, small-worldness, transitivity, and global efficiency. Such patterns provide a new way to show how skilled readers, as compared with less skilled readers, process information more efficiently. Implications of our analyses are discussed in light of current theories of reading comprehension.

The brain basis of handwriting deficits in Chinese children with developmental dyslexia.

Abundant behavioral studies have demonstrated high comorbidity of reading and handwriting difficulties in developmental dyslexia (DD), a neurological condition characterized by unexpectedly low reading ability despite adequate nonverbal intelligence and typical schooling. The neural correlates of handwriting deficits remain largely unknown; however, as well as the extent that handwriting deficits share common neural bases with reading deficits in DD. The present work used functional magnetic resonance imaging to examine brain activity during handwriting and reading tasks in Chinese dyslexic children (n = 18) and age-matched controls (n = 23). Compared to controls, dyslexic children exhibited reduced activation during handwriting tasks in brain regions supporting sensory-motor processing (including supplementary motor area and postcentral gyrus) and visual-orthography processing (including bilateral precuneus and right cuneus). Among these regions, the left supplementary motor area and the right precuneus also showed a trend of reduced activation during reading tasks in dyslexics. Moreover, increased activation was found in the left inferior frontal gyrus and anterior cingulate cortex in dyslexics, which may reflect more efforts of executive control to compensate for the impairments of motor and visual-orthographic processing. Finally, dyslexic children exhibited aberrant functional connectivity among brain areas for cognitive control and sensory-motor processes during handwriting tasks. Together, these findings suggest that handwriting deficits in DD are associated with functional abnormalities of multiple brain regions implicated in motor execution, visual-orthographic processing, and cognitive control, providing important implications for the diagnosis and treatment of dyslexia.

Auditory processing deficit in individuals with dyslexia: A meta-analysis of mismatch negativity.

Several previous studies have used mismatch negativity (MMN) to examine the auditory processing deficit in individuals with dyslexia. However, researchers have not clearly determined whether the deficit is general or specific and how it potentially changes with age. Meta-analysis was adopted to quantitatively identify the auditory processing deficit in individuals with dyslexia. By analysing 81 results within 25 publications that employed passive oddball paradigms to explore auditory processing in individuals with dyslexia, we identified that MMN impairment in auditory processing of speech was observed in children (Cohen's d = 0.296) and adults with dyslexia (Cohen's d = 0.486). Besides, adults with dyslexia showed atypical auditory processing of non-speech (Cohen's d = 0.409), which appeared to be related to the types of stimuli. Based on these findings, for individuals with dyslexia, the auditory processing deficit in speech will persist into adulthood, and the auditory processing deficit is general in adults with dyslexia. Because few studies used appropriate non-speech stimuli to examine the auditory processing in children with dyslexia, future studies should focus more on this area.

Men and women differ in the neural basis of handwriting.

There is an ongoing debate about whether, and to what extent, males differ from females in their language skills. In the case of handwriting, a composite language skill involving language and motor processes, behavioral observations consistently show robust sex differences but the mechanisms underlying the effect are unclear. Using functional magnetic resonance imaging (fMRI) in a copying task, the present study examined the neural basis of sex differences in handwriting in 53 healthy adults (ages 19-28, 27 males). Compared to females, males showed increased activation in the left posterior middle frontal gyrus (Exner's area), a region thought to support the conversion between orthographic and graphomotor codes. Functional connectivity between Exner's area and the right cerebellum was greater in males than in females. Furthermore, sex differences in brain activity related to handwriting were independent of language material. This study identifies a novel neural signature of sex differences in a hallmark of human behavior, and highlights the importance of considering sex as a factor in scientific research and clinical applications involving handwriting.