Structural gray matter features and behavioral preliterate skills predict future literacy - A machine learning approach.

When children learn to read, their neural system undergoes major changes to become responsive to print. There seem to be nuanced interindividual differences in the neurostructural anatomy of regions that later become integral parts of the reading network. These differences might affect literacy acquisition and, in some cases, might result in developmental disorders like dyslexia. Consequently, the main objective of this longitudinal study was to investigate those interindividual differences in gray matter morphology that might facilitate or hamper future reading acquisition. We used a machine learning approach to examine to what extent gray matter macrostructural features and cognitive-linguistic skills measured before formal literacy teaching could predict literacy 2 years later. Forty-two native German-speaking children underwent T1-weighted magnetic resonance imaging and psychometric testing at the end of kindergarten. They were tested again 2 years later to assess their literacy skills. A leave-one-out cross-validated machine-learning regression approach was applied to identify the best predictors of future literacy based on cognitive-linguistic preliterate behavioral skills and cortical measures in a priori selected areas of the future reading network. With surprisingly high accuracy, future literacy was predicted, predominantly based on gray matter volume in the left occipito-temporal cortex and local gyrification in the left insular, inferior frontal, and supramarginal gyri. Furthermore, phonological awareness significantly predicted future literacy. In sum, the results indicate that the brain morphology of the large-scale reading network at a preliterate age can predict how well children learn to read.

Computational Models of Readers' Apperceptive Mass.

Recent progress in machine-learning-based distributed semantic models (DSMs) offers new ways to simulate the apperceptive mass (AM; Kintsch, 1980) of reader groups or individual readers and to predict their performance in reading-related tasks. The AM integrates the mental lexicon with world knowledge, as for example, acquired via reading books. Following pioneering work by Denhière and Lemaire (2004), here, we computed DSMs based on a representative corpus of German children and youth literature (Jacobs et al., 2020) as null models of the part of the AM that represents distributional semantic input, for readers of different reading ages (grades 1-2, 3-4, and 5-6). After a series of DSM quality tests, we evaluated the performance of these models quantitatively in various tasks to simulate the different reader groups' hypothetical semantic and syntactic skills. In a final study, we compared the models' performance with that of human adult and children readers in two rating tasks. Overall, the results show that with increasing reading age performance in practically all tasks becomes better. The approach taken in these studies reveals the limits of DSMs for simulating human AM and their potential for applications in scientific studies of literature, research in education, or developmental science.

Modelling brain representations of abstract concepts.

conceptual representations are critical for human cognition. Despite their importance, key properties of these representations remain poorly understood. Here, we used computational models of distributional semantics to predict multivariate fMRI activity patterns during the activation and contextualization of abstract concepts. We devised a task in which participants had to embed abstract nouns into a story that they developed around a given background context. We found that representations in inferior parietal cortex were predicted by concept similarities emerging in models of distributional semantics. By constructing different model families, we reveal the models' learning trajectories and delineate how abstract and concrete training materials contribute to the formation of brain-like representations. These results inform theories about the format and emergence of abstract conceptual representations in the human brain.

Neural processing of vision and language in kindergarten is associated with prereading skills and predicts future literacy.

The main objective of this longitudinal study was to investigate the neural predictors of reading acquisition. For this purpose, we followed a sample of 54 children from the end of kindergarten to the end of second grade. Preliterate children were tested for visual symbol (checkerboards, houses, faces, written words) and auditory language processing (spoken words) using a passive functional magnetic resonance imaging paradigm. To examine brain-behavior relationships, we also tested cognitive-linguistic prereading skills at kindergarten age and reading performance of 48 of the same children 2 years later. Face-selective response in the bilateral fusiform gyrus was positively associated with rapid automatized naming (RAN). Response to both spoken and written words at preliterate age was negatively associated with RAN in the dorsal temporo-parietal language system. Longitudinally, neural response to faces in the ventral stream predicted future reading fluency. Here, stronger neural activity in inferior and middle temporal gyri at kindergarten age was associated with higher reading performance. Our results suggest that interindividual differences in the neural system of language and reading affect literacy acquisition and thus might serve as a marker for successful reading acquisition in preliterate children.

Neural correlates of affective contributions to lexical decisions in children and adults.

The goal of the present study was to investigate whether 6-9-year old children and adults show similar neural responses to affective words. An event-related neuroimaging paradigm was used in which both age cohorts performed the same auditory lexical decision task (LDT). The results show similarities in (auditory) lexico-semantic network activation as well as in areas associated with affective information. In both age cohorts' activations were stronger for positive than for negative words, thus exhibiting a positivity superiority effect. Children showed less activation in areas associated with affective information in response to all three valence categories than adults. Our results are discussed in the light of computational models of word recognition, and previous findings of affective contributions to LDT in adults.

Neuroimaging of valence decisions in children and adults.

To date, the neural underpinnings of affective components in language processing in children remain largely unknown. To fill this gap, the present study examined behavioural and neural correlates of children and adults performing the same auditory valence decision task with an event-related fMRI paradigm. Based on previous findings in adults, activations in anterior and posterior cingulate cortex, orbitofrontal cortex and left inferior frontal gyrus were expected for both positive and negative valence categories. Recent behavioural findings on valence decisions showed similar ratings and reaction time patterns in children and adults. This finding was successfully replicated in the present study. On a neural level, our analysis of affective language processing showed activations in regions associated with both semantic (superior and middle temporal and frontal) and affective (anterior and posterior cingulate, orbitofrontal and inferior frontal, insula and amygdala) processing. Neural activations in children and adults were systematically different in explicit affective word processing. In particular, adults showed a more distributed semantic network activation while children recruited additional subcortical structures.

Sentiment Analysis of Children and Youth Literature: Is There a Pollyanna Effect?

If the words of natural human language possess a universal positivity bias, as assumed by Boucher and Osgood's (1969) famous Pollyanna hypothesis and computationally confirmed for large text corpora in several languages (Dodds et al., 2015), then children and youth literature (CYL) should also show a Pollyanna effect. Here we tested this prediction applying an unsupervised vector space model-based sentiment analysis tool called SentiArt (Jacobs, 2019) to two CYL corpora, one in English (372 books) and one in German (500 books). Pitching our analysis at the sentence level, and assessing semantic as well as lexico-grammatical information, both corpora show the Pollyanna effect and thus add further evidence to the universality hypothesis. The results of our multivariate sentiment analyses provide interesting testable predictions for future scientific studies of literature.

From Abstract Symbols to Emotional (In-)Sights: An Eye Tracking Study on the Effects of Emotional Vignettes and Pictures.

Reading is known to be a highly complex, emotion-inducing process, usually involving connected and cohesive sequences of sentences and paragraphs. However, most empirical results, especially from studies using eye tracking, are either restricted to simple linguistic materials (e.g., isolated words, single sentences) or disregard valence-driven effects. The present study addressed the need for ecologically valid stimuli by examining the emotion potential of and reading behavior in emotional vignettes, often used in applied psychological contexts and discourse comprehension. To allow for a cross-domain comparison in the area of emotion induction, negatively and positively valenced vignettes were constructed based on pre-selected emotional pictures from the Nencki Affective Picture System (NAPS; Marchewka et al., 2014). We collected ratings of perceived valence and arousal for both material groups and recorded eye movements of 42 participants during reading and picture viewing. Linear mixed-effects models were performed to analyze effects of valence (i.e., valence category, valence rating) and stimulus domain (i.e., textual, pictorial) on ratings of perceived valence and arousal, eye movements in reading, and eye movements in picture viewing. Results supported the success of our experimental manipulation: emotionally positive stimuli (i.e., vignettes, pictures) were perceived more positively and less arousing than emotionally negative ones. The cross-domain comparison indicated that vignettes are able to induce stronger valence effects than their pictorial counterparts, no differences between vignettes and pictures regarding effects on perceived arousal were found. Analyses of eye movements in reading replicated results from experiments using isolated words and sentences: perceived positive text valence attracted shorter reading times than perceived negative valence at both the supralexical and lexical level. In line with previous findings, no emotion effects on eye movements in picture viewing were found. This is the first eye tracking study reporting superior valence effects for vignettes compared to pictures and valence-specific effects on eye movements in reading at the supralexical level.

What Is the Difference? Rereading Shakespeare's Sonnets -An Eye Tracking Study.

Texts are often reread in everyday life, but most studies of rereading have been based on expository texts, not on literary ones such as poems, though literary texts may be reread more often than others. To correct this bias, the present study is based on two of Shakespeare's sonnets. Eye movements were recorded, as participants read a sonnet then read it again after a few minutes. After each reading, comprehension and appreciation were measured with the help of a questionnaire. In general, compared to the first reading, rereading improved the fluency of reading (shorter total reading times, shorter regression times, and lower fixation probability) and the depth of comprehension. Contrary to the other rereading studies using literary texts, no increase in appreciation was apparent. Moreover, results from a predictive modeling analysis showed that readers' eye movements were determined by the same critical psycholinguistic features throughout the two sessions. Apparently, even in the case of poetry, the eye movement control in reading is determined mainly by surface features of the text, unaffected by repetition.

Eye movements and mental imagery during reading of literary texts with different narrative styles.

Based on Kuzmicová's [1] phenomenological typology of narrative styles, we studied the specific contributions of mental imagery to literary reading experience and to reading behavior by combining questionnaires with eye-tracking methodology. Specifically, we focused on the two main categories in Kuzmicová's [1] typology, i.e., texts dominated by an "enactive" style, and texts dominated by a "descriptive" style. "Enactive" style texts render characters interacting with their environment, and "descriptive" style texts render environments dissociated from human action. The quantitative analyses of word category distributions of two dominantly enactive and two dominantly descriptive texts indicated significant differences especially in the number of verbs, with more verbs in enactment compared to descriptive texts. In a second study, participants read two texts (one theoretically cueing descriptive imagery, the other cueing enactment imagery) while their eye movements were recorded. After reading, participants completed questionnaires assessing aspects of the reading experience generally, as well as their text-elicited mental imagery specifically. Results show that readers experienced more difficulties conjuring up mental images during reading descriptive style texts and that longer fixation duration on words were associated with enactive style text. We propose that enactive style involves more imagery processes which can be reflected in eye movement behavior.

The SLS-Berlin: Validation of a German Computer-Based Screening Test to Measure Reading Proficiency in Early and Late Adulthood.

Reading proficiency, i.e., successfully integrating early word-based information and utilizing this information in later processes of sentence and text comprehension, and its assessment is subject to extensive research. However, screening tests for German adults across the life span are basically non-existent. Therefore, the present article introduces a standardized computerized sentence-based screening measure for German adult readers to assess reading proficiency including norm data from 2,148 participants covering an age range from 16 to 88 years. The test was developed in accordance with the children's version of the Salzburger LeseScreening (SLS, Wimmer and Mayringer, 2014). The SLS-Berlin has a high reliability and can easily be implemented in any research setting using German language. We present a detailed description of the test and report the distribution of SLS-Berlin scores for the norm sample as well as for two subsamples of younger (below 60 years) and older adults (60 and older). For all three samples, we conducted regression analyses to investigate the relationship between sentence characteristics and SLS-Berlin scores. In a second validation study, SLS-Berlin scores were compared with two (pseudo)word reading tests, a test measuring attention and processing speed and eye-movements recorded during expository text reading. Our results confirm the SLS-Berlin's sensitivity to capture early word decoding and later text related comprehension processes. The test distinguished very well between skilled and less skilled readers and also within less skilled readers and is therefore a powerful and efficient screening test for German adults to assess interindividual levels of reading proficiency.

Affective iconic words benefit from additional sound-meaning integration in the left amygdala.

Recent studies have shown that a similarity between sound and meaning of a word (i.e., iconicity) can help more readily access the meaning of that word, but the neural mechanisms underlying this beneficial role of iconicity in semantic processing remain largely unknown. In an fMRI study, we focused on the affective domain and examined whether affective iconic words (e.g., high arousal in both sound and meaning) activate additional brain regions that integrate emotional information from different domains (i.e., sound and meaning). In line with our hypothesis, affective iconic words, compared to their non-iconic counterparts, elicited additional BOLD responses in the left amygdala known for its role in multimodal representation of emotions. Functional connectivity analyses revealed that the observed amygdalar activity was modulated by an interaction of iconic condition and activations in two hubs representative for processing sound (left superior temporal gyrus) and meaning (left inferior frontal gyrus) of words. These results provide a neural explanation for the facilitative role of iconicity in language processing and indicate that language users are sensitive to the interaction between sound and meaning aspect of words, suggesting the existence of iconicity as a general property of human language.

Idiomatic expressions evoke stronger emotional responses in the brain than literal sentences.

Recent neuroscientific research shows that metaphors engage readers at the emotional level more strongly than literal expressions. What still remains unclear is what makes metaphors more engaging, and whether this generalises to all figurative expressions, no matter how conventionalised they are. This fMRI study aimed to investigate whether idiomatic expressions - the least creative part of figurative language - indeed trigger a higher affective resonance than literal expressions, and to explore possible interactions between activation in emotion-relevant neural structures and regions associated with figurative language processing. Participants silently read for comprehension a set of emotionally positive, negative and neutral idioms embedded in short sentences, and similarly valenced literal sentences. As in studies on metaphors, we found enhanced activation of the left inferior frontal gyrus and left amygdala in response to idioms, indexing stronger recruitment of executive control functions and enhanced emotional engagement, respectively. This suggests that the comprehension of even highly conventionalised and familiar figurative expressions, namely idioms, recruits regions involved in emotional processing. Furthermore, increased activation of the IFG interacted positively with activation in the amygdala, suggesting that the stronger cognitive engagement driven by idioms may in turn be coupled with stronger involvement at the emotional level.

A model-guided dissociation between subcortical and cortical contributions to word recognition.

Neurocognitive studies of visual word recognition have provided information about brain activity correlated with orthographic processing. Some of these studies related the orthographic neighborhood density of letter strings to the amount of hypothetical global lexical activity (GLA) in the brain as simulated by computational models of word recognition. To further investigate this issue, we used GLA of words and nonwords from the multiple read-out model of visual word recognition (MROM) and related this activity to neural correlates of orthographic processing in the brain by using functional magnetic resonance imaging (fMRI). Words and nonwords elicited linear effects in the cortex with increasing BOLD responses for decreasing values of GLA. In addition, words showed increasing linear BOLD responses for increasing GLA values in subcortical regions comprising the hippocampus, globus pallidus and caudate nucleus. We propose that these regions are involved in the matching of orthographic input onto representations in long-term memory. The results speak to a potential involvement of the basal ganglia in visual word recognition with globus pallidus and caudate nucleus activity potentially reflecting maintenance of orthographic input in working memory supporting the matching of the input onto stored representations by selection of appropriate lexical candidates and the inhibition of orthographically similar but non-matching candidates.

Reading Shakespeare Sonnets: Combining Quantitative Narrative Analysis and Predictive Modeling -an Eye Tracking Study.

As a part of a larger interdisciplinary project on Shakespeare sonnets' reception (1, 2), the present study analyzed the eye movement behavior of participants reading three of the 154 sonnets as a function of seven lexical features extracted via Quantitative Narrative Analysis (QNA). Using a machine learning-based predictive modeling approach five 'surface' features (word length, orthographic neighborhood density, word frequency, orthographic dissimilarity and sonority score) were detected as important predictors of total reading time and fixation probability in poetry reading. The fact that one phonological feature, i.e., sonority score, also played a role is in line with current theorizing on poetry reading. Our approach opens new ways for future eye movement research on reading poetic texts and other complex literary materials(3).

Eye movements during the reading of narrative and poetic texts. Symposium 6 at the 20th European Conference on Eye Movement Research (ECEM) in Alicante, 21.8.2019.

Despite a wealth of studies using eye tracking to investigate mental processes during vision or reading, the investigation of oculomotor activity during natural reading of longer texts -be it newspaper articles, narratives or poetry- is still an exception in this field (as evidenced by the program of ECEM 2017 in Wuppertal). Following up on our symposium at ECEM 2017, here we bring together eye movement research on natural text reading to report recent progress in a coordinated way sharing data, experiences and software skills in this highly complex subfield. More specifically, in this symposium we will address several challenges faced by an eye tracking perspective on the reading of longer texts which involve a surplus of intervening variables and novel methods to analyze the data. In particular, the following issues will be addressed: - Which text-analytical and statistical methods are best to deal with the myriad of surface and affective semantic features potentially influencing eye movements during reading of 'natural' texts? - What are the pros and cons of using machine learning assisted predictive modeling as an alternative to the standard GLM/LMM frameworks? - Which kind of theoretical models can deal with the level of complexity offered by reading longer natural texts? Video stream: https://vimeo.com/358415199.

Sentiment Analysis for Words and Fiction Characters From the Perspective of Computational (Neuro-)Poetics.

Two computational studies provide different sentiment analyses for text segments (e.g., "fearful" passages) and figures (e.g., "Voldemort") from the Harry Potter books (Rowling, 1997, 1998, 1999, 2000, 2003, 2005, 2007) based on a novel simple tool called SentiArt. The tool uses vector space models together with theory-guided, empirically validated label lists to compute the valence of each word in a text by locating its position in a 2d emotion potential space spanned by the words of the vector space model. After testing the tool's accuracy with empirical data from a neurocognitive poetics study, it was applied to compute emotional figure and personality profiles (inspired by the so-called "big five" personality theory) for main characters from the book series. The results of comparative analyses using different machine-learning classifiers (e.g., AdaBoost, Neural Net) show that SentiArt performs very well in predicting the emotion potential of text passages. It also produces plausible predictions regarding the emotional and personality profile of fiction characters which are correctly identified on the basis of eight character features, and it achieves a good cross-validation accuracy in classifying 100 figures into "good" vs. "bad" ones. The results are discussed with regard to potential applications of SentiArt in digital literary, applied reading and neurocognitive poetics studies such as the quantification of the hybrid hero potential of figures.

Simple Co-Occurrence Statistics Reproducibly Predict Association Ratings.

What determines human ratings of association? We planned this paper as a test for association strength (AS) that is derived from the log likelihood that two words co-occur significantly more often together in sentences than is expected from their single word frequencies. We also investigated the moderately correlated interactions of word frequency, emotional valence, arousal, and imageability of both words (r's ≤ .3). In three studies, linear mixed effects models revealed that AS and valence reproducibly account for variance in the human ratings. To understand further correlated predictors, we conducted a hierarchical cluster analysis and examined the predictors of four clusters in competitive analyses: Only AS and word2vec skip-gram cosine distances reproducibly accounted for variance in all three studies. The other predictors of the first cluster (number of common associates, (positive) point-wise mutual information, and word2vec CBOW cosine) did not reproducibly explain further variance. The same was true for the second cluster (word frequency and arousal); the third cluster (emotional valence and imageability); and the fourth cluster (consisting of joint frequency only). Finally, we discuss emotional valence as an important dimension of semantic space. Our results suggest that a simple definition of syntagmatic word contiguity (AS) and a paradigmatic measure of semantic similarity (skip-gram cosine) provide the most general performance-independent explanation of association ratings.

Affective Congruence between Sound and Meaning of Words Facilitates Semantic Decision.

A similarity between the form and meaning of a word (i.e., iconicity) may help language users to more readily access its meaning through direct form-meaning mapping. Previous work has supported this view by providing empirical evidence for this facilitatory effect in sign language, as well as for onomatopoetic words (e.g., cuckoo) and ideophones (e.g., zigzag). Thus, it remains largely unknown whether the beneficial role of iconicity in making semantic decisions can be considered a general feature in spoken language applying also to "ordinary" words in the lexicon. By capitalizing on the affective domain, and in particular arousal, we organized words in two distinctive groups of iconic vs. non-iconic based on the congruence vs. incongruence of their lexical (meaning) and sublexical (sound) arousal. In a two-alternative forced choice task, we asked participants to evaluate the arousal of printed words that were lexically either high or low arousing. In line with our hypothesis, iconic words were evaluated more quickly and more accurately than their non-iconic counterparts. These results indicate a processing advantage for iconic words, suggesting that language users are sensitive to sound-meaning mappings even when words are presented visually and read silently.

Why 'piss' is ruder than 'pee'? The role of sound in affective meaning making.

Most language users agree that some words sound harsh (e.g. grotesque) whereas others sound soft and pleasing (e.g. lagoon). While this prominent feature of human language has always been creatively deployed in art and poetry, it is still largely unknown whether the sound of a word in itself makes any contribution to the word's meaning as perceived and interpreted by the listener. In a large-scale lexicon analysis, we focused on the affective substrates of words' meaning (i.e. affective meaning) and words' sound (i.e. affective sound); both being measured on a two-dimensional space of valence (ranging from pleasant to unpleasant) and arousal (ranging from calm to excited). We tested the hypothesis that the sound of a word possesses affective iconic characteristics that can implicitly influence listeners when evaluating the affective meaning of that word. The results show that a significant portion of the variance in affective meaning ratings of printed words depends on a number of spectral and temporal acoustic features extracted from these words after converting them to their spoken form (study1). In order to test the affective nature of this effect, we independently assessed the affective sound of these words using two different methods: through direct rating (study2a), and through acoustic models that we implemented based on pseudoword materials (study2b). In line with our hypothesis, the estimated contribution of words' sound to ratings of words' affective meaning was indeed associated with the affective sound of these words; with a stronger effect for arousal than for valence. Further analyses revealed crucial phonetic features potentially causing the effect of sound on meaning: For instance, words with short vowels, voiceless consonants, and hissing sibilants (as in 'piss') feel more arousing and negative. Our findings suggest that the process of meaning making is not solely determined by arbitrary mappings between formal aspects of words and concepts they refer to. Rather, even in silent reading, words' acoustic profiles provide affective perceptual cues that language users may implicitly use to construct words' overall meaning.