Lexical Feedback in the Time-Invariant String Kernel (TISK) Model of Spoken Word Recognition.

The Time-Invariant String Kernel (TISK) model of spoken word recognition (Hannagan, Magnuson & Grainger, 2013; You & Magnuson, 2018) is an interactive activation model with many similarities to TRACE (McClelland & Elman, 1986). However, by replacing most time-specific nodes in TRACE with time-invariant open-diphone nodes, TISK uses orders of magnitude fewer nodes and connections than TRACE. Although TISK performed remarkably similarly to TRACE in simulations reported by Hannagan et al., the original TISK implementation did not include lexical feedback, precluding simulation of top-down effects, and leaving open the possibility that adding feedback to TISK might fundamentally alter its performance. Here, we demonstrate that when lexical feedback is added to TISK, it gains the ability to simulate top-down effects without losing the ability to simulate the fundamental phenomena tested by Hannagan et al. Furthermore, with feedback, TISK demonstrates graceful degradation when noise is added to input, although parameters can be found that also promote (less) graceful degradation without feedback. We review arguments for and against feedback in cognitive architectures, and conclude that feedback provides a computationally efficient basis for robust constraint-based processing.

Rethinking Weight Decay for Efficient Neural Network Pruning.

Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.

Origins of the specialization for letters and numbers in ventral occipitotemporal cortex.

Deep in the occipitotemporal cortex lie two functional regions, the visual word form area (VWFA) and the number form area (NFA), which are thought to play a special role in letter and number recognition, respectively. We review recent progress made in characterizing the origins of these symbol form areas in children or adults, sighted or blind subjects, and humans or monkeys. We propose two non-mutually-exclusive hypotheses on the origins of the VWFA and NFA: the presence of a connectivity bias, and a sensitivity to shape features. We assess the explanatory power of these hypotheses, describe their consequences, and offer several experimental tests.

On the time-course of adjacent and non-adjacent transposed-letter priming.

We compared effects of adjacent (e.g., atricle-ARTICLE) and non-adjacent (e.g., actirle-ARTICLE) transposed-letter (TL) primes in an ERP study using the sandwich priming technique. TL priming was measured relative to the standard double-substitution condition. We found significantly stronger priming effects for adjacent transpositions than non-adjacent transpositions (with 2 intervening letters) in behavioral responses (lexical decision latencies), and the adjacent priming effects emerged earlier in the ERP signal, at around 200 ms post-target onset. Non-adjacent priming effects emerged about 50 ms later and were short-lived, being significant only in the 250-300 ms time-window. Adjacent transpositions on the other hand continued to produce priming in the N400 time-window (300-500 ms post-target onset). This qualitatively different pattern of priming effects for adjacent and non-adjacent transpositions is discussed in the light of different accounts of letter transposition effects, and the utility of drawing a distinction between positional flexibility and positional noise.

Deep learning of orthographic representations in baboons.

What is the origin of our ability to learn orthographic knowledge? We use deep convolutional networks to emulate the primate's ventral visual stream and explore the recent finding that baboons can be trained to discriminate English words from nonwords. The networks were exposed to the exact same sequence of stimuli and reinforcement signals as the baboons in the experiment, and learned to map real visual inputs (pixels) of letter strings onto binary word/nonword responses. We show that the networks' highest levels of representations were indeed sensitive to letter combinations as postulated in our previous research. The model also captured the key empirical findings, such as generalization to novel words, along with some intriguing inter-individual differences. The present work shows the merits of deep learning networks that can simulate the whole processing chain all the way from the visual input to the response while allowing researchers to analyze the complex representations that emerge during the learning process.

The lazy visual word form area: computational insights into location-sensitivity.

In a recent study, Rauschecker et al. convincingly demonstrate that visual words evoke neural activation signals in the Visual Word Form Area that can be classified based on where they were presented in the visual fields. This result goes against the prevailing consensus, and begs an explanation. We show that one of the simplest possible models for word recognition, a multilayer feedforward network, will exhibit precisely the same behavior when trained to recognize words at different locations. The model suggests that the VWFA initially starts with information about location, which is not being suppressed during reading acquisition more than is needed to meet the requirements of location-invariant word recognition. Some new interpretations of Rauschecker et al.'s results are proposed, and three specific predictions are derived to be tested in further studies.

Spoken word recognition without a TRACE.

How do we map the rapid input of spoken language onto phonological and lexical representations over time? Attempts at psychologically-tractable computational models of spoken word recognition tend either to ignore time or to transform the temporal input into a spatial representation. TRACE, a connectionist model with broad and deep coverage of speech perception and spoken word recognition phenomena, takes the latter approach, using exclusively time-specific units at every level of representation. TRACE reduplicates featural, phonemic, and lexical inputs at every time step in a large memory trace, with rich interconnections (excitatory forward and backward connections between levels and inhibitory links within levels). As the length of the memory trace is increased, or as the phoneme and lexical inventory of the model is increased to a realistic size, this reduplication of time- (temporal position) specific units leads to a dramatic proliferation of units and connections, begging the question of whether a more efficient approach is possible. Our starting point is the observation that models of visual object recognition-including visual word recognition-have grappled with the problem of spatial invariance, and arrived at solutions other than a fully-reduplicative strategy like that of TRACE. This inspires a new model of spoken word recognition that combines time-specific phoneme representations similar to those in TRACE with higher-level representations based on string kernels: temporally independent (time invariant) diphone and lexical units. This reduces the number of necessary units and connections by several orders of magnitude relative to TRACE. Critically, we compare the new model to TRACE on a set of key phenomena, demonstrating that the new model inherits much of the behavior of TRACE and that the drastic computational savings do not come at the cost of explanatory power.

Learning diagnostic features: the delta rule does Bubbles.

It has been shown (Murray & Gold, 2004a) that the Bubbles paradigm for studying human perceptual identification can be formally analyzed and compared to reverse correlation methods when the underlying identification model is conceived as a linear amplifier (LAM). However the usefulness of a LAM for characterizing human perceptual identification mechanisms has subsequently been questioned (Gosselin & Schyns, 2004). In this article we show that a simple linear model that is formally analogous to the LAM--a linear perceptron trained with the delta rule--can make sense of several Bubbles experiments in the context of letter identification. Specifically, an analysis of input-output connection weights after training revealed that the most positive weights clustered around letter parts in a way that mimicked the diagnostic parts of letters revealed by the Bubbles technique (Fiset et al., 2008). Our results suggest that linear observer models are indeed unreasonably effective, at least as first approximations to human letter identification mechanisms.

Learning spatial invariance with the trace rule in nonuniform distributions.

Convolutional models of object recognition achieve invariance to spatial transformations largely because of the use of a suitably defined pooling operator. This operator typically takes the form of a max or average function defined across units tuned to the same feature. As a model of the brain's ventral pathway, where computations are carried out by weighted synaptic connections, such pooling can lead to spatial invariance only if the weights that connect similarly tuned units to a given pooling unit are of approximately equal strengths. How identical weights can be learned in the face of nonuniformly distributed data remains unclear. In this letter, we show how various versions of the trace learning rule can help solve this problem. This allows us in turn to explain previously published results and make recommendations as to the optimal rule for invariance learning.

Explaining word recognition, reading, the universe, and beyond: a modest proposal.

Frost proposes a new agenda for reading research, whereby cross-linguistic experiments would uncover linguistic universals to be integrated within a universal theory of reading. We reveal the dangers of following such a call, and demonstrate the superiority of the very approach that Frost condemns.

Deciphering CAPTCHAs: what a Turing test reveals about human cognition.

Turning Turing's logic on its head, we used widespread letter-based Turing Tests found on the internet (CAPTCHAs) to shed light on human cognition. We examined the basis of the human ability to solve CAPTCHAs, where machines fail. We asked whether this is due to our use of slow-acting inferential processes that would not be available to machines, or whether fast-acting automatic orthographic processing in humans has superior robustness to shape variations. A masked priming lexical decision experiment revealed efficient processing of CAPTCHA words in conditions that rule out the use of slow inferential processing. This shows that the human superiority in solving CAPTCHAs builds on a high degree of invariance to location and continuous transforms, which is achieved during the very early stages of visual word recognition in skilled readers.

Protein analysis meets visual word recognition: a case for string kernels in the brain.

It has been recently argued that some machine learning techniques known as Kernel methods could be relevant for capturing cognitive and neural mechanisms (Jäkel, Schölkopf, & Wichmann, 2009). We point out that ''String kernels,'' initially designed for protein function prediction and spam detection, are virtually identical to one contending proposal for how the brain encodes orthographic information during reading. We suggest some reasons for this connection and we derive new ideas for visual word recognition that are successfully put to the test. We argue that the versatility and performance of String kernels makes a compelling case for their implementation in the brain.

Holographic string encoding.

In this article, we apply a special case of holographic representations to letter position coding. We translate different well-known schemes into this format, which uses distributed representations and supports constituent structure. We show that in addition to these brain-like characteristics, performances on a standard benchmark of behavioral effects are improved in the holographic format relative to the standard localist one. This notably occurs because of emerging properties in holographic codes, like transposition and edge effects, for which we give formal demonstrations. Finally, we outline the limits of the approach as well as its possible future extensions.

Broken symmetries in a location-invariant word recognition network.

We studied the feedforward network proposed by Dandurand et al. (2010), which maps location-specific letter inputs to location-invariant word outputs, probing the hidden layer to determine the nature of the code. Hidden patterns for words were densely distributed, and K-means clustering on single letter patterns produced evidence that the network had formed semi-location-invariant letter representations during training. The possible confound with superseding bigram representations was ruled out, and linear regressions showed that any word pattern was well approximated by a linear combination of its constituent letter patterns. Emulating this code using overlapping holographic representations (Plate, 1995) uncovered a surprisingly acute and useful correspondence with the network, stemming from a broken symmetry in the connection weight matrix and related to the group-invariance theorem (Minsky & Papert, 1969). These results also explain how the network can reproduce relative and transposition priming effects found in humans.