Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T.

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Models of communication and control for brain networks: distinctions, convergence, and future outlook.

Recent advances in computational models of signal propagation and routing in the human brain have underscored the critical role of white-matter structure. A complementary approach has utilized the framework of network control theory to better understand how white matter constrains the manner in which a region or set of regions can direct or control the activity of other regions. Despite the potential for both of these approaches to enhance our understanding of the role of network structure in brain function, little work has sought to understand the relations between them. Here, we seek to explicitly bridge computational models of communication and principles of network control in a conceptual review of the current literature. By drawing comparisons between communication and control models in terms of the level of abstraction, the dynamical complexity, the dependence on network attributes, and the interplay of multiple spatiotemporal scales, we highlight the convergence of and distinctions between the two frameworks. Based on the understanding of the intertwined nature of communication and control in human brain networks, this work provides an integrative perspective for the field and outlines exciting directions for future work.

Network structure of cascading neural systems predicts stimulus propagation and recovery.

OBJECTIVE: Many neural systems display spontaneous, spatiotemporal patterns of neural activity that are crucial for information processing. While these cascading patterns presumably arise from the underlying network of synaptic connections between neurons, the precise contribution of the network's local and global connectivity to these patterns and information processing remains largely unknown. APPROACH: Here, we demonstrate how network structure supports information processing through network dynamics in empirical and simulated spiking neurons using mathematical tools from linear systems theory, network control theory, and information theory. MAIN RESULTS: In particular, we show that activity, and the information that it contains, travels through cycles in real and simulated networks. SIGNIFICANCE: Broadly, our results demonstrate how cascading neural networks could contribute to cognitive faculties that require lasting activation of neuronal patterns, such as working memory or attention.

Control of brain network dynamics across diverse scales of space and time.

The human brain is composed of distinct regions that are each associated with particular functions and distinct propensities for the control of neural dynamics. However, the relation between these functions and control profiles is poorly understood, as is the variation in this relation across diverse scales of space and time. Here we probe the relation between control and dynamics in brain networks constructed from diffusion tensor imaging data in a large community sample of young adults. Specifically, we probe the control properties of each brain region and investigate their relationship with dynamics across various spatial scales using the Laplacian eigenspectrum. In addition, through analysis of regional modal controllability and partitioning of modes, we determine whether the associated dynamics are fast or slow, as well as whether they are alternating or monotone. We find that brain regions that facilitate the control of energetically easy transitions are associated with activity on short length scales and slow timescales. Conversely, brain regions that facilitate control of difficult transitions are associated with activity on long length scales and fast timescales. Built on linear dynamical models, our results offer parsimonious explanations for the activity propagation and network control profiles supported by regions of differing neuroanatomical structure.

Dynamic representations in networked neural systems.

A group of neurons can generate patterns of activity that represent information about stimuli; subsequently, the group can transform and transmit activity patterns across synapses to spatially distributed areas. Recent studies in neuroscience have begun to independently address the two components of information processing: the representation of stimuli in neural activity and the transmission of information in networks that model neural interactions. Yet only recently are studies seeking to link these two types of approaches. Here we briefly review the two separate bodies of literature; we then review the recent strides made to address this gap. We continue with a discussion of how patterns of activity evolve from one representation to another, forming dynamic representations that unfold on the underlying network. Our goal is to offer a holistic framework for understanding and describing neural information representation and transmission while revealing exciting frontiers for future research.

Limited synapse overproduction can speed development but sometimes with long-term energy and discrimination penalties.

Neural circuit development requires that synapses be formed between appropriate neurons. In addition, for a hierarchical network, successful development involves a sequencing of developmental events. It has been suggested that one mechanism that helps speed up development of proper connections is an early overproduction of synapses. Using a computational model of synapse development, such as adaptive synaptogenesis, it is possible to study such overproduction and its role in speeding up development; it is also possible to study other outcomes of synapse overproduction that are seemingly new to the literature. With a fixed number of neurons, adaptive synaptogenesis can control the speed of synaptic development in two ways: by altering the rate constants of the adaptive processes or by altering the initial number of rapidly but non-selectively accrued synapses. Using either mechanism, the simulations reveal that synapse overproduction appears as an unavoidable concomitant of rapid adaptive synaptogenesis. However, the shortest development times, which always produces the greatest amount of synapse overproduction, reduce adult performance by three measures: energy use, discrimination error rates, and proportional neuron allocation. Thus, the results here lead to the hypothesis that the observed speed of neural network development represents a particular inter-generational compromise: quick development benefits parental fecundity while slow development benefits offspring fecundity.

Development of a novel remote-controlled and self-contained audiovisual-aided interactive system for immobilizing claustrophobic patients.

In radiotherapy, only a few immobilization systems, such as open-face mask and head mold with a bite plate, are available for claustrophobic patients with a certain degree of discomfort. The purpose of this study was to develop a remote-controlled and self-contained audiovisual (AV)-aided interactive system with the iPad mini with Retina display for intrafractional motion management in brain/H&N (head and neck) radiotherapy for claustrophobic patients. The self-contained, AV-aided interactive system utilized two tablet computers: one for AV-aided interactive guidance for the subject and the other for remote control by an operator. The tablet for audiovisual guidance traced the motion of a colored marker using the built-in front-facing camera, and the remote control tablet at the control room used infrastructure Wi-Fi networks for real-time communication with the other tablet. In the evaluation, a programmed QUASAR motion phantom was used to test the temporal and positional accuracy and resolution. Position data were also obtained from ten healthy volunteers with and without guidance to evaluate the reduction of intrafractional head motion in simulations of a claustrophobic brain or H&N case. In the phantom study, the temporal and positional resolution was 24 Hz and 0.2 mm. In the volunteer study, the average superior-inferior and right-left displacement was reduced from 1.9 mm to 0.3 mm and from 2.2 mm to 0.2 mm with AV-aided interactive guidance, respectively. The superior-inferior and right-left positional drift was reduced from 0.5 mm/min to 0.1 mm/min and from 0.4 mm/min to 0.04 mm/min with audiovisual-aided interactive guidance. This study demonstrated a reduction in intrafractional head motion using a remote-controlled and self-contained AV-aided interactive system of iPad minis with Retina display, easily obtainable and cost-effective tablet computers. This approach can potentially streamline clinical flow for claustrophobic patients without a head mask and also allows patients to practice self-motion management before radiation treatment delivery.