Optimal reaching subject to computational and physical constraints reveals structure of the sensorimotor control system.

Optimal feedback control provides an abstract framework describing the architecture of the sensorimotor system without prescribing implementation details such as what coordinate system to use, how feedback is incorporated, or how to accommodate changing task complexity. We investigate how such details are determined by computational and physical constraints by creating a model of the upper limb sensorimotor system in which all connection weights between neurons, feedback, and muscles are unknown. By optimizing these parameters with respect to an objective function, we find that the model exhibits a preference for an intrinsic (joint angle) coordinate representation of inputs and feedback and learns to calculate a weighted feedforward and feedback error. We further show that complex reaches around obstacles can be achieved by augmenting our model with a path-planner based on via points. The path-planner revealed "avoidance" neurons that encode directions to reach around obstacles and "placement" neurons that make fine-tuned adjustments to via point placement. Our results demonstrate the surprising capability of computationally constrained systems and highlight interesting characteristics of the sensorimotor system.

Evolution, biomechanics, and neurobiology converge to explain selective finger motor control.

Humans use their fingers to perform a variety of tasks, from simple grasping to manipulating objects, to typing and playing musical instruments, a variety wider than any other species. The more sophisticated the task, the more it involves individuated finger movements, those in which one or more selected fingers perform an intended action while the motion of other digits is constrained. Here we review the neurobiology of such individuated finger movements. We consider their evolutionary origins, the extent to which finger movements are in fact individuated, and the evolved features of neuromuscular control that both enable and limit individuation. We go on to discuss other features of motor control that combine with individuation to create dexterity, the impairment of individuation by disease, and the broad extent of capabilities that individuation confers on humans. We comment on the challenges facing the development of a truly dexterous bionic hand. We conclude by identifying topics for future investigation that will advance our understanding of how neural networks interact across multiple regions of the central nervous system to create individuated movements for the skills humans use to express their cognitive activity.

Somatosensory cortex microstimulation modulates primary motor and ventral premotor cortex neurons with extensive spatial convergence and divergence.

Intracortical microstimulation (ICMS) is known to affect distant neurons transynaptically, yet the extent to which ICMS pulses delivered in one cortical area modulate neurons in other cortical areas remains largely unknown. Here we assessed how the individual pulses of multi-channel ICMS trains delivered in the upper extremity representation of the macaque primary somatosensory area (S1) modulate neuron firing in the primary motor cortex (M1) and in the ventral premotor cortex (PMv). S1-ICMS pulses modulated the majority of units recorded both in the M1 upper extremity representation and in PMv, producing more inhibition than excitation. Effects converged on individual neurons in both M1 and PMv from extensive S1 territories. Conversely, effects of ICMS delivered in a small region of S1 diverged to wide territories in both M1 and PMv. The effects of this direct modulation of M1 and PMv neurons produced by multi-electrode S1-ICMS like that used here may need to be taken into account by bidirectional brain-computer interfaces that decode intended movements from neural activity in these cortical motor areas. Significance Statement: Although ICMS is known to produce effects transynaptically, relatively little is known about how ICMS in one cortical area affects neurons in other cortical areas. We show that the effects of multi-channel ICMS in a small patch of S1 diverge to affect neurons distributed widely in both M1 and PMv, and conversely, individual neurons in each of these areas can be affected by ICMS converging from much of the S1 upper extremity representation. Such direct effects of ICMS may complicate the decoding of motor intent from M1 or PMv when artificial sensation is delivered via S1-ICMS in bidirectional brain-computer interfaces.

Cortico-cortical drive in a coupled premotor-primary motor cortex dynamical system.

In the conventional view of sensorimotor control, the premotor cortex (PM) plans actions that are executed by the primary motor cortex (M1). This notion arises in part from many experiments that have imposed a preparatory "planning" period, during which PM becomes active without M1. But during many natural movements, PM and M1 are co-activated, making it difficult to distinguish their functional roles. We leverage coupled dynamical systems models (cDSMs) to uncover interactions between PM and M1 during movements performed with no preparatory period. We build cDSMs using neural and behavioral data recorded from two non-human primates as they performed a reach-grasp-manipulate task. PM and M1 interact dynamically throughout these movements. Whereas PM drives the M1 in some situations, in other situations, M1 drives PM activity, contrary to the conventional assumption. Our DSM framework provides additional predictions differentiating the roles of PM and M1 in controlling movement.

A computational perspective on coordinate systems for motor control.

It is currently unknown what coordinate system or systems the primate motor cortex uses to represent movement, although experimental evidence has suggested several candidates. In order to understand how the physical geometry of the arm combines with computational constraints to influence the optimal choice of coordinate system, we construct a two-dimensional, physics-based arm model and couple it to a linear model of the motor cortex. The cortical model is provided with target positions and real time feedback of the current hand position in two different coordinate systems: cartesian and joint angle. We then optimize the parameters of the model subject to penalties on neural connectivity and muscle and neural energy use. We find that the optimized model strongly prefers to work in the joint angle coordinate system, suggesting that for neurons whose activity is closely tied to muscle activation, this is computationally the most efficient coordinate system in which to represent movement.

Cyclic, Condition-Independent Activity in Primary Motor Cortex Predicts Corrective Movement Behavior.

Reaching movements are known to have large condition-independent (CI) neural activity and cyclic neural dynamics. A new precision center-out task was performed by rhesus macaques to test the hypothesis that cyclic, CI neural activity in the primary motor cortex (M1) occurs not only during initial reaching movements but also during subsequent corrective movements. Corrective movements were observed to be discrete with time courses and bell-shaped speed profiles similar to the initial movements. CI cyclic neural trajectories were similar and repeated for initial and each additional corrective submovement. The phase of the cyclic CI neural activity predicted the time of peak movement speed more accurately than regression of instantaneous firing rate, even when the subject made multiple corrective movements. Rather than being controlled as continuations of the initial reach, a discrete cycle of motor cortex activity encodes each corrective submovement.

Neural Encoding of Reaches in a Linear Cortical Model.

To effectively control the arm, motor cortical neurons must produce complex patterns of activation that vary with the position and orientation of the arm and reach direction. In order to better understand how such a finely tuned dynamical system could arise and what its basic organizing principles are, we develop a model of the motor cortex as a linear dynamical system with feedback coupled to a two-joint model of the macaque arm. By optimizing the connections between neural populations with respect to an objective function that penalizes error between hand and target, as well as neural and muscular energy use, we show that certain properties of the motor cortex, such as muscle synergies, can naturally be obtained. We also demonstrate that the optimization process produces a stable neural system in which targets in the physical space are mapped to attracting fixed points in the neural state space. Finally, we show that this optimization process produces neural units with complex spatial and temporal activation patterns.

Injecting Information into the Mammalian Cortex: Progress, Challenges, and Promise.

For 150 years artificial stimulation has been used to study the function of the nervous system. Such stimulation-whether electrical or optogenetic-eventually may be used in neuroprosthetic devices to replace lost sensory inputs and to otherwise introduce information into the nervous system. Efforts toward this goal can be classified broadly as either biomimetic or arbitrary. Biomimetic stimulation aims to mimic patterns of natural neural activity, so that the subject immediately experiences the artificial stimulation as if it were natural sensation. Arbitrary stimulation, in contrast, makes no attempt to mimic natural patterns of neural activity. Instead, different stimuli-at different locations and/or in different patterns-are assigned different meanings randomly. The subject's time and effort then are required to learn to interpret different stimuli, a process that engages the brain's inherent plasticity. Here we will examine progress in using artificial stimulation to inject information into the cerebral cortex and discuss the challenges for and the promise of future development.

Neuronal Activity Distributed in Multiple Cortical Areas during Voluntary Control of the Native Arm or a Brain-Computer Interface.

Voluntary control of visually-guided upper extremity movements involves neuronal activity in multiple areas of the cerebral cortex. Studies of brain-computer interfaces (BCIs) that use spike recordings for input, however, have focused largely on activity in the region from which those neurons that directly control the BCI, which we call BCI units, are recorded. We hypothesized that just as voluntary control of the arm and hand involves activity in multiple cortical areas, so does voluntary control of a BCI. In two subjects (Macaca mulatta) performing a center-out task both with a hand-held joystick and with a BCI directly controlled by four primary motor cortex (M1) BCI units, we recorded the activity of other, non-BCI units in M1, dorsal premotor cortex (PMd) and ventral premotor cortex (PMv), primary somatosensory cortex (S1), dorsal posterior parietal cortex (dPPC), and the anterior intraparietal area (AIP). In most of these areas, non-BCI units were active in similar percentages and at similar modulation depths during both joystick and BCI trials. Both BCI and non-BCI units showed changes in preferred direction (PD). Additionally, the prevalence of effective connectivity between BCI and non-BCI units was similar during both tasks. The subject with better BCI performance showed increased percentages of modulated non-BCI units with increased modulation depth and increased effective connectivity during BCI as compared with joystick trials; such increases were not found in the subject with poorer BCI performance. During voluntary, closed-loop control, non-BCI units in a given cortical area may function similarly whether the effector is the native upper extremity or a BCI-controlled device.

Mirror neurons precede non-mirror neurons during action execution.

Mirror neurons are thought to represent an individual's ability to understand the actions of others by discharging as one individual performs or observes another individual performing an action. Studies typically have focused on mirror neuron activity during action observation, examining activity during action execution primarily to validate mirror neuron involvement in the motor act. As a result, little is known about the precise role of mirror neurons during action execution. In this study, during execution of reach-grasp-manipulate movements, we found activity of mirror neurons generally preceded that of non-mirror neurons. Not only did the onset of task-related modulation occur earlier in mirror neurons, but state transitions detected by hidden Markov models also occurred earlier in mirror neuron populations. Our findings suggest that mirror neurons may be at the forefront of action execution.NEW & NOTEWORTHY Mirror neurons commonly are thought to provide a neural substrate for understanding the actions of others, but mirror neurons also are active during action execution, when additional, non-mirror neurons are active as well. Examining the timing of activity during execution of a naturalistic reach-grasp-manipulate task, we found that mirror neuron activity precedes that of non-mirror neurons at both the unit and the population level. Thus mirror neurons may be at the leading edge of action execution.

Stimulating Cerebellar Outflow Reveals Temporal Control of Motor Cortical Activity.

Nashef et al. (2019) show that high-frequency stimulation of the superior cerebellar peduncle produces a temporary cerebellar deficit. While the deficit is present, motor cortex neurons that receive cerebellar input maintain their directional tuning but lose their noise correlation.

How is electrical stimulation of the brain experienced, and how can we tell? Selected considerations on sensorimotor function and speech.

Electrical stimulation of the nervous system is a powerful tool for localizing and examining the function of numerous brain regions. Delivered to certain regions of the cerebral cortex, electrical stimulation can evoke a variety of first-order effects, including observable movements or an urge to move, or somatosensory, visual, or auditory percepts. In still other regions the subject may be oblivious to the stimulation. Often overlooked, however, is whether the subject is aware of the stimulation, and if so, how the stimulation is experienced by the subject. In this review of how electrical stimulation has been used to study selected aspects of sensorimotor and language function, we raise questions that future studies might address concerning the subjects' second-order experiences of intention and agency regarding evoked movements, of the naturalness of evoked sensory percepts, and of other qualia that might be evoked in the absence of an overt first-order experience.

Condition-Dependent Neural Dimensions Progressively Shift during Reach to Grasp.

Neural population space analysis was performed to assess the dimensionality and dynamics of the neural population in the primary motor cortex (M1) during a reach-grasp-manipulation task in which both the reach location and the object being grasped were varied. We partitioned neural activity into three components: (1) general task-related activity independent of location and object, (2) location- and/or object-related activity, and (3) noise. Neural modulation related to location and/or object was only one-third the size of either general task modulation or noise. The neural dimensions of location and/or object-related activity overlapped with both the general task and noise dimensions. Rather than large amplitude modulation in a fixed set of dimensions, the active dimensions of location and/or object modulation shifted progressively over the time course of a trial.

Single-Finger Neural Basis Information-Based Neural Decoder for Multi-Finger Movements.

In this paper, we investigate the relationship between single and multi-finger movements. By exploiting the neural correlation between the temporal firing patterns between movements, we show that the Pearson's correlation coefficient for the physically related movement pairs are greater than those of others; the firing rates of the neurons that are tuned to a single-finger movements also increases when the corresponding multi-finger movements are instructed. We also use a hierarchical cluster analysis to verify not only the relationship between the single and multi-finger movements, but also the relationship between the flexion and extension movements. Furthermore, we propose a novel decoding method of modeling neural firing patterns while omitting the training process of the multi-finger movements. For the decoding, the Skellam and Gaussian probability distributions are used as mathematical models. The probabilistic distribution model of the multi-finger movements was estimated using the neural activity that was acquired during single-finger movements. As a result, the proposed neural decoding accuracy comparable with that of the supervised neural decoding accuracy when all of the neurons were used for the multi-finger movements. These results suggest that only the neural activities of single-finger movements can be exploited for the control of dexterous multi-finger neuroprosthetics.

Mirror Neuron Populations Represent Sequences of Behavioral Epochs During Both Execution and Observation.

Mirror neurons (MNs) have the distinguishing characteristic of modulating during both execution and observation of an action. Although most studies of MNs have focused on various features of the observed movement, MNs also may monitor the behavioral circumstances in which the movement is embedded, including time periods preceding and following the observed movement. Here, we recorded multiple MNs simultaneously from implanted electrode arrays as two male monkeys executed and observed a reach, grasp, and manipulate task involving different target objects. MNs were recorded from premotor cortex (PM-MNs) and primary motor cortex (M1-MNs). During execution trials, hidden Markov models (HMMs) applied to the activity of either PM-MN or M1-MN populations most often detected sequences of four hidden states, which we named according to the behavioral epoch during which each state began: initial, reaction, movement, and final. The hidden states of MN populations thus reflected not only the movement, but also three behavioral epochs during which no movement occurred. HMMs trained on execution trials could decode similar sequences of hidden states in observation trials, with complete hidden state sequences decoded more frequently from PM-MN populations than from M1-MN populations. Moreover, population trajectories projected in a 2D plane defined by execution trials were preserved in observation trials more for PM-MN than for M1-MN populations. These results suggest that MN populations represent entire behavioral sequences, including both movement and non-movement. PM-MN populations showed greater similarity than M1-MN populations in their representation of behavioral sequences during execution versus observation.SIGNIFICANCE STATEMENT Mirror neurons (MNs) are thought to provide a neural mechanism for understanding the actions of others. However, for an action to be understood, both the movement per se and the non-movement context before and after the movement need to be represented. We found that simultaneously recorded MN populations encoded sequential hidden neural states corresponding approximately to sequential behavioral epochs of a reach, grasp, and manipulate task. During observation trials, hidden state sequences were similar to those identified in execution trials. Hidden state similarity was stronger for MN populations in premotor cortex than for those in primary motor cortex. Execution/observation similarity of hidden state sequences may contribute to understanding the actions of others without actually performing the action oneself.

Modeling task-specific neuronal ensembles improves decoding of grasp.

OBJECTIVE: Dexterous movement involves the activation and coordination of networks of neuronal populations across multiple cortical regions. Attempts to model firing of individual neurons commonly treat the firing rate as directly modulating with motor behavior. However, motor behavior may additionally be associated with modulations in the activity and functional connectivity of neurons in a broader ensemble. Accounting for variations in neural ensemble connectivity may provide additional information about the behavior being performed. APPROACH: In this study, we examined neural ensemble activity in primary motor cortex (M1) and premotor cortex (PM) of two male rhesus monkeys during performance of a center-out reach, grasp and manipulate task. We constructed point process encoding models of neuronal firing that incorporated task-specific variations in the baseline firing rate as well as variations in functional connectivity with the neural ensemble. Models were evaluated both in terms of their encoding capabilities and their ability to properly classify the grasp being performed. MAIN RESULTS: Task-specific ensemble models correctly predicted the performed grasp with over 95% accuracy and were shown to outperform models of neuronal activity that assume only a variable baseline firing rate. Task-specific ensemble models exhibited superior decoding performance in 82% of units in both monkeys (p < 0.01). Inclusion of ensemble activity also broadly improved the ability of models to describe observed spiking. Encoding performance of task-specific ensemble models, measured by spike timing predictability, improved upon baseline models in 62% of units. SIGNIFICANCE: These results suggest that additional discriminative information about motor behavior found in the variations in functional connectivity of neuronal ensembles located in motor-related cortical regions is relevant to decode complex tasks such as grasping objects, and may serve the basis for more reliable and accurate neural prosthesis.

Injecting Instructions into Premotor Cortex.

The premotor cortex (PM) receives inputs from parietal cortical areas representing processed visuospatial information, translates that information into programs for particular movements, and communicates those programs to the primary motor cortex (M1) for execution. Consistent with this general function, intracortical microstimulation (ICMS) in the PM of sufficient frequency, amplitude, and duration has been shown to evoke complex movements of the arm and hand that vary systematically depending on the locus of stimulation. Using frequencies and amplitudes too low to evoke muscle activity, however, we found that ICMS in the PM can provide instructions to perform specific reach, grasp, and manipulate movements. These instructed actions were not fixed but rather were learned through associations between the arbitrary stimulation locations and particular movements. Low-amplitude ICMS at different PM locations thus evokes distinguishable experiences that can become associated with specific movements arbitrarily, providing a novel means of injecting information into the nervous system.

Decoding Kinematics Using Task-Independent Movement-Phase-Specific Encoding Models.

Neural decoders of kinematic variables have largely relied on task-dependent (TD) encoding models of the neural activity. TD decoders, though, require prior knowledge of the tasks, which may be unavailable, lack scalability as the number of tasks grows, and require a large number of trials per task to reduce the effects of neuronal variability. The execution of movements involves a sequence of phases (e.g., idle, planning, and so on) whose progression contributes to the neuronal variability. We hypothesize that information about the movement phase facilitates the decoding of kinematics and compensates for the lack of prior knowledge about the task. We test this hypothesis by designing a task-independent movement-phase-specific (TI-MPS) decoding algorithm. The algorithm assumes that movements proceed through a consistent sequence of phases regardless of the specific task, and it builds one model per phase by combining data from different tasks. Phase transitions are detected online from neural data and, for each phase, a specific encoding model is used. The TI-MPS algorithm was tested on single-unit recordings from 437 neurons in the dorsal and ventral pre-motor cortices from two nonhuman primates performing 3-D multi-object reach-to-grasp tasks. The TI-MPS decoder accurately decoded kinematics from tasks it was not trained for and outperformed TD approaches (one-way ANOVA with Tukey's post-hoc test and -value <0.05). Results indicate that a TI paradigm with MPS models may help decoding kinematics when prior information about the task is unavailable and pave the way toward clinically viable prosthetics.