An accurate and rapidly calibrating speech neuroprosthesis.

Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40's with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.

BRAND: a platform for closed-loop experiments with deep network models.

Objective.Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g. Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g. C and C++).Approach.To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termednodes, which communicate with each other in agraphvia streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis, an in-memory database, to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes.Main results.In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1 ms chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 ms of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial (ClinicalTrials.gov Identifier: NCT00912041) performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems.Significance.By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

BRAND: A platform for closed-loop experiments with deep network models.

Artificial neural networks (ANNs) are state-of-the-art tools for modeling and decoding neural activity, but deploying them in closed-loop experiments with tight timing constraints is challenging due to their limited support in existing real-time frameworks. Researchers need a platform that fully supports high-level languages for running ANNs (e.g., Python and Julia) while maintaining support for languages that are critical for low-latency data acquisition and processing (e.g., C and C++). To address these needs, we introduce the Backend for Realtime Asynchronous Neural Decoding (BRAND). BRAND comprises Linux processes, termed nodes , which communicate with each other in a graph via streams of data. Its asynchronous design allows for acquisition, control, and analysis to be executed in parallel on streams of data that may operate at different timescales. BRAND uses Redis to send data between nodes, which enables fast inter-process communication and supports 54 different programming languages. Thus, developers can easily deploy existing ANN models in BRAND with minimal implementation changes. In our tests, BRAND achieved <600 microsecond latency between processes when sending large quantities of data (1024 channels of 30 kHz neural data in 1-millisecond chunks). BRAND runs a brain-computer interface with a recurrent neural network (RNN) decoder with less than 8 milliseconds of latency from neural data input to decoder prediction. In a real-world demonstration of the system, participant T11 in the BrainGate2 clinical trial performed a standard cursor control task, in which 30 kHz signal processing, RNN decoding, task control, and graphics were all executed in BRAND. This system also supports real-time inference with complex latent variable models like Latent Factor Analysis via Dynamical Systems. By providing a framework that is fast, modular, and language-agnostic, BRAND lowers the barriers to integrating the latest tools in neuroscience and machine learning into closed-loop experiments.

Cortical connectivity is embedded in resting state at columnar resolution.

Resting state (RS) fMRI is now widely used for gaining insight into the organization of brain networks. Functional connectivity (FC) inferred from RS-fMRI is typically at macroscale, which is too coarse for much of the detail in cortical architecture. Here, we examined whether imaging RS at higher contrast and resolution could reveal cortical connectivity with columnar granularity. In longitudinal experiments (~1.5 years) in squirrel monkeys, we partitioned sensorimotor cortex using dense microelectrode mapping and then recorded RS with intrinsic signal optical imaging (RS-ISOI, 20 µm/pixel). FC maps were benchmarked against microstimulation-evoked activation and traced anatomical connections. These direct comparisons showed high correspondence in connectivity patterns across methods. The fidelity of FC maps to cortical connections indicates that granular details of network organization are embedded in RS. Thus, for recording RS, the field-of-view and effective resolution achieved with ISOI fills a wide gap between fMRI and invasive approaches (2-photon imaging, electrophysiology). RS-ISOI opens exciting opportunities for high resolution mapping of cortical networks in living animals.

Principles of Intrinsic Motor Cortex Connectivity in Primates.

The forelimb representation in motor cortex (M1) is an important model system in contemporary neuroscience. Efforts to understand the organization of the M1 forelimb representation in monkeys have focused on inputs and outputs. In contrast, intrinsic M1 connections remain mostly unexplored, which is surprising given that intra-areal connections universally outnumber extrinsic connections. To address this knowledge gap, we first mapped the M1 forelimb representation with intracortical microstimulation (ICMS) in male squirrel monkeys. Next, we determined the connectivity of individual M1 sites with ICMS + intrinsic signal optical imaging (ISOI). Every stimulation site activated a distinctive pattern of patches (∼0.25 to 1.0 mm radius) that we quantified in relation to the motor map. Arm sites activated patches that were mostly in arm zones. Hand sites followed the same principle, but to a lesser extent. The results collectively indicate that preferential connectivity between functionally matched patches is a prominent organizational principle in M1. Connectivity patterns for a given site were conserved across a range of current amplitudes, train durations, pulse frequencies, and microelectrode depths. In addition, we found close correspondence in somatosensory cortex between connectivity that we revealed with ICMS+ISOI and connections known from tracers. ICMS+ISOI is therefore an effective tool for mapping cortical connectivity and is particularly advantageous for sampling large numbers of sites. This feature was instrumental in revealing the spatial specificity of intrinsic M1 connections, which appear to be woven into the somatotopic organization of the forelimb representation. Such a framework invokes the modular organization well-established for sensory cortical areas.SIGNIFICANCE STATEMENT Intrinsic connections are fundamental to the operations of any cortical area. Surprisingly little is known about the organization of intrinsic connections in motor cortex (M1). We addressed this knowledge gap using intracortical microstimulation (ICMS) concurrently with intrinsic signal optical imaging (ISOI). Quantifying the activation patterns from dozens of M1 sites allowed us to uncover a fundamental principle of M1 organization: M1 patches are preferentially connected with functionally matched patches. Relationship between intrinsic connections and neurophysiological map is well-established for sensory cortical areas, but our study is the first to extend this framework to M1. Microstimulation+imaging opened a unique possibility for investigating the connectivity of dozens of tightly spaced M1 sites, which was the linchpin for uncovering organizational principles.

Assessing vibrotactile feedback strategies by controlling a cursor with unstable dynamics.

Brain computer interface (BCI) control predominately uses visual feedback. Real arm movements, however, are controlled under a diversity of feedback mechanisms. The lack of additional BCI feedback modalities forces users to maintain visual contact while performing tasks. Such stringent requirements result in poor BCI control during tasks that inherently lack visual feedback, such as grasping, or when visual attention is diverted. Using a modified version of the Critical Tracking Task which we call the Critical Stability Task (CST), we tested the ability of 9 human subjects to control an unstable system using either free arm movements or pinch force. The subjects were provided either visual feedback, 'proportional' vibrotactile feedback, or 'on-off' vibrotactile feedback about the state of the unstable system. We increased the difficulty of the control task by making the virtual system more unstable. We judged the effectiveness of a particular form of feedback as the maximal instability the system could reach before the subject lost control of it. We found three main results. First, subjects can use solely vibrotactile feedback to control an unstable system, although control was better using visual feedback. Second, 'proportional' vibrotactile feedback provided slightly better control than 'on-off' vibrotactile feedback. Third, there was large intra-subject variability in terms of the most effective input and feedback methods. This highlights the need to tailor the input and feedback methods to the subject when a high degree of control is desired. Our new task can provide a complement to traditional center-out paradigms to help boost the real-world relevance of BCI research in the lab.