Reinforcement learning for closed-loop regulation of cardiovascular system with vagus nerve stimulation: a computational study.

Objective. Vagus nerve stimulation (VNS) is being investigated as a potential therapy for cardiovascular diseases including heart failure, cardiac arrhythmia, and hypertension. The lack of a systematic approach for controlling and tuning the VNS parameters poses a significant challenge. Closed-loop VNS strategies combined with artificial intelligence (AI) approaches offer a framework for systematically learning and adapting the optimal stimulation parameters. In this study, we presented an interactive AI framework using reinforcement learning (RL) for automated data-driven design of closed-loop VNS control systems in a computational study.Approach.Multiple simulation environments with a standard application programming interface were developed to facilitate the design and evaluation of the automated data-driven closed-loop VNS control systems. These environments simulate the hemodynamic response to multi-location VNS using biophysics-based computational models of healthy and hypertensive rat cardiovascular systems in resting and exercise states. We designed and implemented the RL-based closed-loop VNS control frameworks in the context of controlling the heart rate and the mean arterial pressure for a set point tracking task. Our experimental design included two approaches; a general policy using deep RL algorithms and a sample-efficient adaptive policy using probabilistic inference for learning and control.Main results.Our simulation results demonstrated the capabilities of the closed-loop RL-based approaches to learn optimal VNS control policies and to adapt to variations in the target set points and the underlying dynamics of the cardiovascular system. Our findings highlighted the trade-off between sample-efficiency and generalizability, providing insights for proper algorithm selection. Finally, we demonstrated that transfer learning improves the sample efficiency of deep RL algorithms allowing the development of more efficient and personalized closed-loop VNS systems.Significance.We demonstrated the capability of RL-based closed-loop VNS systems. Our approach provided a systematic adaptable framework for learning control strategies without requiring prior knowledge about the underlying dynamics.

CONTROL-CORE: A Framework for Simulation and Design of Closed-Loop Peripheral Neuromodulation Control Systems.

Closed-loop Vagus Nerve Stimulation (VNS) based on physiological feedback signals is a promising approach to regulate organ functions and develop therapeutic devices. Designing closed-loop neurostimulation systems requires simulation environments and computing infrastructures that support i) modeling the physiological responses of organs under neuromodulation, also known as physiological models, and ii) the interaction between the physiological models and the neuromodulation control algorithms. However, existing simulation platforms do not support closed-loop VNS control systems modeling without extensive rewriting of computer code and manual deployment and configuration of programs. The CONTROL-CORE project aims to develop a flexible software platform for designing and implementing closed-loop VNS systems. This paper proposes the software architecture and the elements of the CONTROL-CORE platform that allow the interaction between a controller and a physiological model in feedback. CONTROL-CORE facilitates modular simulation and deployment of closed-loop peripheral neuromodulation control systems, spanning multiple organizations securely and concurrently. CONTROL-CORE allows simulations to run on different operating systems, be developed in various programming languages (such as Matlab, Python, C++, and Verilog), and be run locally, in containers, and in a distributed fashion. The CONTROL-CORE platform allows users to create tools and testbenches to facilitate sophisticated simulation experiments. We tested the CONTROL-CORE platform in the context of closed-loop control of cardiac physiological models, including pulsatile and nonpulsatile rat models. These were tested using various controllers such as Model Predictive Control and Long-Short-Term Memory based controllers. Our wide range of use cases and evaluations show the performance, flexibility, and usability of the CONTROL-CORE platform.

Automated Tuning of Closed-loop Neuromodulation Control Systems using Bayesian Optimization.

Tuning the parameters of controllers to attain the best performance is a challenging task in designing effective closed-loop neuromodulation systems. In this paper, we present a distributed architecture for automated tuning and adaptation of closed-loop neuromodulation control systems. We use this approach for the automated parameter tuning of a Proportional-Integral (PI) neuromodulation controller using Bayesian optimization. We use a biophysically-grounded mean-field model of neural populations under electrical stimulation as a simulation environment for testing and prototyping the proposed framework and characterizing its performance. Our results demonstrate the feasibility of using Bayesian optimization for performance-based automated tuning of a PI controller in closed-loop set-point neuromodulation control tasks.

Automated deep brain stimulation programming with safety constraints for tremor suppression in patients with Parkinson's disease and essential tremor.

Objective.Deep brain stimulation (DBS) programming for movement disorders requires systematic fine tuning of stimulation parameters to ameliorate tremor and other symptoms while avoiding side effects. DBS programming can be a time-consuming process and requires clinical expertise to assess response to DBS to optimize therapy for each patient. In this study, we describe and evaluate an automated, closed-loop, and patient-specific framework for DBS programming that measures tremor using a smartwatch and automatically changes DBS parameters based on the recommendations from a closed-loop optimization algorithm thus eliminating the need for an expert clinician.Approach.Bayesian optimization which is a sample-efficient global optimization method was used as the core of this DBS programming framework to adaptively learn each patient's response to DBS and suggest the next best settings to be evaluated. Input from a clinician was used initially to define a maximum safe amplitude, but we also implemented 'safe Bayesian optimization' to automatically discover tolerable exploration boundaries.Main results.We tested the system in 15 patients (nine with Parkinson's disease and six with essential tremor). Tremor suppression at best automated settings was statistically comparable to previously established clinical settings. The optimization algorithm converged after testing15.1±0.7settings when maximum safe exploration boundaries were predefined, and17.7±4.9when the algorithm itself determined safe exploration boundaries.Significance.We demonstrate that fully automated DBS programming framework for treatment of tremor is efficient and safe while providing outcomes comparable to that achieved by expert clinicians.

Yin-Yang: Programming Abstractions for Cross-Domain Multi-Acceleration.

FPGA accelerators offer performance and efficiency gains by narrowing the scope of acceleration to one algorithmic domain. However, real-life applications are often not limited to a single domain, which naturally makes Cross-Domain Multi-Acceleration a crucial next step. The challenge is, existing FPGA accelerators are built upon their specific vertically-specialized stacks, which prevents utilizing multiple accelerators from different domains. To that end, we propose a pair of dual abstractions, called Yin-Yang, which work in tandem and enable programmers to develop cross-domain applications using multiple accelerators on a FPGA. The Yin abstraction enables cross-domain algorithmic specification, while the Yang abstraction captures the accelerator capabilities. We also develop a dataflow virtual machine, dubbed XLVM, that transparently maps domain functions (Yin) to best-fit accelerator capabilities (Yang). With six real-world cross-domain applications, our evaluations show that Yin-Yang unlocks 29.4× speedup, while the best single-domain acceleration achieves 12.0×.

Software-Defined Workflows for Distributed Interoperable Closed-Loop Neuromodulation Control Systems.

Closed-loop neuromodulation control systems facilitate regulating abnormal physiological processes by recording neurophysiological activities and modifying those activities through feedback loops. Designing such systems requires interoperable service composition, consisting of cycles. Workflow frameworks enable standard modular architectures, offering reproducible automated pipelines. However, those frameworks limit their support to executions represented by directed acyclic graphs (DAGs). DAGs need a pre-defined start and end execution step with no cycles, thus preventing the researchers from using the standard workflow languages as-is for closed-loop workflows and pipelines. In this paper, we present NEXUS, a workflow orchestration framework for distributed analytics systems. NEXUS proposes a Software-Defined Workflows approach, inspired by Software-Defined Networking (SDN), which separates the data flows across the service instances from the control flows. NEXUS enables creating interoperable workflows with closed loops by defining the workflows in a logically centralized approach, from microservices representing each execution step. The centralized NEXUS orchestrator facilitates dynamically composing and managing scientific workflows from the services and existing workflows, with minimal restrictions. NEXUS represents complex workflows as directed hypergraphs (DHGs) rather than DAGs. We illustrate a seamless execution of neuromodulation control systems by supporting loops in a workflow as the use case of NEXUS. Our evaluations highlight the feasibility, flexibility, performance, and scalability of NEXUS in modeling and executing closed-loop workflows.

Towards automated patient-specific optimization of deep brain stimulation for movement disorders.

In this paper we present a simulation framework for automated optimization of deep brain stimulation (DBS) parameters based on the hand kinematics signal as the feedback signal, in patients with essential tremor. We used Gaussian Process regression (GPR) models to develop patient-specific models for predicting the effect of DBS on the hand kinematics using the clinical data that was recorded during DBS programming. In this framework, we characterized the performance of a Bayesian Optimization method to identify the optimal DBS parameters that maximized the clinical efficacy. Our results demonstrate the feasibility of using black-box optimization methods for automated identification of optimal DBS parameters in clinical settings.