Reinforcement learning versus proportional-integral-derivative control of hypnosis in a simulated intraoperative patient.

BACKGROUND: Research has demonstrated the efficacy of closed-loop control of anesthesia using bispectral index (BIS) as the controlled variable. Model-based and proportional-integral-derivative (PID) controllers outperform manual control. We investigated the application of reinforcement learning (RL), an intelligent systems control method, to closed-loop BIS-guided, propofol-induced hypnosis in simulated intraoperative patients. We also compared the performance of the RL agent against that of a conventional PID controller. METHODS: The RL and PID controllers were evaluated during propofol induction and maintenance of hypnosis. The patient-hypnotic episodes were designed to challenge both controllers with varying degrees of interindividual variation and noxious surgical stimulation. Each controller was tested in 1000 simulated patients, and control performance was assessed by calculating the median performance error (MDPE), median absolute performance error (MDAPE), Wobble, and Divergence for each controller group. A separate analysis was performed for the induction and maintenance phases of hypnosis. RESULTS: During maintenance, RL control demonstrated an MDPE of -1% and an MDAPE of 3.75%, with 80% of the time at BIS(target) ± 5. The PID controller yielded a MDPE of -8.5% and an MDAPE of 8.6%, with 57% of the time at BIS(target) ± 5. In comparison, the MDAPE in the worst-controlled patient of the RL group was observed to be almost half that of the worst-controlled patient in the PID group. CONCLUSIONS: When compared with the PID controller, RL control resulted in slower induction but less overshoot and faster attainment of steady state. No difference in interindividual patient variation and noxious destabilizing challenge on control performance was observed between the 2 patient groups.

Reinforcement learning: a novel method for optimal control of propofol-induced hypnosis.

Reinforcement learning (RL) is an intelligent systems technique with a history of success in difficult robotic control problems. Similar machine learning techniques, such as artificial neural networks and fuzzy logic, have been successfully applied to clinical control problems. Although RL presents a mathematically robust method of achieving optimal control in systems challenged with noise, nonlinearity, time delay, and uncertainty, no application of RL in clinical anesthesia has been reported.

Fuzzy control for closed-loop, patient-specific hypnosis in intraoperative patients: a simulation study.

Research has demonstrated the efficacy of closed-loop control of anesthesia using bispectral index (BIS) as the controlled variable, and the recent development of model-based, patient-adaptive systems has considerably improved anesthetic control. To further explore the use of model-based control in anesthesia, we investigated the application of fuzzy control in the delivery of patient-specific propofol-induced hypnosis. In simulated intraoperative patients, the fuzzy controller demonstrated clinically acceptable performance, suggesting that further study is warranted.