Safe Reinforcement Learning via a Model-Free Safety Certifier.

This article presents a data-driven safe reinforcement learning (RL) algorithm for discrete-time nonlinear systems. A data-driven safety certifier is designed to intervene with the actions of the RL agent to ensure both safety and stability of its actions. This is in sharp contrast to existing model-based safety certifiers that can result in convergence to an undesired equilibrium point or conservative interventions that jeopardize the performance of the RL agent. To this end, the proposed method directly learns a robust safety certifier while completely bypassing the identification of the system model. The nonlinear system is modeled using linear parameter varying (LPV) systems with polytopic disturbances. To prevent the requirement for learning an explicit model of the LPV system, data-based λ -contractivity conditions are first provided for the closed-loop system to enforce robust invariance of a prespecified polyhedral safe set and the system's asymptotic stability. These conditions are then leveraged to directly learn a robust data-based gain-scheduling controller by solving a convex program. A significant advantage of the proposed direct safe learning over model-based certifiers is that it completely resolves conflicts between safety and stability requirements while assuring convergence to the desired equilibrium point. Data-based safety certification conditions are then provided using Minkowski functions. They are then used to seemingly integrate the learned backup safe gain-scheduling controller with the RL controller. Finally, we provide a simulation example to verify the effectiveness of the proposed approach.

Soft-tissue modeling and image-guided control of steerable needles.

In this paper, we exploit a fuzzy controller on a flexible bevel-tip needle to manipulate the needle's base in order to steer its tip in a preset obstacle-free and target-tracking path. Although the needle tends to follow a curvature path, spinning the needle with an extremely high rotational velocity makes it symmetric with respect to the tissue to follow a straight path. The fuzzy controller determines an appropriate spinning to generate the planned trajectory and, the closed-loop system tries to match the needle body with that trajectory. The swine's brain tissue model, extracted from an in-vitro experimental setup, is a non-homogenous, uncertain and fast-updatable network to model real tissues, needle and their interactions providing the essential visual feedback for the control system. The simulation results illustrate a precise path tracking of the bevel-tip needle based on the fuzzy controller's commands with two degrees of freedom.

Hierarchical optimal control of large-scale nonlinear chemical processes.

In this paper, a new approach is presented for optimal control of large-scale chemical processes. In this approach, the chemical process is decomposed into smaller sub-systems at the first level, and a coordinator at the second level, for which a two-level hierarchical control strategy is designed. For this purpose, each sub-system in the first level can be solved separately, by using any conventional optimization algorithm. In the second level, the solutions obtained from the first level are coordinated using a new gradient-type strategy, which is updated by the error of the coordination vector. The proposed algorithm is used to solve the optimal control problem of a complex nonlinear chemical stirred tank reactor (CSTR), where its solution is also compared with the ones obtained using the centralized approach. The simulation results show the efficiency and the capability of the proposed hierarchical approach, in finding the optimal solution, over the centralized method.