H∞ Static Output-Feedback Control Design for Discrete-Time Systems Using Reinforcement Learning.

This paper provides necessary and sufficient conditions for the existence of the static output-feedback (OPFB) solution to the H∞ control problem for linear discrete-time systems. It is shown that the solution of the static OPFB H∞ control is a Nash equilibrium point. Furthermore, a Q-learning algorithm is developed to find the H∞ OPFB solution online using data measured along the system trajectories and without knowing the system matrices. This is achieved by solving a game algebraic Riccati equation online and using the measured data. A simulation example shows the effectiveness of the proposed method.

Editorial Special Issue on Deep Reinforcement Learning and Adaptive Dynamic Programming.

The sixteen papers in this special section focus on deep reinforcement learning and adaptive dynamic programming (deep RL/ADP). Deep RL is able to output control signal directly based on input images, which incorporates both the advantages of the perception of deep learning (DL) and the decision making of RL or adaptive dynamic programming (ADP). This mechanism makes the artificial intelligence much closer to human thinking modes. Deep RL/ADP has achieved remarkable success in terms of theory and applications since it was proposed. Successful applications cover video games, Go, robotics, smart driving, healthcare, and so on. However, it is still an open problem to perform the theoretical analysis on deep RL/ADP, e.g., the convergence, stability, and optimality analyses. The learning efficiency needs to be improved by proposing new algorithms or combined with other methods. More practical demonstrations are encouraged to be presented. Therefore, the aim of this special issue is to call for the most advanced research and state-of-the-art works in the field of deep RL/ADP.

Reinforcement learning for partially observable dynamic processes: adaptive dynamic programming using measured output data.

Approximate dynamic programming (ADP) is a class of reinforcement learning methods that have shown their importance in a variety of applications, including feedback control of dynamical systems. ADP generally requires full information about the system internal states, which is usually not available in practical situations. In this paper, we show how to implement ADP methods using only measured input/output data from the system. Linear dynamical systems with deterministic behavior are considered herein, which are systems of great interest in the control system community. In control system theory, these types of methods are referred to as output feedback (OPFB). The stochastic equivalent of the systems dealt with in this paper is a class of partially observable Markov decision processes. We develop both policy iteration and value iteration algorithms that converge to an optimal controller that requires only OPFB. It is shown that, similar to Q -learning, the new methods have the important advantage that knowledge of the system dynamics is not needed for the implementation of these learning algorithms or for the OPFB control. Only the order of the system, as well as an upper bound on its "observability index," must be known. The learned OPFB controller is in the form of a polynomial autoregressive moving-average controller that has equivalent performance with the optimal state variable feedback gain.

Neural-network predictive control for nonlinear dynamic systems with time-delay.

A new recurrent neural-network predictive feedback control structure for a class of uncertain nonlinear dynamic time-delay systems in canonical form is developed and analyzed. The dynamic system has constant input and feedback time delays due to a communications channel. The proposed control structure consists of a linearized subsystem local to the controlled plant and a remote predictive controller located at the master command station. In the local linearized subsystem, a recurrent neural network with on-line weight tuning algorithm is employed to approximate the dynamics of the time-delay-free nonlinear plant. No linearity in the unknown parameters is required. No preliminary off-line weight learning is needed. The remote controller is a modified Smith predictor that provides prediction and maintains the desired tracking performance; an extra robustifying term is needed to guarantee stability. Rigorous stability proofs are given using Lyapunov analysis. The result is an adaptive neural net compensation scheme for unknown nonlinear systems with time delays. A simulation example is provided to demonstrate the effectiveness of the proposed control strategy.

Neural-network approximation of piecewise continuous functions: application to friction compensation.

One of the most important properties of neural nets (NNs) for control purposes is the universal approximation property. Unfortunately,, this property is generally proven for continuous functions. In most real industrial control systems there are nonsmooth functions (e.g., piecewise continuous) for which approximation results in the literature are sparse. Examples include friction, deadzone, backlash, and so on. It is found that attempts to approximate piecewise continuous functions using smooth activation functions require many NN nodes and many training iterations, and still do not yield very good results. Therefore, a novel neural-network structure is given for approximation of piecewise continuous functions of the sort that appear in friction, deadzone, backlash, and other motion control actuator nonlinearities. The novel NN consists of neurons having standard sigmoid activation functions, plus some additional neurons having a special class of nonsmooth activation functions termed "jump approximation basis function." Two types of nonsmooth jump approximation basis functions are determined- a polynomial-like basis and a sigmoid-like basis. This modified NN with additional neurons having "jump approximation" activation functions can approximate any piecewise continuous function with discontinuities at a finite number of known points. Applications of the new NN structure are made to rigid-link robotic systems with friction nonlinearities. Friction is a nonlinear effect that can limit the performance of industrial control systems; it occurs in all mechanical systems and therefore is unavoidable in control systems. It can cause tracking errors, limit cycles, and other undesirable effects. Often, inexact friction compensation is used with standard adaptive techniques that require models that are linear in the unknown parameters. It is shown here how a certain class of augmented NN, capable of approximating piecewise continuous functions, can be used for friction compensation.

Robust backstepping control of induction motors using neural networks.

In this paper, we present a new robust control technique for induction motors using neural networks (NNs). The method is systematic and robust to parameter variations. Motivated by the well-known backstepping design technique, we first treat certain signals in the system as fictitious control inputs to a simpler subsystem. A two-layer NN is used in this stage to design the fictitious controller. Then we apply a second two-layer NN to robustly realize the fictitious NN signals designed in the previous step. A new tuning scheme is proposed which can guarantee the boundedness of tracking error and weight updates. A main advantage of our method is that we do not require regression matrices, so that no preliminary dynamical analysis is needed. Another salient feature of our NN approach is that the off-line learning phase is not needed. Full state feedback is needed for implementation. Load torque and rotor resistance can be unknown but bounded.

Robust neural-network control of rigid-link electrically driven robots.

A robust neural-network (NN) controller is proposed for the motion control of rigid-link electrically driven (RLED) robots. Two-layer NN's are used to approximate two very complicated nonlinear functions. The main advantage of our approach is that the NN weights are tuned on-line, with no off-line learning phase required. Most importantly, we can guarantee the uniformly ultimately bounded (UUB) stability of tracking errors and NN weights. When compared with standard adaptive robot controllers, we do not require lengthy and tedious preliminary analysis to determine a regression matrix. The controller can be regarded as a universal reusable controller because the same controller can be applied to any type of RLED robots without any modifications.

Control of a nonholonomic mobile robot using neural networks.

A control structure that makes possible the integration of a kinematic controller and a neural network (NN) computed-torque controller for nonholonomic mobile robots is presented. A combined kinematic/torque control law is developed using backstepping and stability is guaranteed by Lyapunov theory. This control algorithm can be applied to the three basic nonholonomic navigation problems: tracking a reference trajectory, path following, and stabilization about a desired posture. Moreover, the NN controller proposed in this work can deal with unmodeled bounded disturbances and/or unstructured unmodeled dynamics in the vehicle. On-line NN weight tuning algorithms do no require off-line learning yet guarantee small tracking errors and bounded control signals are utilized.

Multilayer discrete-time neural-net controller with guaranteed performance.

A family of novel multilayer discrete-time neural-net (NN) controllers is presented for the control of a class of multi-input multi-output (MIMO) dynamical systems. The neural net controller includes modified delta rule weight tuning and exhibits a learning while-functioning-features. The structure of the NN controller is derived using a filtered error/passivity approach. Linearity in the parameters is not required and certainty equivalence is not used. This overcomes several limitations of standard adaptive control. The notion of persistency of excitation (PE) for multilayer NN is defined and explored. New online improved tuning algorithms for discrete-time systems are derived, which are similar to sigma or epsilon-modification for the case of continuous-time systems, that include a modification to the learning rate parameter plus a correction term. These algorithms guarantee tracking as well as bounded NN weights in nonideal situations so that PE is not needed. An extension of these novel weight tuning updates to NN with an arbitrary number of hidden layers is discussed. The notions of discrete-time passive NN, dissipative NN, and robust NN are introduced. The NN makes the closed-loop system passive.

Multilayer neural-net robot controller with guaranteed tracking performance.

A multilayer neural-net (NN) controller for a general serial-link rigid robot arm is developed. The structure of the NN controller is derived using a filtered error/passivity approach. No off-line learning phase is needed for the proposed NN controller and the weights are easily initialized. The nonlinear nature of the NN, plus NN functional reconstruction inaccuracies and robot disturbances, mean that the standard delta rule using backpropagation tuning does not suffice for closed-loop dynamic control. Novel online weight tuning algorithms, including correction terms to the delta rule plus an added robust signal, guarantee bounded tracking errors as well as bounded NN weights. Specific bounds are determined, and the tracking error bound can be made arbitrarily small by increasing a certain feedback gain. The correction terms involve a second-order forward-propagated wave in the backpropagation network. New NN properties including the notions of a passive NN, a dissipative NN, and a robust NN are introduced.

Neural net robot controller with guaranteed tracking performance.

A neural net (NN) controller for a general serial-link robot arm is developed. The NN has two layers so that linearity in the parameters holds, but the "net functional reconstruction error" and robot disturbance input are taken as nonzero. The structure of the NN controller is derived using a filtered error/passivity approach, leading to new NN passivity properties. Online weight tuning algorithms including a correction term to backpropagation, plus an added robustifying signal, guarantee tracking as well as bounded NN weights. The NN controller structure has an outer tracking loop so that the NN weights are conveniently initialized at zero, with learning occurring online in real-time. It is shown that standard backpropagation, when used for real-time closed-loop control, can yield unbounded NN weights if (1) the net cannot exactly reconstruct a certain required control function or (2) there are bounded unknown disturbances in the robot dynamics. The role of persistency of excitation is explored.

Control of respiratory therapy in flail chest.

Pulmonary mechanics and oxygenation were measured in 24 consecutive patients with posttraumatic flail chest requiring continuous mechanical ventilation. The mean duration of mechanical ventilation was fourteen days. Mortality was 38% for all patients, 29% if deaths from head injury are excluded. Pneumonia occurred in 4 patients (17%) and pneumothorax in 1 (4%). Vital capacity and maximal inspiratory force measurements were useful in assessing chest wall stabilization. Total lung compliance correlated negatively with fatal outcome from respiratory failure. The alveolar-arterial oxygen gradient was not useful in assessing chest wall stabilization.