Application of Quantum-Markov Open System Models to Human Cognition and Decision.

Markov processes, such as random walk models, have been successfully used by cognitive and neural scientists to model human choice behavior and decision time for over 50 years. Recently, quantum walk models have been introduced as an alternative way to model the dynamics of human choice and confidence across time. Empirical evidence points to the need for both types of processes, and open system models provide a way to incorporate them both into a single process. However, some of the constraints required by open system models present challenges for achieving this goal. The purpose of this article is to address these challenges and formulate open system models that have good potential to make important advancements in cognitive science.

The child brain computes and utilizes internalized maternal choices.

As children grow, they gradually learn how to make decisions independently. However, decisions like choosing healthy but less-tasty foods can be challenging for children whose self-regulation and executive cognitive functions are still maturing. We propose a computational decision-making process in which children estimate their mother's choices for them as well as their individual food preferences. By employing functional magnetic resonance imaging during real food choices, we find that the ventromedial prefrontal cortex (vmPFC) encodes children's own preferences and the left dorsolateral prefrontal cortex (dlPFC) encodes the projected mom's choices for them at the time of children's choice. Also, the left dlPFC region shows an inhibitory functional connectivity with the vmPFC at the time of children's own choice. Our study suggests that in part, children utilize their perceived caregiver's choices when making choices for themselves, which may serve as an external regulator of decision-making, leading to optimal healthy decisions.

Fixed-final-time optimal control of nonlinear systems with terminal constraints.

A model-based reinforcement learning algorithm is developed in this paper for fixed-final-time optimal control of nonlinear systems with soft and hard terminal constraints. Convergence of the algorithm, for linear in the weights neural networks, is proved through a novel idea by showing that the training algorithm is a contraction mapping. Once trained, the developed neurocontroller is capable of solving this class of optimal control problems for different initial conditions, different final times, and different terminal constraint surfaces providing some mild conditions hold. Three examples are provided and the numerical results demonstrate the versatility and the potential of the developed technique.

Optimal neurocontroller synthesis for impulse-driven systems.

This paper presents a new controller design technique for systems driven with impulse inputs. Necessary conditions for optimal impulse control are derived. A neural network structure to solve the resulting equations for optimal control is presented. Solution concepts are illustrated with example problems that exhibit increasing levels of difficulty. Two linear problems, one scalar and one vector and a benchmark nonlinear problem, the Van Der Pol oscillator, are used as case studies. Numerical results show the efficacy of the new solution process for impulse driven systems. Since the theoretical development and the design technique are free from restrictive assumptions, this technique is applicable to many problems in engineering and science.

Issues on stability of ADP feedback controllers for dynamical systems.

This paper traces the development of neural-network (NN)-based feedback controllers that are derived from the principle of adaptive/approximate dynamic programming (ADP) and discusses their closed-loop stability. Different versions of NN structures in the literature, which embed mathematical mappings related to solutions of the ADP-formulated problems called "adaptive critics" or "action-critic" networks, are discussed. Distinction between the two classes of ADP applications is pointed out. Furthermore, papers in "model-free" development and model-based neurocontrollers are reviewed in terms of their contributions to stability issues. Recent literature suggests that work in ADP-based feedback controllers with assured stability is growing in diverse forms.

Robust/optimal temperature profile control of a high-speed aerospace vehicle using neural networks.

An approximate dynamic programming (ADP)-based suboptimal neurocontroller to obtain desired temperature for a high-speed aerospace vehicle is synthesized in this paper. A 1-D distributed parameter model of a fin is developed from basic thermal physics principles. "Snapshot" solutions of the dynamics are generated with a simple dynamic inversion-based feedback controller. Empirical basis functions are designed using the "proper orthogonal decomposition" (POD) technique and the snapshot solutions. A low-order nonlinear lumped parameter system to characterize the infinite dimensional system is obtained by carrying out a Galerkin projection. An ADP-based neurocontroller with a dual heuristic programming (DHP) formulation is obtained with a single-network-adaptive-critic (SNAC) controller for this approximate nonlinear model. Actual control in the original domain is calculated with the same POD basis functions through a reverse mapping. Further contribution of this paper includes development of an online robust neurocontroller to account for unmodeled dynamics and parametric uncertainties inherent in such a complex dynamic system. A neural network (NN) weight update rule that guarantees boundedness of the weights and relaxes the need for persistence of excitation (PE) condition is presented. Simulation studies show that in a fairly extensive but compact domain, any desired temperature profile can be achieved starting from any initial temperature profile. Therefore, the ADP and NN-based controllers appear to have the potential to become controller synthesis tools for nonlinear distributed parameter systems.

A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems.

Even though dynamic programming offers an optimal control solution in a state feedback form, the method is overwhelmed by computational and storage requirements. Approximate dynamic programming implemented with an Adaptive Critic (AC) neural network structure has evolved as a powerful alternative technique that obviates the need for excessive computations and storage requirements in solving optimal control problems. In this paper, an improvement to the AC architecture, called the "Single Network Adaptive Critic (SNAC)" is presented. This approach is applicable to a wide class of nonlinear systems where the optimal control (stationary) equation can be explicitly expressed in terms of the state and costate variables. The selection of this terminology is guided by the fact that it eliminates the use of one neural network (namely the action network) that is part of a typical dual network AC setup. As a consequence, the SNAC architecture offers three potential advantages: a simpler architecture, lesser computational load and elimination of the approximation error associated with the eliminated network. In order to demonstrate these benefits and the control synthesis technique using SNAC, two problems have been solved with the AC and SNAC approaches and their computational performances are compared. One of these problems is a real-life Micro-Electro-Mechanical-system (MEMS) problem, which demonstrates that the SNAC technique is applicable to complex engineering systems.

Proper orthogonal decomposition based optimal neurocontrol synthesis of a chemical reactor process using approximate dynamic programming.

The concept of approximate dynamic programming and adaptive critic neural network based optimal controller is extended in this study to include systems governed by partial differential equations. An optimal controller is synthesized for a dispersion type tubular chemical reactor, which is governed by two coupled nonlinear partial differential equations. It consists of three steps: First, empirical basis functions are designed using the 'Proper Orthogonal Decomposition' technique and a low-order lumped parameter system to represent the infinite-dimensional system is obtained by carrying out a Galerkin projection. Second, approximate dynamic programming technique is applied in a discrete time framework, followed by the use of a dual neural network structure called adaptive critics, to obtain optimal neurocontrollers for this system. In this structure, one set of neural networks captures the relationship between the state variables and the control, whereas the other set captures the relationship between the state and the costate variables. Third, the lumped parameter control is then mapped back to the spatial dimension using the same basis functions to result in a feedback control. Numerical results are presented that illustrate the potential of this approach. It should be noted that the procedure presented in this study can be used in synthesizing optimal controllers for a fairly general class of nonlinear distributed parameter systems.