Cooperative Finitely Excited Learning for Dynamical Games.

In this article, we propose a way to enhance the learning framework for zero-sum games with dynamics evolving in continuous time. In contrast to the conventional centralized actor-critic learning, a novel cooperative finitely excited learning approach is developed to combine the online recorded data with instantaneous data for efficiency. By using an experience replay technique for each agent and distributed interaction amongst agents, we are able to replace the classical persistent excitation condition with an easy-to-check cooperative excitation condition. This approach also guarantees the consensus of the distributed actor-critic learning on the solution to the Hamilton-Jacobi-Isaacs (HJI) equation. It is shown that both the closed-loop stability of the equilibrium point and convergence to the Nash equilibrium can be guaranteed. Simulation results demonstrate the efficacy of this approach compared to previous methods.

Online and Robust Intermittent Motion Planning in Dynamic and Changing Environments.

In this article, we propose RRT-Q X∞ , an online and intermittent kinodynamic motion planning framework for dynamic environments with unknown robot dynamics and unknown disturbances. We leverage RRT X for global path planning and rapid replanning to produce waypoints as a sequence of boundary-value problems (BVPs). For each BVP, we formulate a finite-horizon, continuous-time zero-sum game, where the control input is the minimizer, and the worst case disturbance is the maximizer. We propose a robust intermittent Q-learning controller for waypoint navigation with completely unknown system dynamics, external disturbances, and intermittent control updates. We execute a relaxed persistence of excitation technique to guarantee that the Q-learning controller converges to the optimal controller. We provide rigorous Lyapunov-based proofs to guarantee the closed-loop stability of the equilibrium point. The effectiveness of the proposed RRT-Q X∞ is illustrated with Monte Carlo numerical experiments in numerous dynamic and changing environments.

Intermittent Learning Through Operant Conditioning for Cyber-Physical Systems.

This article presents a novel scheme, namely, an intermittent learning scheme based on Skinner's operant conditioning techniques that approximates the optimal policy while decreasing the usage of the communication buses transferring information. While traditional reinforcement learning schemes continuously evaluate and subsequently improve, every action taken by a specific learning agent based on received reinforcement signals, this form of continuous transmission of reinforcement signals and policy improvement signals can cause overutilization of the system's inherently limited resources. Moreover, the highly complex nature of the operating environment for cyber-physical systems (CPSs) creates a gap for malicious individuals to corrupt the signal transmissions between various components. The proposed schemes will increase uncertainty in the learning rate and the extinction rate of the acquired behavior of the learning agents. In this article, we investigate the use of fixed/variable interval and fixed/variable ratio schedules in CPSs along with their rate of success and loss in their optimal behavior incurred during intermittent learning. Simulation results show the efficacy of the proposed approach.

Recursive Reasoning With Reduced Complexity and Intermittency for Nonequilibrium Learning in Stochastic Games.

In this article, we propose a computationally and communicationally efficient approach for decision-making in nonequilibrium stochastic games. In particular, due to the inherent complexity of computing Nash equilibria, as well as the innate tendency of agents to choose nonequilibrium strategies, we construct two models of bounded rationality based on recursive reasoning. In the first model, named level- k thinking, each agent assumes that everyone else has a cognitive level immediately lower than theirs and-given such an assumption-chooses their policy to be a best response to them. In the second model, named cognitive hierarchy, each agent conjectures that the rest of the agents have a cognitive level that is lower than theirs, but follows a distribution instead of being deterministic. To explicitly compute the boundedly rational policies, a level-recursive algorithm and a level-paralleled algorithm are constructed, where the latter one can have an overall reduced computational complexity. To further reduce the complexity in the communication layer, modifications of the proposed nonequilibrium strategies are presented, which do not require the action of a boundedly rational agent to be updated at each step of the stochastic game. Simulations are performed that demonstrate our results.

Safety-Aware Pursuit-Evasion Games in Unknown Environments Using Gaussian Processes and Finite-Time Convergent Reinforcement Learning.

This article develops a safe pursuit-evasion game for enabling finite-time capture, optimal performance as well as adaptation to an unknown cluttered environment. The pursuit-evasion game is formulated as a zero-sum differential game wherein the pursuer seeks to minimize its relative distance to the target while the evader attempts to maximize it. A critic-only reinforcement learning (RL)-based algorithm is then proposed for learning online and in finite time the pursuit-evasion policies and thus enabling finite-time capture of the evader. Safety is ensured by means of barrier functions associated with the obstacles, which are integrated into the running cost. Using Gaussian processes (GPs), a learning-based mechanism is devised for safely learning the unknown environment. Simulation results illustrate the efficacy of the proposed approach.

Mapping Glucose Uptake, Transport and Metabolism in the Bovine Lens Cortex.

Purpose: To spatially correlate the pattern of glucose uptake to glucose transporter distributions in cultured lenses and map glucose metabolism in different lens regions. Methods: Ex vivo bovine lenses were incubated in artificial aqueous humour containing normoglycaemic stable isotopically-labelled (SIL) glucose (5 mM) for 5 min-20 h. Following incubations, lenses were frozen for subsequent matrix-assisted laser desorption/ionisation (MALDI) imaging mass spectrometry (IMS) analysis using high resolution mass spectrometry. Manually dissected, SIL-incubated lenses were subjected to gas chromatography-mass spectrometry (GC-MS) to verify the identity of metabolites detected by MALDI-IMS. Normal, unincubated lenses were manually dissected into epithelium flat mounts and fibre cell fractions and then subjected to either gel-based proteomic analysis (Gel-LC/MS) to detect facilitative glucose transporters (GLUTs) by liquid chromatography tandem mass spectrometry (LC-MS/MS). Indirect immunofluorescence and confocal microscopy of axial lens sections from unincubated fixed lenses labelled with primary antibodies specific for GLUT 1 or GLUT 3 were utilised for protein localisation. Results: SIL glucose uptake at 5 min was concentrated in the equatorial region of the lens. At later timepoints, glucose gradually distributed throughout the epithelium and the cortical lens fibres, and eventually the deeper lens nucleus. SIL glucose metabolites found in glycolysis, the sorbitol pathway, the pentose phosphate pathway, and UDP-glucose formation were mapped to specific lens regions, with distinct regional signal changes up to 20 h of incubation. Spatial proteomic analysis of the lens epithelium detected GLUT1 and GLUT3. GLUT3 was in higher abundance than GLUT1 throughout the epithelium, while GLUT1 was more abundant in lens fibre cells. Immunohistochemical mapping localised GLUT1 to epithelial and cortical fibre cell membranes. Conclusion: The major uptake site of glucose in the bovine lens has been mapped to the lens equator. SIL glucose is rapidly metabolised in epithelial and fibre cells to many metabolites, which are most abundant in the metabolically more active cortical fibre cells in comparison to central fibres, with low levels of metabolic activity observed in the nucleus.

Adaptive Neural Network Stochastic-Filter-Based Controller for Attitude Tracking With Disturbance Rejection.

This article proposes a real-time neural network (NN) stochastic filter-based controller on the Lie group of the special orthogonal group [Formula: see text] as a novel approach to the attitude tracking problem. The introduced solution consists of two parts: a filter and a controller. First, an adaptive NN-based stochastic filter is proposed, which estimates attitude components and dynamics using measurements supplied by onboard sensors directly. The filter design accounts for measurement uncertainties inherent to the attitude dynamics, namely, unknown bias and noise corrupting angular velocity measurements. The closed-loop signals of the proposed NN-based stochastic filter have been shown to be semiglobally uniformly ultimately bounded (SGUUB). Second, a novel control law on [Formula: see text] coupled with the proposed estimator is presented. The control law addresses unknown disturbances. In addition, the closed-loop signals of the proposed filter-based controller have been shown to be SGUUB. The proposed approach offers robust tracking performance by supplying the required control signal given data extracted from low-cost inertial measurement units. While the filter-based controller is presented in continuous form, the discrete implementation is also presented. In addition, the unit-quaternion form of the proposed approach is given. The effectiveness and robustness of the proposed filter-based controller are demonstrated using its discrete form and considering low sampling rate, high initialization error, high level of measurement uncertainties, and unknown disturbances.

Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors.

In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, "min-Hamiltonian," is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.

Population Viability and Conservation Strategies for the Eurasian Black Vulture (Aegypius monachus) in Southeast Europe.

The Eurasian Black Vulture is a globally threatened raptor that in Southeast Europe only occurs in an isolated population in Greece. We examined the population viability for the species under demographic fluctuations and conservation scenarios. The current population showed no possibility of extinction for the next 100 years. However, simulated scenarios showed that the most important factor affecting the viability of the species was medium and high poisoning, leading to 94.8% and 100% probability of extinction, respectively. Furthermore, high reduction of supplementary feeding highlighted an 18.6% extinction possibility. Also, a high increase of wind farms in the area may result in 17.4% extinction possibility. Additionally, the non-establishment of the feeding station in 1987 in the study area would have resulted in an extinction risk of 7%. The species can be translocated to the Olympus National Park by releasing 80 juveniles over 10 years. The implementation of the conservation scenarios concerning the establishment of a supplementary feeding site network, and the reintroduction of the Eurasian Black Vulture in its historic range, along with the elimination of threats posed by poisoning, low food availability, and wind farms would increase the probability of the species persistence and allow the population to become a source for dispersal across Southeast Europe.

Cyclic peptides bearing the d-Phe-2-Abz turn motif: Structural characterization and antimicrobial potential.

The effect on secondary structure and antimicrobial activity of introducing different cyclic constraints in linear ß-hairpin antimicrobial peptides has been investigated with the intention of generating cyclic ß sheets as promising antimicrobials with improved therapeutic potential. The linear peptides were cyclized head to tail either directly or after the addition of either a second turn motif or a disulfide bridge. The propensity of these peptides to adopt a cyclic ß-sheet structure has been correlated to their antibacterial activity. All cyclic peptides showed enhanced activity, compared with their linear counterparts against methicillin-resistant Staphylococcus aureus. Scanning electron microscopy and transmission electron microscopy studies showed that this family kills bacteria through membrane lysis. The peptide that showed the best efficacy against all strains (exhibiting nanomolar activity), while retaining low haemolysis, bears two symmetrical, homochiral d-phe-2-Abz-d-ala turns and adopted a flexible structure. Its twin peptide that bears heterochiral turns (one with d-ala and one with L-Ala) showed reduced antibacterial activity and higher percentage of haemolysis. Circular dichroism and nuclear magnetic resonance spectroscopy indicate that heterochirality in the two turns leads to oligomerization of the peptide at higher concentrations, stabilizing the ß-sheet secondary structure. More rigid secondary structure is associated with lower activity against bacteria and loss of selectivity.

Safe Approximate Dynamic Programming via Kernelized Lipschitz Estimation.

We develop a method for obtaining safe initial policies for reinforcement learning via approximate dynamic programming (ADP) techniques for uncertain systems evolving with discrete-time dynamics. We employ the kernelized Lipschitz estimation to learn multiplier matrices that are used in semidefinite programming frameworks for computing admissible initial control policies with provably high probability. Such admissible controllers enable safe initialization and constraint enforcement while providing exponential stability of the equilibrium of the closed-loop system.

A Secure Control Learning Framework for Cyber-Physical Systems Under Sensor and Actuator Attacks.

In this article, we develop a learning-based secure control framework for cyber-physical systems in the presence of sensor and actuator attacks. Specifically, we use a bank of observer-based estimators to detect the attacks while introducing a threat-detection level function. Under nominal conditions, the system operates with a nominal-feedback controller with the developed attack monitoring process checking the reliance of the measurements. If there exists an attacker injecting attack signals to a subset of the sensors and/or actuators, then the attack mitigation process is triggered and a two-player, zero-sum differential game is formulated with the defender being the minimizer and the attacker being the maximizer. Next, we solve the underlying joint state estimation and attack mitigation problem and learn the secure control policy using a reinforcement-learning-based algorithm. Finally, two illustrative numerical examples are provided to show the efficacy of the proposed framework.

Synthesis and Antibacterial Analysis of Analogues of the Marine Alkaloid Pseudoceratidine.

In an effort to gain more understanding on the structure activity relationship of pseudoceratidine 1, a di-bromo pyrrole spermidine alkaloid derived from the marine sponge Pseudoceratina purpurea that has been shown to exhibit potent biofouling, anti-fungal, antibacterial, and anti-malarial activities, a large series of 65 compounds that incorporated several aspects of structural variation has been synthesised through an efficient, divergent method that allowed for a number of analogues to be generated from common precursors. Subsequently, all analogues were assessed for their antibacterial activity against both Gram-positive (Staphylococcus aureus) and Gram-negative (Escherichia coli) bacteria. Overall, several compounds exhibited comparable or better activity than that of pseudoceratidine 1, and it was found that this class of compounds is generally more effective against Gram-positive than Gram-negative bacteria. Furthermore, altering several structural features allowed for the establishment of a comprehensive structure activity relationship (SAR), where it was concluded that several structural features are critical for potent anti-bacterial activity, including di-halogenation (preferable bromine, but chlorine is also effective) on the pyrrole ring, two pyrrolic units in the structure and with one or more secondary amines in the chain adjoining these units, with longer chains giving rise to better activities.

An investigation into the effect of ribosomal protein S15 phosphorylation on its intermolecular interactions by using phosphomimetic mutant.

An investigation using recombinant ribosomal proteins and synthetic peptide models was conducted to uncover the effect of the introduction of a negative charge at the C-terminal tail of ribosomal protein S15. Our results help provide a chemical rationale towards understanding how G2019S LRRK2, a common clinical mutation, causes Parkinson's disease.

Safe Intermittent Reinforcement Learning With Static and Dynamic Event Generators.

In this article, we present an intermittent framework for safe reinforcement learning (RL) algorithms. First, we develop a barrier function-based system transformation to impose state constraints while converting the original problem to an unconstrained optimization problem. Second, based on optimal derived policies, two types of intermittent feedback RL algorithms are presented, namely, a static and a dynamic one. We finally leverage an actor/critic structure to solve the problem online while guaranteeing optimality, stability, and safety. Simulation results show the efficacy of the proposed approach.

Dynamic Intermittent Feedback Design for H∞ Containment Control on a Directed Graph.

This article develops a novel distributed intermittent control framework with the ultimate goal of reducing the communication burden in containment control of multiagent systems communicating via a directed graph. Agents are assumed to be under disturbance and communicate on a directed graph. Both static and dynamic intermittent protocols are proposed. Intermittent H∞ containment control design is considered to attenuate the effect of the disturbance and the game algebraic Riccati equation (GARE) is employed to design the coupling and feedback gains for both static and dynamic intermittent feedback. A novel scheme is then used to unify continuous, static, and dynamic intermittent containment protocols. Finally, simulation results verify the efficacy of the proposed approach.

A Compliant, Underactuated Finger for Anthropomorphic Hands.

This paper presents a compliant, underactuated finger for the development of anthropomorphic robotic and prosthetic hands. The finger achieves both flexion/extension and adduction/abduction on the metacarpophalangeal joint, by using two actuators. The design employs moment arm pulleys to drive the tendon laterally and amplify the abduction motion, while also maintaining the flexion motion. Particular emphasis has been given to the analysis of the mechanism. The proposed finger has been fabricated with the hybrid deposition manufacturing technique and the actuation mechanism's efficiency has been validated with experiments that include the computation of the reachable workspace, the assessment of the exerted forces at the fingertip, the demonstration of the feasible motions, and the presentation of the grasping and manipulation capabilities. The proposed mechanism facilitates the collaboration of the two actuators to increase the exerted finger forces. Moreover, the extended workspace allows the execution of dexterous manipulation tasks.

Cyclic Tetrapeptides from Nature and Design: A Review of Synthetic Methodologies, Structure, and Function.

Small cyclic peptides possess a wide range of biological properties and unique structures that make them attractive to scientists working in a range of areas from medicinal to materials chemistry. However, cyclic tetrapeptides (CTPs), which are important members of this family, are notoriously difficult to synthesize. Various synthetic methodologies have been developed that enable access to natural product CTPs and their rationally designed synthetic analogues having novel molecular structures. These methodologies include the use of reversible protecting groups such as pseudoprolines that restrict conformational freedom, ring contraction strategies, on-resin cyclization approaches, and optimization of coupling reagents and reaction conditions such as temperature and dilution factors. Several fundamental studies have documented the impacts of amino acid configurations, N-alkylation, and steric bulk on both synthetic success and ensuing conformations. Carefully executed retrosynthetic ring dissection and the unique structural features of the linear precursor sequences that result from the ring dissection are crucial for the success of the cyclization step. Other factors that influence the outcome of the cyclization step include reaction temperature, solvent, reagents used as well as dilution levels. The purpose of this review is to highlight the current state of affairs on naturally occurring and rationally designed cyclic tetrapeptides, including strategies investigated for their syntheses in the literature, the conformations adopted by these molecules, and specific examples of their function. Using selected examples from the literature, an in-depth discussion of the synthetic techniques and reaction parameters applied for the successful syntheses of 12-, 13-, and 14-membered natural product CTPs and their novel analogues are presented, with particular focus on the cyclization step. Selected examples of the three-dimensional structures of cyclic tetrapeptides studied by NMR, and X-ray crystallography are also included.

Kinodynamic Motion Planning With Continuous-Time Q-Learning: An Online, Model-Free, and Safe Navigation Framework.

This paper presents an online kinodynamic motion planning algorithmic framework using asymptotically optimal rapidly-exploring random tree (RRT*) and continuous-time Q-learning, which we term as RRT-Q⋆. We formulate a model-free Q-based advantage function and we utilize integral reinforcement learning to develop tuning laws for the online approximation of the optimal cost and the optimal policy of continuous-time linear systems. Moreover, we provide rigorous Lyapunov-based proofs for the stability of the equilibrium point, which results in asymptotic convergence properties. A terminal state evaluation procedure is introduced to facilitate the online implementation. We propose a static obstacle augmentation and a local replanning framework, which are based on topological connectedness, to locally recompute the robot's path and ensure collision-free navigation. We perform simulations and a qualitative comparison to evaluate the efficacy of the proposed methodology.

Making Solid-Phase Peptide Synthesis Greener: A Review of the Literature.

To date, the synthesis of peptides is concurrent with the production of enormous amounts of toxic waste. DMF, CH2 Cl2 , and NMP are three of the most toxic organic solvents used in chemical synthesis and are the most common solvents used for peptide synthesis. Additionally, concerns about the hepatotoxicity caused by exposure to DMF and from the toxic and allergenic nature of additives used in peptide synthesis necessitates the need for a green, environmentally friendly, and safer protocol for peptide synthesis. This review summarizes the current literature on green solid-phase peptide synthesis successes and challenges encountered. The review concludes with suggestions for future research towards a simple and efficient green peptide synthesis protocol.