Parameterized MDPs and Reinforcement Learning Problems-A Maximum Entropy Principle-Based Framework.

We present a framework to address a class of sequential decision-making problems. Our framework features learning the optimal control policy with robustness to noisy data, determining the unknown state and action parameters, and performing sensitivity analysis with respect to problem parameters. We consider two broad categories of sequential decision-making problems modeled as infinite horizon Markov decision processes (MDPs) with (and without) an absorbing state. The central idea underlying our framework is to quantify exploration in terms of the Shannon entropy of the trajectories under the MDP and determine the stochastic policy that maximizes it while guaranteeing a low value of the expected cost along a trajectory. This resulting policy enhances the quality of exploration early on in the learning process, and consequently allows faster convergence rates and robust solutions even in the presence of noisy data as demonstrated in our comparisons to popular algorithms, such as Q -learning, Double Q -learning, and entropy regularized Soft Q -learning. The framework extends to the class of parameterized MDP and RL problems, where states and actions are parameter dependent, and the objective is to determine the optimal parameters along with the corresponding optimal policy. Here, the associated cost function can possibly be nonconvex with multiple poor local minima. Simulation results applied to a 5G small cell network problem demonstrate the successful determination of communication routes and the small cell locations. We also obtain sensitivity measures to problem parameters and robustness to noisy environment data.

Multi-objective optimization of peel and shear strengths in ultrasonic metal welding using machine learning-based response surface methodology.

Ultrasonic metal welding (UMW) is a solid-state joining technique with varied industrial applications. Despite of its numerous advantages, UMW has a relative narrow operating window and is sensitive to variations in process conditions. As such, it is imperative to quantitatively characterize the influence of welding parameters on the resulting joint quality. The quantification model can be subsequently used to optimize the parameters. Conventional response surface methodology (RSM) usually employs linear or polynomial models, which may not be able to capture the intricate, nonlin-ear input-output relationships in UMW. Furthermore, some UMW applications call for simultaneous optimization of multiple quality indices such as peel strength, shear strength, electrical conductivity, and thermal conductivity. To address these challenges, this paper develops a machine learning (ML)- based RSM to model the input-output relationships in UMW and jointly optimize two quality indices, namely, peel and shear strengths. The performance of various ML methods including spline regression, Gaussian process regression (GPR), support vector regression (SVR), and conventional polynomial re-gression models with different orders is compared. A case study using experimental data shows that GPR with radial basis function (RBF) kernel and SVR with RBF kernel achieve the best prediction accuracy. The obtained response surface models are then used to optimize a compound joint strength indicator that is defined as the average of normalized shear and peel strengths. In addition, the case study reveals different patterns in the response surfaces of shear and peel strengths, which has not been systematically studied in the literature. While developed for the UMW application, the method can be extended to other manufacturing processes.

Extreme Antiscaling Performance of Slippery Omniphobic Covalently Attached Liquids.

Scale formation presents an enormous cost to the global economy. Classical nucleation theory dictates that to reduce the heterogeneous nucleation of scale, the surface should have low surface energy and be as smooth as possible. Past approaches have focused on lowering surface energy via the use of hydrophobic coatings and have created atomically smooth interfaces to eliminate nucleation sites, or both, via the infusion of low-surface-energy lubricants into rough superhydrophobic substrates. Although lubricant-based surfaces are promising candidates for antiscaling, lubricant drainage inhibits their utilization. Here, we develop methodologies to deposit slippery omniphobic covalently attached liquids (SOCAL) on arbitrary substrates. Similar to lubricant-based surfaces, SOCAL has ultralow roughness and surface energy, enabling low nucleation rates and eliminating the need to replenish the lubricant. To enable SOCAL coating on metals, we investigated the surface chemistry required to ensure high-quality functionalization as measured by ultralow contact angle hysteresis (<3°). Using a multilayer deposition approach, we first electrophoretically deposit (EPD) silicon dioxide (SiO2) as an intermediate layer between the metallic substrate and SOCAL. The necessity of EPD SiO2 is to smooth (<10 nm roughness) as well as to enable the proper surface chemistry for SOCAL bonding. To characterize antiscaling performance, we utilized calcium sulfate (CaSO4) scale tests, showing a 20× reduction in scale deposition rate than untreated metallic substrates. Descaling tests revealed that SOCAL dramatically decreases scale adhesion, resulting in rapid removal of scale buildup. Our work not only demonstrates a robust methodology for depositing antiscaling SOCAL coatings on metals but also develops design guidelines for the creation of antifouling coatings for alternate applications such as biofouling and high-temperature coking.

Robust atomic force microscopy using multiple sensors.

Atomic force microscopy typically relies on high-resolution high-bandwidth cantilever deflection measurements based control for imaging and estimating sample topography and properties. More precisely, in amplitude-modulation atomic force microscopy (AM-AFM), the control effort that regulates deflection amplitude is used as an estimate of sample topography; similarly, contact-mode AFM uses regulation of deflection signal to generate sample topography. In this article, a control design scheme based on an additional feedback mechanism that uses vertical z-piezo motion sensor, which augments the deflection based control scheme, is proposed and evaluated. The proposed scheme exploits the fact that the piezo motion sensor, though inferior to the cantilever deflection signal in terms of resolution and bandwidth, provides information on piezo actuator dynamics that is not easily retrievable from the deflection signal. The augmented design results in significant improvements in imaging bandwidth and robustness, especially in AM-AFM, where the complicated underlying nonlinear dynamics inhibits estimating piezo motions from deflection signals. In AM-AFM experiments, the two-sensor based design demonstrates a substantial improvement in robustness to modeling uncertainties by practically eliminating the peak in the sensitivity plot without affecting the closed-loop bandwidth when compared to a design that does not use the piezo-position sensor based feedback. The contact-mode imaging results, which use proportional-integral controllers for cantilever-deflection regulation, demonstrate improvements in bandwidth and robustness to modeling uncertainties, respectively, by over 30% and 20%. The piezo-sensor based feedback is developed using H∞ control framework.

Fast and robust control of nanopositioning systems: Performance limits enabled by field programmable analog arrays.

This paper aims at control design and its implementation for robust high-bandwidth precision (nanoscale) positioning systems. Even though modern model-based control theoretic designs for robust broadband high-resolution positioning have enabled orders of magnitude improvement in performance over existing model independent designs, their scope is severely limited by the inefficacies of digital implementation of the control designs. High-order control laws that result from model-based designs typically have to be approximated with reduced-order systems to facilitate digital implementation. Digital systems, even those that have very high sampling frequencies, provide low effective control bandwidth when implementing high-order systems. In this context, field programmable analog arrays (FPAAs) provide a good alternative to the use of digital-logic based processors since they enable very high implementation speeds, moreover with cheaper resources. The superior flexibility of digital systems in terms of the implementable mathematical and logical functions does not give significant edge over FPAAs when implementing linear dynamic control laws. In this paper, we pose the control design objectives for positioning systems in different configurations as optimal control problems and demonstrate significant improvements in performance when the resulting control laws are applied using FPAAs as opposed to their digital counterparts. An improvement of over 200% in positioning bandwidth is achieved over an earlier digital signal processor (DSP) based implementation for the same system and same control design, even when for the DSP-based system, the sampling frequency is about 100 times the desired positioning bandwidth.

Model based control of dynamic atomic force microscope.

A model-based robust control approach is proposed that significantly improves imaging bandwidth for the dynamic mode atomic force microscopy. A model for cantilever oscillation amplitude and phase dynamics is derived and used for the control design. In particular, the control design is based on a linearized model and robust H(∞) control theory. This design yields a significant improvement when compared to the conventional proportional-integral designs and verified by experiments.

A scalable approach to combinatorial library design.

In this chapter, we describe an algorithm for the design of lead-generation libraries required in combinatorial drug discovery. This algorithm addresses simultaneously the two key criteria of diversity and representativeness of compounds in the resulting library and is computationally efficient when applied to a large class of lead-generation design problems. At the same time, additional constraints on experimental resources are also incorporated in the framework presented in this chapter. A computationally efficient scalable algorithm is developed, where the ability of the deterministic annealing algorithm to identify clusters is exploited to truncate computations over the entire dataset to computations over individual clusters. An analysis of this algorithm quantifies the trade-off between the error due to truncation and computational effort. Results applied on test datasets corroborate the analysis and show improvement by factors as large as ten or more depending on the datasets.

Measuring robustness of the postural control system to a mild impulsive perturbation.

We propose a new metric to assess robustness of the human postural control system to an impulsive perturbation (in this case, a mild backward impulse force at the pelvis). By applying concepts from robust control theory, we use the inverse of the maximum value of the system's sensitivity function (1/MaxSens) as a measure for robustness of the human postural control system, e.g., a highly sensitive system has low robustness to perturbation. The sensitivity function, which in this case is the frequency response function, is obtained directly using spectral analysis of experimental measurements, without need to develop a model of the postural control system. Common measures of robustness, gain and phase margins, however require a model to assess system robustness. To examine the efficacy of this approach, we tested thirty healthy subjects across three age groups: young (YA: 20-30 years), middle-aged (MA: 42-53 years), and older adults (OA: 71-79 years). The OA group was found to have reduced postural stability during quiet stance as detected by center of pressure measures of postural sway. The proposed robustness measure of 1/MaxSens was also found to be significantly smaller for OA than YA or MA ( p=0.001), implying reduced robustness among the older subjects in response to the perturbation. Gain and phase margins failed to detect any age-related differences. In summary, the proposed robustness characterization method is easy to implement, does not require a model for the postural control system, and was better able to detect differences in system robustness than model-based robustness metrics.

Robust broadband nanopositioning: fundamental trade-offs, analysis, and design in a two-degree-of-freedom control framework.

This paper studies and analyses fundamental trade-offs between positioning resolution, tracking bandwidth, and robustness to modeling uncertainties in two-degree-of-freedom (2DOF) control designs for nanopositioning systems. The analysis of these systems is done in optimal control setting with various architectural constraints imposed on the 2DOF framework. In terms of these trade-offs, our analysis shows that the primary role of feedback is providing robustness to the closed-loop device whereas the feedforward component is mainly effective in overcoming fundamental algebraic constraints that limit the feedback-only designs. This paper presents (1) an optimal prefilter model matching design for a system with an existing feedback controller, (2) a simultaneous feedforward and feedback control design in an optimal mixed sensitivity framework, and (3) a 2DOF optimal robust model matching design. The experimental results on applying these controllers show a significant improvement, as high as 330% increase in bandwidth for similar robustness and resolution over optimal feedback-only designs. Other performance objectives can be improved similarly. We demonstrate that the 2DOF freedom design achieves performance specifications that are analytically impossible for feedback-only designs.

A scalable approach to combinatorial library design for drug discovery.

In this paper, we propose an algorithm for the design of lead generation libraries required in combinatorial drug discovery. This algorithm addresses simultaneously the two key criteria of diversity and representativeness of compounds in the resulting library and is computationally efficient when applied to a large class of lead generation design problems. At the same time, additional constraints on experimental resources are also incorporated in the framework presented in this paper. A computationally efficient scalable algorithm is developed, where the ability of the deterministic annealing algorithm to identify clusters is exploited to truncate computations over the entire data set to computations over individual clusters. An analysis of this algorithm quantifies the tradeoff between the error due to truncation and computational effort. Results applied on test data sets corroborate the analysis and show improvement by factors as large as 10 or more, depending on the data sets.

Feedback based simultaneous correction of imaging artifacts due to geometrical and mechanical cross-talk and tip-sample stick in atomic force microscopy.

This paper presents a feedback scheme that simultaneously corrects, in real time, for the imaging artifacts caused by cantilever and photosensor misalignments as well as misinterpretations in relative lateral position of the tip with respect to the sample due to the tip-sample stick in atomic force microscopy (AFM). The optical beam bounce method, typically used in AFM for imaging, is sensitive to inaccuracies of cantilever geometry and the relative misalignment of the laser source, cantilever, and the laser sensitive diode from the intended design. These inaccuracies, which contribute to the geometrical cross-talk between the normal and the lateral signals, become prominent at the atomic and subnanometer scales, and thereby impede high resolution imaging studies. The feedback scheme accounts for these artifacts and makes imaging insensitive to, in fact, practically independent of these inaccuracies. This scheme counteracts the lateral twisting dynamics of the cantilever, and as a result, it avoids the misinterpretation problem of the relative lateral position of the cantilever tip from the sample and thereby avoids the corresponding imaging artifacts that are typically prominent in contact mode friction force microscopy (FFM). The feedback scheme consists of simultaneously regulating the normal as well as the lateral cantilever deflection signal at their respective set points. This not only removes the imaging artifacts due to geometrical misalignments, mechanical cross-talk, and irregular sliding but also the corresponding compensatory control signal gives a more accurate real time measure of the lateral interaction force between the sample and the cantilever as compared to the lateral deflection signal used in FFM. Experimental results show significant improvement, and in some cases, practical elimination of the artifacts. The design and implementation of a split piezoassembly needed for the lateral actuation for the feedback scheme are also presented.