Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain.

Deep reinforcement learning (DRL) and deep multiagent reinforcement learning (MARL) have achieved significant success across a wide range of domains, including game artificial intelligence (AI), autonomous vehicles, and robotics. However, DRL and deep MARL agents are widely known to be sample inefficient that millions of interactions are usually needed even for relatively simple problem settings, thus preventing the wide application and deployment in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how efficiently exploring the environment and collecting informative experiences that could benefit policy learning toward the optimal ones. This problem becomes more challenging in complex environments with sparse rewards, noisy distractions, long horizons, and nonstationary co-learners. In this article, we conduct a comprehensive survey on existing exploration methods for both single-agent RL and multiagent RL. We start the survey by identifying several key challenges to efficient exploration. Then, we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. Beyond the above two main branches, we also include other notable exploration methods with different ideas and techniques. In addition to algorithmic analysis, we provide a comprehensive and unified empirical comparison of different exploration methods for DRL on a set of commonly used benchmarks. According to our algorithmic and empirical investigation, we finally summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.

Biomimetic Liquid Metal-Elastomer Composited Foam with Adjustable Thermal Conductivity for Heat Control.

The application of traditional materials with constant thermal conductivity in time-varying thermal environments poses great challenges due to their inability of adjusting thermal conductivity according to different requirements, for which reason materials with adjustable thermal conductivity have attracted much attention. However, certain limitations induced by those materials' low softness or harsh adjustment conditions restrict them from being applied in heat dissipation and heat transfer scenarios. In this study, we report a biomimetic liquid metal-elastomer composited foam with adjustable thermal conductivity (B-LM-ECF). Inspired by the rationale of homeothermic animals regulating the thermal conductivity of their subcutaneous tissue, the prepared material adjusts its thermal conductivity via adjusting the volume proportion of liquid metal within it. The thermal conductivity of B-LM-ECF can be adjusted within the range of 0.11-8.4 W·m-1K-1. The adjustment factor η of B-LM-ECF is 76, which is defined as the ratio of the highest to the lowest thermal conductivity of the material. The material enabling reversible switching for itself from thermal insulation to heat dissipation. The prepared material exhibits 45 KPa of Young's modulus with the maximum fracture tensile rate of 600%, facilitating better covering for thermal management objects. We selected a power lithium battery and a smartphone as specific thermal management objects to demonstrate its practical application in thermal management experiment.