RoMAT: Role-based multi-agent transformer for generalizable heterogeneous cooperation.

Multi-task multi-agent systems (MASs) are challenging to model because they involve heterogeneous agents with different behavior patterns that need to cooperate across various tasks. Existing networks for single-agent policies are not suitable for this setting, as they cannot share policies among agents without losing task-specific performance. We propose a novel framework called Role-based Multi-Agent Transformer (RoMAT), which uses a sequence modeling technique and a role-based actor to enable agents to adapt to different tasks and roles in MASs. RoMAT has a modular model architecture, where backbone networks are shared by all agents, but a small part of the parameters (role-based actor) is independent, depending on the agents' exclusive structures. We evaluate RoMAT on several benchmark tasks and show that it can capture the behavior patterns of heterogeneous agents and achieve better performance and generalization than other methods in both single and multi-task settings.

Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation.

Achieving human-level dexterity in robotics remains a critical open problem. Even simple dexterous manipulation tasks pose significant difficulties due to the high number of degrees of freedom and the need for cooperation among heterogeneous agents (e.g., finger joints). While some researchers have utilized reinforcement learning (RL) to control a single hand in manipulating objects, tasks that require coordinated bimanual cooperation are still under-explored due to the fewer suitable environments, which can result in difficulties and sub-optimal performance. To address these challenges, we introduce Bi-DexHands, a simulator with two dexterous hands featuring 20 bimanual manipulation tasks and thousands of target objects, designed to match various levels of human motor skills based on cognitive science research. We developed Bi-DexHands in Issac Gym, enabling highly efficient RL training at over 30,000 frames per second using a single NVIDIA RTX 3090. Based on Bi-DexHands, we present a comprehensive evaluation of popular RL algorithms in different settings, including single-agent/multi-agent RL, offline RL, multi-task RL, and meta RL. Our findings show that on-policy algorithms, such as PPO, can master simple manipulation tasks that correspond to those of 48-month-old babies, such as catching a flying object or opening a bottle. Furthermore, multi-agent RL can improve the ability to perform manipulations that require skilled bimanual cooperation, such as lifting a pot or stacking blocks. Despite achieving success in individual tasks, current RL algorithms struggle to learn multiple manipulation skills in most multi-task and few-shot learning scenarios. This highlights the need for further research and development within the RL community.

[Gynostemma pentaphyllum saponins alleviate non-alcoholic fatty liver disease in rats by regulating intestinal flora and short-chain fatty acid metabolism].

This study aimed to explore the effects of Gynostemma pentaphyllum saponins(GPs) on non-alcoholic fatty liver disease(NAFLD) induced by high-fat diet in rats and reveal the underlying mechanism. The NAFLD model rats were prepared with high-fat diet. Forty male Sprague Dawley(SD) rats were randomly assigned into the control group, model group, and low-, moderate-, and high-dose GPs(50, 100, and 150 mg·kg~(-1), respectively) groups. After intragastric administration for 8 continuous weeks, we determined the body weight, liver weight, the levels of total cholesterol(TC), triglyceride(TG), low-density lipoprotein cholesterol(LDL-c), high-density lipoprotein cholesterol(HDL-c), alanine aminotransferase(ALT), and aspartate aminotransferase(AST) in serum, and the levels of TC, TG, malondialdehyde(MDA), superoxide dismutase(SOD), catalase(CAT), and interleukin 6(IL-6) in the liver. Furthermore, we observed the pathological changes of liver tissue by oil red O staining and hematoxylin-eosin(HE) staining, sequenced the 16 S rRNA of the intestinal flora in rat feces, and determined the content of short-chain fatty acids in rat feces. The results showed that GPs inhibited the excessive weight gain of high-fat diet-induced NAFLD in rats, reduced the liver weight, lowered the TC, TG, LDL-c, AST, and ALT levels in serum(P<0.05), and rose the HDL-c level in serum(P<0.01). GPs relieved the liver damage caused by high-fat diet, mainly manifested by the lowered levels of TC, TG, MDA, and IL-6 in the liver(P<0.01) and elevated levels of CAT and SOD in the liver. Furthermore, GPs reversed the intestinal flora disorder caused by high-fat diet, restored the diversity of intestinal flora, increased the relative abundance of Bacteroides, and reduced the relative abundance of Firmicutes and the ratio of Firmicutes to Bacteroides. Moreover, GPs promoted the proliferation of beneficial bacteria such as Akkermansia, Bacteroides, and Parabacteroides, and inhibited the growth of harmful bacteria such as Desulfovibrio, Escherichia-Shigella, and Helicobacter. GPs increased the content of short-chain fatty acids(acetic acid, propionic acid, and butyric acid)(P<0.01). These findings indicate that GPs can alleviate the high-fat diet-induced NAFLD in rats via regulating the intestinal flora and short-chain fatty acid metabolism.

AD-VAT+: An Asymmetric Dueling Mechanism for Learning and Understanding Visual Active Tracking.

Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations. To learn a robust tracker for VAT, in this article, we propose a novel adversarial reinforcement learning (RL) method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT. In the mechanism, the tracker and target, viewed as two learnable agents, are opponents and can mutually enhance each other during the dueling/competition: i.e., the tracker intends to lockup the target, while the target tries to escape from the tracker. The dueling is asymmetric in that the target is additionally fed with the tracker's observation and action, and learns to predict the tracker's reward as an auxiliary task. Such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker. To improve the performance of the tracker in the case of challenging scenarios such as obstacles, we employ more advanced environment augmentation technique and two-stage training strategies, termed as AD-VAT+. For a better understanding of the asymmetric dueling mechanism, we also analyze the target's behaviors as the training proceeds and visualize the latent space of the tracker. The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios. The potential of the active tracker is also shown in real-world videos.

End-to-End Active Object Tracking and Its Real-World Deployment via Reinforcement Learning.

We study active object tracking, where a tracker takes visual observations (i.e., frame sequences) as input and produces the corresponding camera control signals as output (e.g., move forward, turn left, etc.). Conventional methods tackle tracking and camera control tasks separately, and the resulting system is difficult to tune jointly. These methods also require significant human efforts for image labeling and expensive trial-and-error system tuning in the real world. To address these issues, we propose, in this paper, an end-to-end solution via deep reinforcement learning. A ConvNet-LSTM function approximator is adopted for the direct frame-to-action prediction. We further propose an environment augmentation technique and a customized reward function, which are crucial for successful training. The tracker trained in simulators (ViZDoom and Unreal Engine) demonstrates good generalization behaviors in the case of unseen object moving paths, unseen object appearances, unseen backgrounds, and distracting objects. The system is robust and can restore tracking after occasional lost of the target being tracked. We also find that the tracking ability, obtained solely from simulators, can potentially transfer to real-world scenarios. We demonstrate successful examples of such transfer, via experiments over the VOT dataset and the deployment of a real-world robot using the proposed active tracker trained in simulation.