Q-ADER: An Effective Q-Learning for Recommendation With Diminishing Action Space.

Deep reinforcement learning (RL) has been widely applied to personalized recommender systems (PRSs) as they can capture user preferences progressively. Among RL-based techniques, deep Q-network (DQN) stands out as the most popular choice due to its simple update strategy and superior performance. Typically, many recommendation scenarios are accompanied by the diminishing action space setting, where the available action space will gradually decrease to avoid recommending duplicate items. However, existing DQN-based recommender systems inherently grapple with a discrepancy between the fixed full action space inherent in the Q-network and the diminishing available action space during recommendation. This article elucidates how this discrepancy induces an issue termed action diminishing error in the vanilla temporal difference (TD) operator. Due to this discrepancy, standard DQN methods prove impractical for learning accurate value estimates, rendering them ineffective in the context of diminishing action space. To mitigate this issue, we propose the Q-learning-based action diminishing error reduction (Q-ADER) algorithm to modify the value estimate error at each step. In practice, Q-ADER augments the standard TD learning with an error reduction term which is straightforward to implement on top of the existing DQN algorithms. Experiments are conducted on four real-world datasets to verify the effectiveness of our proposed algorithm.

ARHGAP4 Inhibits Proliferation and Growth of SW620 Colon Cancer Cells by Cell Cycle and Differentiation Pathways.

The aim of this study is to explore the mechanism by which ARHGAP4 regulates the proliferation and growth of colon cancer cells, and it relates to the metastasis of colorectal cancer (CRC). Various techniques including western blot, CCK8, qRT-PCR, RNA seq assay, plate cloning, subcutaneous tumorigenesis assays, and bioinformatics tools were employed to identify genes that were upregulated or downregulated upon ARHGAP4 knockdown and their involvement in tumor cell proliferation and growth. The expression of ARHGAP4 in T and M stages of CRC uses immunohistochemistry. The expression levels of ARHGAP4 were found to be high in SW620, SW480, and HCT116 cell lines, while they were being low in HT29, LoVo, and NCM460 cell lines. Depletion of ARHGAP4 resulted in inhibited proliferation and growth in SW620 cells and inhibited subcutaneous tumorigenesis in nude mice, whereas overexpression of ARHGAP4 promoted proliferation and growth in HT29 cells and promoted subcutaneous tumorigenesis in nude mice. A total of 318 upregulated genes and 637 downregulated genes were identified in SW620 cells upon ARHGAP4 knockdown. The downregulated genes were primarily associated with cell cycle pathways, while the upregulated genes were enriched in differentiation-related pathways. Notable upregulated genes involved in cell differentiation included KRT10, KRT13, KRT16, IVL, and CD24, while significant downregulation was observed in genes related to the cell cycle such as CCNA2, CDKN2C, CDKN3, CENPA, and CENPF. ARHGAP4 expression is markedly elevated in the M1 stage of CRC compared to the M0 stage, suggesting ARHGAP4 linked to the metastatic in CRC. ARHGAP4 regulates the proliferation and growth of colon cancer cells by up- and downregulated cell cycle and differentiation-related molecules, which may be related to the metastasis of CRC.

Urban vs. rural: colorectal cancer survival and prognostic disparities from 2000 to 2019.

This study aimed to analyze the differences in colorectal cancer (CRC) survival between urban and rural areas over the past 20 years, as well as investigate potential prognostic factors for CRC survival in both populations. Using registry data from Surveillance, Epidemiology, and End Results (SEER) from 2000 to 2019, 463,827 CRC cases were identified, with 85.8% in urban and 14.2% in rural areas. The mortality of CRC surpassed its survival rate by the sixth year after diagnosis in urban areas and the fifth year in rural areas. Furthermore, the 5-year overall survival (OS) of CRC increased by 2.9-4.3 percentage points in urban and 0.6-1.5 percentage points in rural areas over the past two decades. Multivariable Cox regression models identified independent prognostic factors for OS and disease-specific survival (DSS) of CRC in urban and rural areas, including age over 40, Black ethnicity, and tumor size greater than 5 cm. In addition, household income below $75,000 was found to be an independent prognostic factor for OS and DSS of CRC in urban areas, while income below $55,000 was a significant factor for rural areas. In conclusion, this study found a notable difference in CRC survival between rural and urban areas. Independent prognostic factors shared among both rural and urban areas include age, tumor size, and race, while household income seem to be area-specific predictive variables. Collaboration between healthcare providers, patients, and communities to improve awareness and early detection of CRC may help to further advance survival rates.

Salivary and fecal microbiota: potential new biomarkers for early screening of colorectal polyps.

Objective: Gut microbiota plays an important role in colorectal cancer (CRC) pathogenesis through microbes and their metabolites, while oral pathogens are the major components of CRC-associated microbes. Multiple studies have identified gut and fecal microbiome-derived biomarkers for precursors lesions of CRC detection. However, few studies have used salivary samples to predict colorectal polyps. Therefore, in order to find new noninvasive colorectal polyp biomarkers, we searched into the differences in fecal and salivary microbiota between patients with colorectal polyps and healthy controls. Methods: In this case-control study, we collected salivary and fecal samples from 33 patients with colorectal polyps (CP) and 22 healthy controls (HC) between May 2021 and November 2022. All samples were sequenced using full-length 16S rRNA sequencing and compared with the Nucleotide Sequence Database. The salivary and fecal microbiota signature of colorectal polyps was established by alpha and beta diversity, Linear discriminant analysis Effect Size (LEfSe) and random forest model analysis. In addition, the possibility of microbiota in identifying colorectal polyps was assessed by Receiver Operating Characteristic Curve (ROC). Results: In comparison to the HC group, the CP group's microbial diversity increased in saliva and decreased in feces (p < 0.05), but there was no significantly difference in microbiota richness (p > 0.05). The principal coordinate analysis revealed significant differences in ß-diversity of salivary and fecal microbiota between the CP and HC groups. Moreover, LEfSe analysis at the species level identified Porphyromonas gingivalis, Fusobacterium nucleatum, Leptotrichia wadei, Prevotella intermedia, and Megasphaera micronuciformis as the major contributors to the salivary microbiota, and Ruminococcus gnavus, Bacteroides ovatus, Parabacteroides distasonis, Citrobacter freundii, and Clostridium symbiosum to the fecal microbiota of patients with polyps. Salivary and fecal bacterial biomarkers showed Area Under ROC Curve of 0.8167 and 0.8051, respectively, which determined the potential of diagnostic markers in distinguishing patients with colorectal polyps from controls, and it increased to 0.8217 when salivary and fecal biomarkers were combined. Conclusion: The composition and diversity of the salivary and fecal microbiota were significantly different in colorectal polyp patients compared to healthy controls, with an increased abundance of harmful bacteria and a decreased abundance of beneficial bacteria. A promising non-invasive tool for the detection of colorectal polyps can be provided by potential biomarkers based on the microbiota of the saliva and feces.

Acyloxyacyl hydrolase promotes pulmonary defense by preventing alveolar macrophage tolerance.

Although alveolar macrophages (AMs) play important roles in preventing and eliminating pulmonary infections, little is known about their regulation in healthy animals. Since exposure to LPS often renders cells hyporesponsive to subsequent LPS exposures ("tolerant"), we tested the hypothesis that LPS produced in the intestine reaches the lungs and stimulates AMs, rendering them tolerant. We found that resting AMs were more likely to be tolerant in mice lacking acyloxyacyl hydrolase (AOAH), the host lipase that degrades and inactivates LPS; isolated Aoah-/- AMs were less responsive to LPS stimulation and less phagocytic than were Aoah+/+ AMs. Upon innate stimulation in the airways, Aoah-/- mice had reduced epithelium- and macrophage-derived chemokine/cytokine production. Aoah-/- mice also developed greater and more prolonged loss of body weight and higher bacterial burdens after pulmonary challenge with Pseudomonas aeruginosa than did wildtype mice. We also found that bloodborne or intrarectally-administered LPS desensitized ("tolerized") AMs while antimicrobial drug treatment that reduced intestinal commensal Gram-negative bacterial abundance largely restored the innate responsiveness of Aoah-/- AMs. Confirming the role of LPS stimulation, the absence of TLR4 prevented Aoah-/- AM tolerance. We conclude that commensal LPSs may stimulate and desensitize (tolerize) alveolar macrophages in a TLR4-dependent manner and compromise pulmonary immunity. By inactivating LPS in the intestine, AOAH promotes antibacterial host defenses in the lung.

A Maximum Divergence Approach to Optimal Policy in Deep Reinforcement Learning.

Model-free reinforcement learning algorithms based on entropy regularized have achieved good performance in control tasks. Those algorithms consider using the entropy-regularized term for the policy to learn a stochastic policy. This work provides a new perspective that aims to explicitly learn a representation of intrinsic information in state transition to obtain a multimodal stochastic policy, for dealing with the tradeoff between exploration and exploitation. We study a class of Markov decision processes (MDPs) with divergence maximization, called divergence MDPs. The goal of the divergence MDPs is to find an optimal stochastic policy that maximizes the sum of both the expected discounted total rewards and a divergence term, where the divergence function learns the implicit information of state transition. Thus, it can provide better-off stochastic policies to improve both in robustness and performance in a high-dimension continuous setting. Under this framework, the optimality equations can be obtained, and then a divergence actor-critic algorithm is developed based on the divergence policy iteration method to address large-scale continuous problems. The experimental results, compared to other methods, show that our approach achieved better performance and robustness in the complex environment particularly. The code of DivAC can be found in https://github.com/yzyvl/DivAC.

Improving Exploration in Actor-Critic With Weakly Pessimistic Value Estimation and Optimistic Policy Optimization.

Deep off-policy actor-critic algorithms have been successfully applied to challenging tasks in continuous control. However, these methods typically suffer from the poor sample efficiency problem, limiting their widespread adoption in real-world domains. To mitigate this issue, we propose a novel actor-critic algorithm with weakly pessimistic value estimation and optimistic policy optimization (WPVOP) for continuous control. WPVOP integrates two key ingredients: 1) a weakly pessimistic value estimation, which compensates the pessimism of lower confidence bound in conventional value function (i.e., clipped double Q -learning) to trigger exploration in low-value state-action regions and 2) an optimistic policy optimization algorithm by sampling actions that could benefit the policy learning most toward optimal Q -values for efficient exploration. We theoretically analyze that the proposed weakly pessimistic value estimation method is lower and upper bounded, and empirically show that it could avoid extremely over-optimistic value estimates. We show that these two ideas are largely complementary, and can be fruitfully integrated to improve performance and promote sample efficiency of exploration. We evaluate WPVOP on the suite of continuous control tasks from MuJoCo, achieving state-of-the-art sample efficiency and performance.

A Distributional Perspective on Multiagent Cooperation With Deep Reinforcement Learning.

Among various value decomposition-based multiagent reinforcement learning (MARL) algorithms, the overall performance of the multiagent system is represented by a scalar global Q value and optimized by minimizing the temporal difference (TD) error with respect to that global Q value. However, the global Q value cannot accurately model the distributed dynamics of the multiagent system, since it is only a simplified representation for different individual Q values of agents. To explicitly consider the correlations between different cooperative agents, in this article, we propose a distributional framework and construct a practical model called distributional multiagent cooperation (DMAC) from a novel distributional perspective. Specifically, in DMAC, we view the individual Q value for the executed action of a random agent as a value distribution, whose expectation can further represent the overall performance. Then, we employ distributional RL to minimize the difference between the estimated distribution and its target for the optimization. The advantage of DMAC is that the distributed dynamics of agents can be explicitly modeled, and this results in better performance. To verify the effectiveness of DMAC, we conduct extensive experiments under nine different scenarios of the StarCraft Multiagent Challenge (SMAC). Experimental results show that the DMAC can significantly outperform the baselines with respect to the average median test win rate.

Analysis of ARHGAP4 Expression With Colorectal Cancer Clinical Characteristics and Prognosis.

Background: This study aims to analyze the correlation between ARHGAP4 in the expression and clinical characteristics of colorectal cancer (CRC), and the influence of ARHGAP4 expression on the prognosis of CRC, and to evaluate whether ARHGAP4 is a potential prognostic oncotarget for CRC. Methods: ARHGAP4 was identified using the Gene Expression Omnibus database through weighted gene coexpression network analysis. Using the Gene Expression Profiling Interactive Analysis to perform and analyze the expression and prognosis of ARHGAP4 in CRC. The expression of AGRGAP4 and immune cells was analyzed by the Tumor IMmune Estimation Resource online database. Finally, immunohistochemistry was used to analyze the expression difference and prognosis of ARHGAP4 in CRC and adjacent normal tissues, as well as the relationship between AGRGAP4 expression and clinical features of CRC. Results: We identified ARHGAP4 that is related to the recurrence of CRC from GSE97781 data. ARHGAP4 has not been reported in CRC. The high expression of ARHGAP4 in select colon adenocarcinoma indicates a poor prognosis by database analysis. In our clinical data results, ARHGAP4 is highly expressed in CRC and lowly expressed in normal tissues adjacent to cancer. Compared with the low-expression group, the high-expression group has a significantly poorer prognosis. In colon cancer, the B-cell, macrophage, neutrophil, and dendritic-cell levels are downregulated after ARHGAP4 gene knockout; the levels of CD8+ and CD4+ T cells, neutrophils, and dendritic cells are upregulated after the amplification of the ARHGAP4 gene. In addition, ARHGAP4 expression is related to N,M staging and clinical staging. Conclusion: ARHGAP4 is highly expressed in CRC, and the high expression of ARHGAP4 has a poor prognosis. The expression of ARHGAP4 in CRC is related to the immune cells such as B cells, CD8+ and CD4+ T cells, macrophages, neutrophils, and dendritic cells. ARHGAP4 is correlated with N,M staging and clinical staging in CRC. ARHGAP4 may be a potential biomarker for the prognosis of CRC.

The effects of the oral administration of graphene oxide on the gut microbiota and ultrastructure of the colon of mice.

Background: Graphene oxide (GO) has been widely used in the field of biomedicine and has shown great potential in drug delivery. Oral administration is an important mode of administration, but there are few studies on the effects of oral GO on gastrointestinal tract and gut microbiota. This study sought to explore the effects of oral GO on the gastrointestinal tract and gut microbiota. Methods: In total, 20 C57BL/6 male mice, aged 5 weeks old, were randomly divided into the following 4 groups (n=5): the control group, the GO30 group, the GO60 group, and the GO120 group. The GO sample solution was administered intragastrically at the doses of 30, 60, or 120 mg/kg every 3 days, and the control group was given an equal volume of distilled water. On the 16th day, mouse feces were taken for 16S ribosomal ribonucleic acid (rRNA) sequencing analysis, and the mice were dissected, and the heart, liver, kidney, and colon removed for histological analysis. Additionally, the ultrastructure of the colon was observed by transmission electron microscopy. Results: No obvious damage was observed in the hearts, livers, and kidneys of the mice. However, the intestinal ultrastructure of the mice in the GO group was damaged. The main manifestations were an uneven arrangement and local atrophy of the microvilli, swelling of the mitochondria and endoplasmic reticulum, and the widening of the intercellular spaces. The damage was positively correlated with increasing GO doses. The 16S rRNA sequencing results showed that the structure of the gut microbiota in the GO group was altered, and the contents of Alistipes, Enterobacteriaceae, Eubacterium, and Xanthobacteraceae were decreased. Conclusions: The oral administration of GO had no obvious toxicity effects on the hearts, livers, and kidneys of the mice. However, it did destroy the ultrastructure of the mouse colon and shift the structure of the gut microbiota, decreasing the contents of Alistipes, Enterobacteriaceae, Eubacterium, and Xanthobacteraceae.

Deep Reinforcement Learning Framework for Category-Based Item Recommendation.

Deep reinforcement learning (DRL)-based recommender systems have recently come into the limelight due to their ability to optimize long-term user engagement. A significant challenge in DRL-based recommender systems is the large action space required to represent a variety of items. The large action space weakens the sampling efficiency and thereby, affects the recommendation accuracy. In this article, we propose a DRL-based method called deep hierarchical category-based recommender system (DHCRS) to handle the large action space problem. In DHCRS, categories of items are used to reconstruct the original flat action space into a two-level category-item hierarchy. DHCRS uses two deep Q -networks (DQNs): 1) a high-level DQN for selecting a category and 2) a low-level DQN to choose an item in this category for the recommendation. Hence, the action space of each DQN is significantly reduced. Furthermore, the categorization of items helps capture the users' preferences more effectively. We also propose a bidirectional category selection (BCS) technique, which explicitly considers the category-item relationships. The experiments show that DHCRS can significantly outperform state-of-the-art methods in terms of hit rate and normalized discounted cumulative gain for long-term recommendations.

miR-504 Promoted Gastric Cancer Cell Proliferation and Inhibited Cell Apoptosis by Targeting RBM4.

BACKGROUND: The purpose of this study was to explore the role and underlying mechanism of miR-504 and RBM4 in gastric cancer. METHODS: The qRT-PCR or Western blot was performed to determine the expressions of miR-504 and RBM4 in the gastric cancer tissues and normal tissues. Human SGC-7901 cells were transfected with miR-504 mimic/inhibitor or pcDNA-RBM4. Cell proliferation and cell apoptosis were assessed by colony formation assay and flow cytometry, respectively. Luciferase reporter gene assays were used to investigate interactions between miR-504 and RBM4 in SGC-7901 cells. RESULTS: The relative expression of miR-504 was significantly upregulated in the gastric cancer group (n = 25) than in the paired normal group (n = 25), but the relative RBM4 expression was remarkably downregulated in the gastric tumor group, compared with the normal group. Additionally, miR-504 overexpression increased the viability of gastric cancer cells. Moreover, RBM4 is a functional target of miR-504 in gastric cancer cells. miR-504 was further confirmed to promote SGC-7901 cell proliferation and inhibit cell apoptosis by downregulation RBM4 in vitro. CONCLUSIONS: miR-504 promotes gastric cancer cell proliferation and inhibits cell apoptosis by targeting RBM4, and this provides a potential diagnostic biomarker and treatment for patients with gastric cancer.

A Novel Deep Learning-Based Collaborative Filtering Model for Recommendation System.

The collaborative filtering (CF) based models are capable of grasping the interaction or correlation of users and items under consideration. However, existing CF-based methods can only grasp single type of relation, such as restricted Boltzmann machine which distinctly seize the correlation of user-user or item-item relation. On the other hand, matrix factorization explicitly captures the interaction between them. To overcome these setbacks in CF-based methods, we propose a novel deep learning method which imitates an effective intelligent recommendation by understanding the users and items beforehand. In the initial stage, corresponding low-dimensional vectors of users and items are learned separately, which embeds the semantic information reflecting the user-user and item-item correlation. During the prediction stage, a feed-forward neural networks is employed to simulate the interaction between user and item, where the corresponding pretrained representational vectors are taken as inputs of the neural networks. Several experiments based on two benchmark datasets (MovieLens 1M and MovieLens 10M) are carried out to verify the effectiveness of the proposed method, and the result shows that our model outperforms previous methods that used feed-forward neural networks by a significant margin and performs very comparably with state-of-the-art methods on both datasets.

Association between glucose-regulated protein and neutrophil apoptosis in severe acute pancreatitis.

OBJECTIVE: This study aimed to investigate the role of glucose regulated protein 78 (GRP-78) in the apoptosis of neutrophils in rats with severe acute pancreatitis. METHODS: A total of 54 SD male rats were randomly assigned into 2 groups: sham group (n=24) and pancreatitis group (n=30). Severe acute pancreatitis was induced by retrograde cholangiopancreatography injection of sodium taurocholate. Rats were sacrified at 3 h, 6 h and 12 h after injection. In control group, rats received laparotomy, but the pancreates remained intact. The serum amylase was detected at different time points, and flow cytometry was done to detect the apoptosis of neutrophils. Proteins were extracted from neutrophils and subjected to detection of GRP78 and Mcl-1 expression by Western blot assay. HE staining was performed for pathological scoring of the pancreas. RESULTS: The serum amylase in pancreatitis group increased markedly when compared with control group (P<0.01). In SAP group, the serum amylase increased gradually over time (P<0.01). HE staining showed a lot of inflammatory cells and infiltration of red blood cells and the apoptosis rate of neutrophils reduced gradually (P<0.01). Western blot assay showed the protein expression of GRP-78 and Mcl-1 increased in neutrophils over time. CONCLUSION: In rats with SAP, the apoptosis rate of neutrophils reduced over time, which may be associated to the stress induced expression of GRP78 and subsequent activation of Mcl-1 resulting in suppression of neutrphil apoptosis over time.