Model-Based Reinforcement Learning with Automated Planning for Network Management.

Reinforcement Learning (RL) comes with the promise of automating network management. However, due to its trial-and-error learning approach, model-based RL (MBRL) is not applicable in some network management scenarios. This paper explores the potential of using Automated Planning (AP) to achieve this MBRL in the functional areas of network management. In addition, a comparison of several integration strategies of AP and RL is depicted. We also describe an architecture that realizes a cognitive management control loop by combining AP and RL. Our experiments evaluate on a simulated environment evidence that the combination proposed improves model-free RL but demonstrates lower performance than Deep RL regarding the reward and convergence time metrics. Nonetheless, AP-based MBRL is useful when the prediction model needs to be understood and when the high computational complexity of Deep RL can not be used.

A Semantic and Knowledge-Based Approach for Handover Management.

Handover Management (HM) is pivotal for providing service continuity, enormous reliability and extreme-low latency, and meeting sky-high data rates, in wireless communications. Current HM approaches based on a single criterion may lead to unnecessary and frequent handovers due to a partial network view that is constrained to information about link quality. In turn, HM approaches based on multicriteria may present a failure of handovers and wrong network selection, decreasing the throughput and increasing the packet loss in the network. This paper proposes SIM-Know, an approach for improving HM. SIM-Know improves HM by including a Semantic Information Model (SIM) that enables context-aware and multicriteria handover decisions. SIM-Know also introduces a SIM-based distributed Knowledge Base Profile (KBP) that provides local and global intelligence to make contextual and proactive handover decisions. We evaluated SIM-Know in an emulated wireless network. When the end-user device moves at low and moderate speeds, the results show that our approach outperforms the Signal Strong First (SSF, single criterion approach) and behaves similarly to the Analytic Hierarchy Process combined with the Technique for Order Preferences by Similarity to the Ideal Solution (AHP-TOPSIS, multicriteria approach) regarding the number of handovers and the number of throughput drops. SSF outperforms SIM-Know and AHP-TOPSIS regarding the handover latency metric because SSF runs a straightforward process for making handover decisions. At high speeds, SIM-Know outperforms SSF and AHP-TOPSIS regarding the number of handovers and the number of throughput drops and, further, improves the throughput, delay, jitter, and packet loss in the network. Considering the obtained results, we conclude that SIM-Know is a practical and attractive solution for cognitive HM.

NORA: An Approach for Transforming Network Management Policies into Automated Planning Problems.

Realizing autonomic management control loops is pivotal for achieving self-driving networks. Some studies have recently evidence the feasibility of using Automated Planning (AP) to carry out these loops. However, in practice, the use of AP is complicated since network administrators, who are non-experts in Artificial Intelligence, need to define network management policies as AP-goals and combine them with the network status and network management tasks to obtain AP-problems. AP planners use these problems to build up autonomic solutions formed by primitive tasks that modify the initial network state to achieve management goals. Although recent approaches have investigated transforming network management policies expressed in specific languages into low-level configuration rules, transforming these policies expressed in natural language into AP-goals and, subsequently, build up AP-based autonomic management loops remains unexplored. This paper introduces a novel approach, called NORA, to automatically generate AP-problems by translating Goal Policies expressed in natural language into AP-goals and combining them with both the network status and the network management tasks. NORA uses Natural Language Processing as the translation technique and templates as the combination technique to avoid network administrators to learn policy languages or AP-notations. We used a dataset containing Goal Policies to evaluate the NORA's prototype. The results show that NORA achieves high precision and spends a short-time on generating AP-problems, which evinces NORA aids to overcome barriers to using AP in autonomic network management scenarios.