Gated Memory Unit: A Novel Recurrent Neural Network Architecture for Sequential Analysis
Lecture Notes on Data Engineering and Communications Technologies
; 165:316-328, 2023.
Article
in English
| Scopus | ID: covidwho-2298258
ABSTRACT
The predominant models used to analyze sequential data today are recurrent neural networks, specifically Long-Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, which utilize a temporal value known as the hidden state. These recurrent neural networks process sequential data by storing and modifying a hidden state through the use of mathematical functions known as gates. However, these networks hold many flaws such as limited temporal vision, insufficient memory capacity, and ineffective training times. In response, we propose a simple architecture, the Gated Memory Unit, which utilizes a new element, the hidden stack, a data stack implementation of the hidden state, as well as novel gates. This, along with a parameterized bounded activation function (PBA), allows the Gated Memory Unit (GMU) to outperform existing recurrent models effectively and efficiently. Trials on three datasets were used to display the new architecture's superior performance and reduced training time as well as the utility of the novel hidden stack compared to existing recurrent networks. On data which measures the daily death rate of SARS-Cov-2, the GMU was able to reduce losses to half that of comparable models and did so in nearly half the training time. Additionally, through the use of a generated spiking dataset, the GMU depicted its ability to use its hidden stack to store information past directly observable time steps. We prove that the Gated Memory Unit performs well on a variety of tasks and can outperform existing recurrent architectures. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
Gated Memory Unit; Gated Recurrent Unit; Long-Short Term Memory; Recurrent neural network; Sequential Analysis; Brain; Memory architecture; Network architecture; SARS; Hidden state; Memory units; Network process; Recurrent neural network architectures; Sequential data; Temporal values; Training time; Long short-term memory
Full text:
Available
Collection:
Databases of international organizations
Database:
Scopus
Language:
English
Journal:
Lecture Notes on Data Engineering and Communications Technologies
Year:
2023
Document Type:
Article
Similar
MEDLINE
...
LILACS
LIS