FLAT: Fusing layer representations for more efficient transfer learning in NLP.
Neural Netw
; 179: 106631, 2024 Aug 13.
Article
em En
| MEDLINE
| ID: mdl-39159536
ABSTRACT
Parameter efficient transfer learning (PETL) methods provide an efficient alternative for fine-tuning. However, typical PETL methods inject the same structures to all Pre-trained Language Model (PLM) layers and only use the final hidden states for downstream tasks, regardless of the knowledge diversity across PLM layers. Additionally, the backpropagation path of existing PETL methods still passes through the frozen PLM during training, which is computational and memory inefficient. In this paper, we propose FLAT, a generic PETL method that explicitly and individually combines knowledge across all PLM layers based on the tokens to perform a better transferring. FLAT considers the backbone PLM as a feature extractor and combines the features in a side-network, hence the backpropagation does not involve the PLM, which results in much less memory requirement than previous methods. The results on the GLUE benchmark show that FLAT outperforms other tuning techniques in the low-resource scenarios and achieves on-par performance in the high-resource scenarios with only 0.53% trainable parameters per task and 3.2× less GPU memory usagewith BERTbase. Besides, further ablation study is conducted to reveal that the proposed fusion layer effectively combines knowledge from PLM and helps the classifier to exploit the PLM knowledge to downstream tasks. We will release our code for better reproducibility.
Texto completo:
1
Coleções:
01-internacional
Base de dados:
MEDLINE
Idioma:
En
Revista:
Neural Netw
Assunto da revista:
NEUROLOGIA
Ano de publicação:
2024
Tipo de documento:
Article