A Predictive Model of COVID-19 mRNA Vaccine Reactivity based on Dual Attention for Federated Learning Scenarios

ABSTRACT

The base reactivity of the mRNA sequence has a significant impact on the effectiveness of the mRNA vaccine in fighting against the pandemic of COVID-19. The annotation of mRNA sequence reactivity value is a time-consuming and labor-intensive work, which belongs to the private digital assets of each medical institution. It is not practical to train a predictive model by pooling private data from various parties. Fortunately, federated learning techniques can serve to collaboratively train a predictive model among medical institutions while preserving respective digital assets. However, due to the scarcity of data from each participant, conventional sequential prediction mod-els often fail to perform well. To overcome such a challenge, we propose a reactivity value prediction model based on both the self-attention and the convolutional attention mechanisms only requiring a small dataset of labeled samples. Inspired by BERT, we first train a self-attention feature extraction model through self-supervision using both labeled and unlabeled mRNA samples. In this way, the information of mRNA in the semantic space is deeply mined. Then, a convolutional attention block follows the self-attention block, to extract the attention matrix from the base-pair probability matrix and adjacency matrix. By doing so, the attention matrix can compensate for the insensitivity of the self-attention mechanism to the spatial information of mRNA. By using the Open Vaccine RNA database, experiments show that our prediction model for unseen mRNA has a better performance than other state-of-the-art deep learning models that are used to process gene sequences. Further ablation experiments demonstrate that the existence of the dual attention mechanism reduces the risk of overfitting, resulting in an excellent generalization capability of our model. © 2022 IEEE.