EDDIE-Transformer: Enriched Disease Embedding Transformer for X-Ray Report Generation

ABSTRACT

Automatic medical report generation is an emerging field that aims to generate medical reports based on medical images. The report writing process can be tedious for senior radiologists and challenging for junior ones. Thus it is of great importance to expedite the process. In this work, we propose an EnricheD DIsease Embedding based Transformer (Eddie-Transformer) model, which jointly performs disease detection and medical report generation. This is done by decoupling the latent visual features into semantic disease embeddings and disease states via our state-aware mechanism. Then, our model entangles the learned diseases and their states, enabling explicit and precise disease representations. Finally, the Transformer model receives the enriched disease representations to generate high-quality medical reports. Our approach shows promising results on the widely-used Open-I benchmark and COVID-19 dataset. © 2022 IEEE.