ABSTRACT
Objective:To apply the multi-modal deep learning model to automatically classify the ultra-widefield fluorescein angiography (UWFA) images of diabetic retinopathy (DR).Methods:A retrospective study. From 2015 to 2020, 798 images of 297 DR patients with 399 eyes who were admitted to Eye Center of Renmin Hospital of Wuhan University and were examined by UWFA were used as the training set and test set of the model. Among them, 119, 171, and 109 eyes had no retinopathy, non-proliferative DR (NPDR), and proliferative DR (PDR), respectively. Localization and assessment of fluorescein leakage and non-perfusion regions in early and late orthotopic images of UWFA in DR-affected eyes by jointly optimizing CycleGAN and a convolutional neural network (CNN) classifier, an image-level supervised deep learning model. The abnormal images with lesions were converted into normal images with lesions removed using the improved CycleGAN, and the difference images containing the lesion areas were obtained; the difference images were classified by the CNN classifier to obtain the prediction results. A five-fold cross-test was used to evaluate the classification accuracy of the model. Quantitative analysis of the marker area displayed by the differential images was performed to observe the correlation between the ischemia index and leakage index and the severity of DR.Results:The generated fake normal image basically removed all the lesion areas while retaining the normal vascular structure; the difference images intuitively revealed the distribution of biomarkers; the heat icon showed the leakage area, and the location was basically the same as the lesion area in the original image. The results of the five-fold cross-check showed that the average classification accuracy of the model was 0.983. Further quantitative analysis of the marker area showed that the ischemia index and leakage index were significantly positively correlated with the severity of DR ( β=6.088, 10.850; P<0.001). Conclusion:The constructed multimodal joint optimization model can accurately classify NPDR and PDR and precisely locate potential biomarkers.