Clinical evaluation of artificial intelligence system based on fundus photograph in diabetic retinopathy screening / 中华实验眼科杂志

ABSTRACT

Objective To study the efficiency and accuracy of artificial intelligence (AI) system based on fundus photograph in diabetic retinopathy(DR)screening,and evaluate the clinical application value of AI system. Methods A diagnostic trial was adopted. Total of 13683 color fundus photos were collected in Zhaoqing Gaoyao People's Hospital from March,2017 to November,2018. The AI system for DR (ZOC-DR-V1) was established,based on transfer learning + NASNet algorithm,by training 4465 precisely labeled fundus images (2510 normal,and 1955 with any stage of DR). One thousand confirmed fundus images (300 normal and 700 with any stage of DR),diagnosed by AI ( AI group ) and doctors ( 3 ophthalmologist doctors and 3 endocrinologist doctors ) ( doctor group ) , respectively. Ophthalmologist group and endocrinologist group were both composed of primary,intermediate and senior physicians. The mean reading time of each image and the total time of 1000 images were recorded. The accuracy and efficiency of AI system and doctor groups were compared. The reading process was divided into two stages. The diagnostic coincidence rate and the average reading time of each group between the two parts were calculated and compared. This study protocol was approved by Ethic Committee of Zhongshan Ophthalmic Center, Sun Yat-sen University (No. 2017KYPJ104). Results After training,the diagnostic coincidence rate of AI system (ZOC-DR-V1) in test set was 94. 7％,AUC was 0. 994. In this "man-machine to war",the diagnostic coincidence rate of primary,intermediate and senior endocrinologist was 94. 0％,91. 4％ and 93. 4％;the diagnostic coincidence rate of primary,intermediate and senior ophthalmologist was 92. 7％,94. 4％ and 95. 6％;the diagnostic coincidence rate of AI system was 95. 2％. There was no difference in the diagnostic coincidence rate between AI system and senior ophthalmologist ( P = 0. 749 ) . The mean reading time of each image of primary, intermediate and senior endocrinologists was (4. 63±1. 87),(3. 74±3. 47) and (5. 71±3. 47) seconds,and the total time of 1000 images of primary,intermediate and senior endocrinologists was 1. 29,1. 04 and 1. 58 hours;the mean reading time of each image of primary,intermediate and senior ophthalmologists was ( 7. 25 ± 6. 58 ) , ( 5. 18 ± 5. 01 ) and ( 5. 18 ± 3. 47 ) seconds,and the total time of 1000 images of primary,intermediate and senior endocrinologists was 2. 02,1. 44 and 1. 44 hours;the mean and total time of AI system was (1. 62±0. 67) seconds and 0. 45 hours,and the reading time of AI system was significantly shorter than that of the doctor groups (all at P=0. 000). The diagnostic coincidence rates between previous and posterior part of primary endocrinologist, primary and intermediate ophthalmologist were significantly different (χ2=11. 986,6. 517,10. 896;all at P<0. 05),and the mean reading time in the posterior part was significantly shorter than that in the previous part of intermediate and senior endocrinologist and primary ophthalmologist (t=4. 175,8. 189,5. 160;all at P<0. 01). While the reading time of AI system remained stable throughout the process(χ2=3. 151,P=0. 103;t=0. 038,P=0. 970). Conclusions The ophthalmic AI system based on fundus images has a good diagnostic efficiency,and its diagnostic coincidence rate can compare with senior ophthalmologist,providing a new method and platform for large-scale DR screening.