Artificial Intelligence Analysis of Diabetic Retinopathy in Retinal Screening Events

doi:10.21203/rs.3.rs-2562995/v1

Download PDF

Research Article

Artificial Intelligence Analysis of Diabetic Retinopathy in Retinal Screening Events

https://doi.org/10.21203/rs.3.rs-2562995/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 17 Oct, 2023

Read the published version in International Ophthalmology →

You are reading this latest preprint version

Purpose: Patients with Type II Diabetes are at increased risk Diabetic Retinopathy (DR). Early detection and treatment of DR is critical for saving vision and preventing blindness. To address the need for increased DR detection and referrals, we evaluated the use of artificial intelligence (AI) for screening DR.

Methods: Patient images were obtained using a 45-degree Canon Non-Mydriatic CR-2 Plus AF retinal camera in the Department of Endocrinology Clinic (Newark, NJ) and in a community screening event (Newark, NJ). Images were initially classified by an on-site grader and uploaded for analysis by EyeArt, a cloud-based AI software developed by Eyenuk (California, USA). The images were also graded by an off-site retina specialist. Using Fleiss kappa analysis, a correlation was investigated between the three grading systems, the AI, onsite grader, and a US board-certified retina specialist, for a diagnosis of DR and referral pattern.

Results: The EyeArt results, onsite grader, and the retina specialist had a 79% overall agreement on the diagnosis of DR. The kappa value for concordance on a diagnosis was 0.69 (95% CI: 0.61-0.77), indicating substantial agreement. Referral patterns by EyeArt, the onsite grader, and the ophthalmologist had an 85% overall agreement. The kappa value for concordance on “whether to refer” was 0.70 (95% CI: 0.60-0.80), indicating substantial agreement.

Conclusions: This retrospective cross-sectional analysis offers insights into use of AI in diabetic screenings and the significant role it will play in automated detection of DR. The EyeArt readings were beneficial with some limitations in a community screening environment.

artificial intelligence

community vision outreach

diabetic retinopathy

Currently, nearly 96 million individuals in the United States are prediabetic (Type II) [1]. In addition, there are 1.4 million new cases of Type II Diabetes Mellitus (T2DM) diagnosed annually [2]. T2DM is associated with many health complications, including heart disease, kidney disorders, depression, lack of mobility, and strokes. Individuals with diabetes are also at high risk for developing diabetic retinopathy (DR) [3]. Hyperglycemia related retinal vascular injury results in retinal microaneurysms, retinal hemorrhages, edema, vitreous hemorrhage, and tractional retinal detachment, all of which can cause vision loss (Fig. 1). Individuals with diabetes are also at higher risk for other vision threatening diseases (VTDs): cataracts are noted in 32.2%, glaucoma in 7.1%, and age-related macular degeneration in 4.3% [4].

Previous surveillance studies have estimated that half of all individuals with T2DM are unaware of their underlying disease [1]. Almost 15% of US individuals aged 40 years or older with diabetes have DR, as compared to 2.5% of the overall population in the same age group. Early detection and patient education about T2DM and the associated DR can play a significant role in the early diagnosis and treatment of DR and potentially saving vision.

Deep Learning Artificial Intelligence (DLAI) is a type of artificial intelligence technology that utilizes multiple layers of analysis to abstract information from images in a data set. EyeArt, produced by EyeNuk (California, USA), is the only DLAI system that is FDA-approved to detect DR using the Canon CR-2 Plus AF digital non-mydriatic retinal camera (Tokyo, Japan). EyeArt uses uploaded retinal images to classify the images as - DR not present, more than mild DR, or vision-threatening DR. Use of this technology to detect DR has been reported to have a sensitivity of 94% and a specificity of 72% [5]. Early detection, intervention, and education in some cases can limit or reverse the progression of DR. Therefore, evaluation of new technologies to improve the diagnosis process is critical. In times such as COVID-19 pandemic, when regular in-person visits may be difficult to achieve, implementation of an AI system that reduces close contact between individuals may help to address gaps in access to eye care.

We retrospectively evaluated the fundus images from community diabetic retinal screenings conducted in 2022: one from endocrine clinic with diabetics and one from community screenings in Newark. A total of 62 patients were identified who all underwent retinal imaging screening using a Canon Non Mydriatic Retinal Camera CR-2 Plus AF with a resolution of 23 mega-pixel (Tokyo, Japan). Optical coherence tomography (OCT) scans of the macula using the Optovue RT-Vue (California, USA) OCT-B (Fig. 2B) were also performed on patients identified from a community screening in Newark, NJ. To evaluate AI in identifying DR, criteria for patient inclusion included a diagnosis or self-reported T2DM, age over 18 years old, and a record of at least two posterior pole retinal images (one macula-centered, one optic nerve-center) per eye. The equipment expense to perform these comprehensive VTD screenings was approximately US $138,000. The cost to cover screening equipment, AI, and van to transport equipment had been obtained through grant funding.

Each patient record from the retinal camera consisted of three images per eye. The first was an anterior segment image, and the second and third were posterior pole (retinal) images: one macula-centered, and one optic nerve-centered (Fig. 2A). EyeArt requires four retinal images (macula-centered and optic nerve-centered for each eye) for single patient analysis. The retinal images for 124 eyes were uploaded to EyeArt, which produces a report that characterizes each eye as “more than mild DR”, “vision-threatening DR”, negative for both more than mild and vision-threatening DR, or “ungradable” (Fig. 3). If EyeArt detects an image of “poor quality” or “insufficient retinal coverage”, it produces the “ungradable” output⁶. The images were also evaluated by an onsite grader during the screenings. The same fundus images were also evaluated by a board-certified retina specialist and marked the eyes for referral for clinical retinal examination. Diagnoses by the onsite grader and retina specialist were either “positive for DR,” “negative for DR,” or “ungradable.” Each patient’s record had 3 reports that were analyzed and compared – from AI by EyeArt, the onsite grader, and the off-site retinal specialist.

Concordance between EyeArt, the onsite grader, and retina specialist was evaluated for both the DR diagnosis and a referral for a clinical retinal examination was made. (Table 1). Diagnosis was considered concordant when the human graders designated “positive for DR” and EyeArt identified any severity level of DR (more than mild or vision-threatening). Classification of “negative for DR” by the human graders was considered the same as “negative for more than mild DR, or, negative for vision-threatening DR” by EyeArt. Images marked ungradable by human graders and EyeArt were also considered to be concordant. Referral for a clinical diabetic retinal exam by human graders was categorized as “referral” or “no referral.” EyeArt’s classification of any severity level of DR or ungradable was considered concordant with “referral” by the human graders. EyeArt classification of “negative for more than mild DR, negative for vision-threatening DR” agreed with the “no referral” by the human graders.

Table 1

Classification of concordance between EyeArt and human graders (retinal specialist and on-site grader).
	Human grader	EyeArt
Diagnosis	“Positive for DR”	“More than mild DR” OR “Vision-threatening DR”
	“Negative for DR”	“Negative for more than mild DR, negative for vision threatening DR”
	“Ungradable”	“Ungradable”
Referrals	“Referral”	“More than mild DR” OR “Vision-threatening DR” OR “Ungradable”
Referrals	“No referral”	“Negative for more than mild DR, negative for vision threatening DR”

The results produced by EyeArt were compared to the reports from the onsite grader at the screening events and a retina specialist. Free marginal multi-rater kappa values and the associated confidence intervals were calculated using free-marginal Fleiss kappa analysis to evaluate the agreement between the AI, onsite grader, and ophthalmologist. Analysis was conducted for diagnosis rates and for referral rates separately. This study was approved by the Rutgers University Institutional Review Board (Pro20140001070) and is in compliance with the World Medical Association’s Declaration of Helsinki. All participants provided verbal informed consent.

Our tele-ophthalmology community screenings over the last 6 months assessed 124 eyes of 62 patients with a mean age of 64 years old. Forty-five percent of patients had confirmed or self-reported T2DM, 32% reported no history of T2DM, and 23% of patients did not provide this information. Twenty-seven (44%) patients were male, and 34 were female, with an average self-reported BMI of 29.7. Sixty-nine percent of the patients were Caucasian (Portuguese), 21% Hispanic, and 10% were African American (Table 2).

Table 2

Demographic characteristics of patients evaluated for diabetic retinopathy.
Average Body Mass Index (BMI)	29.7 (25-29.9: overweight)
Average Male BMI	29.5 (25-29.9: overweight)
Average Female BMI	29.9 (25-29.9: overweight)
Race/Ethnicity	Proportion of patients
Caucasian (Portuguese)	0.69
Hispanic	0.21
African American	0.10

Retinal fundus photographs images were assessed for DR by an onsite grader, retina specialist, and by AI, EyeArt. Fundus photos uploaded to EyeArt yielded positive DR in 24% of the patients. The onsite grader found DR in 27% of patients, and the retina specialist found DR in 24% of patients (Table 3). Eleven percent of the patients had at least one image that was ungradable by EyeArt, compared to 2% by the onsite grader and 15% by the retina specialist (Table 3). Excluding patients with ungradable images for both eyes, AI found DR in at least one eye in 50% of patients with confirmed or self-reported T2DM, none in the patients who denied T2DM, and 25% of patients with unknown status. Similarly, the onsite grader found DR in at least one eye in 43% of patients with confirmed or self-reported T2DM, 13% of the patients who denied T2DM, and 17% of patients with unknown status. The retina specialist found DR in at least one eye in 55% of patients with confirmed or self-reported T2DM and none in the patients who denied T2DM or had unknown status (Table 4).

Table 3

Proportion of patients with positive DR diagnosis by EyeArt, onsite grader, and board-certified ophthalmologist (retina specialist).
	Proportion of Patients with a positive DR diagnosis	Proportion of Patients with an ungradable image
EyeArt	0.24	0.11
Onsite grader	0.27	0.02
Board-certified ophthalmologist	0.24	0.15

Table 4

Proportions of patients with a positive DR diagnosis with either confirmed or self-reported T2DM, denial of T2DM, or unknown T2DM status.
	Proportion of Patients with a positive DR diagnosis with Confirmed or Self-Reported T2DM	Proportion of Patients with a positive DR diagnosis who denied T2DM	Proportion of Patients with a positive DR diagnosis with unknown status
EyeArt	0.5	0	0.25
Onsite grader	0.43	0.13	0.17
Board-certified ophthalmologist	0.55	0	0

Fifteen percent of the eyes evaluated had cataracts graded 2 + or greater. EyeArt ability to grade eyes (marked ungradable) or grade them correctly (as compared to the board-certified ophthalmologist) decreased by 38% in the presence of cataracts 2 + or greater (Table 5).

Table 5

EyeArt grading of eyes with cataracts graded 2 + or higher
	Eyes with a cataract graded 2 + or higher	Eyes without a cataract graded 2 + or higher
Total	18	106
Proportion graded incorrectly or ungradable	0.61	0.23

The EyeArt results, onsite grader, and ophthalmologist produced a 79% overall agreement on diagnosis of the eyes. This included results where the AI and ophthalmologist agreed on a diagnosis (positive or negative for DR), or both determined the images to be ungradable. The kappa value for concordance on a diagnosis was 0.69 (95% CI: 0.61–0.77), indicating substantial agreement (Table 6–7). Similarly, referrals made by EyeArt, the onsite grader, and the ophthalmologist had an 85% overall agreement. Referral consistency defined both positive and ungradable results as referrable, while negative results were not referrable. The kappa value for concordance on whether to refer was 0.70 (95% CI: 0.60–0.80), indicating substantial agreement (Table 6–7).

Table 6

Measure of interrater agreement between EyeArt, the onsite grader, and board-certified ophthalmologist.
	Percent Overall Agreement	Fixed Marginal Kappa Value
Diagnosis	79.30%	0.69 (95% CI: 0.61–0.77)
Referral	84.95%	0.70 (95% CI: 0.60–0.80)

Table 7

Interpretation of kappa values
Kappa value	Interpretation of Kappa Value
≤ 0	No agreement
0.1–0.20	Slight agreement
0.21–0.40	Fair agreement
0.41–0.60	Moderate agreement
0.61–0.80	Substantial agreement
0.81–0.99	Near perfect agreement
1	Perfect agreement

EyeArt AI is an analysis tool intended to improve the reach of DR screenings both within medical clinics and in community screenings. The high concordance of referral rates between AI, the onsite grader, and the retina specialist suggests that EyeArt may be well suited for clinical use for diabetic screenings. In addition, the high agreement of diagnosis suggests that EyeArt was not overly conservative in the categorization of images.

However, EyeArt’s ability to grade images was not consistent across all cases. EyeArt’s ability to produce a result or grade accurately decreased by 38% when the patient had a cataract graded as 2 + or greater. Older individuals, who are also more likely to be afflicted with VTDs, are more likely to be afflicted with cataracts as well. Obtaining four gradable images for the software can be a challenge for screeners. The inability to grade images in patients with cataracts is problematic since DR is more prevalent with longer duration of diabetes and may be inflicted with more severe pathology.

EyeArt had a higher false positive rate when compared to the retina specialist’s assessment. A false positive is preferred than a false negative in a screening setting to err on the side of caution for possible undetected DR. It is worth noting that discrepancies between the AI and human grader were cases that were mild DR or moderate DR. For some cases where the human grader classified the images as “mild DR,” the AI marked them as “more than mild.”

Eyes that were considered ungradable, many of which had cataracts, were referred to the general eye clinic. Utilization of the Dual Align™ feature, a proprietary software of the Canon retinal camera, may allow for an image with higher quality to be extracted from less informative images. In this study, the images that were deemed ungradable were reuploaded to AI using Opacity Suppression™, a proprietary software of the Canon retina camera. The Opacity Suppression software was useful in increasing the readability of ungradable images. However, the need to use Opacity Suppression™ or the Dual Align™ feature presents limitations for the effectiveness of EyeArt in a community screening setting. EyeArt requires four retinal images (macula-centered and optic nerve-centered for each eye) for single patient analysis at a cost of $23 per patient. Reimbursement of the AI use via Medicare varies between $40–65 based on the state. After uploading to the cloud base software, the images cannot be altered or reuploaded. Therefore, if screeners may need to reupload images for ungradable patients after running Opacity Suppression or Dual Align, this decreases the cost-effectiveness of the software for community-based screenings. Community-based screenings are critical in the detection of undiagnosed DR in the community. Given extenuating circumstances, such as the COVID-19 pandemic, where patient access to eye care may be further limited, accurate and cost-effective screenings are critical.

This retrospective study found EyeArt AI to be an effective DR screening tool in our patient pool. The incorporation of AI analysis with gold standard retinal photography can expand the reach of DR awareness and care. In addition to accuracy, the speed of the system (under one minute) and the convenience of cloud upload and report by the EyeArt AI adds to the efficacy of this community based diabetic retinopathy screening system. EyeArt can be used by screeners to refer patients for DR when an ophthalmologist may not be readily available. Therefore, it can be deployed in non-ophthalmology medical settings, such as the endocrine clinic or emergency room, to address specific populations of patients at risk for DR.

Alongside EyeArt’s high level of accuracy, it still has some limitations in older patients with cataracts. Uploading images to AI with accurate results, without the need of an on-site ophthalmologist for screening, could revolutionize the way retinal screenings are conducted, especially in times of a pandemic when patient and/or physician availability and contact may be limited.

(AI) artificial intelligence, (BMI) body mass index, (DLAI) deep learning artificial intelligence, (DR) diabetic retinopathy, (OCT) optical coherence tomography, (T2DM) type II diabetes mellitus, (VTDs) vision threatening diseases

Funding And/or Competing Interests: Author Dr. B Szirth receives grant funding from New Jersey Health Foundation. The New Jersey Health Foundation had no involvement in the study design; data collection, analysis, and interpretation; and in the decision to submit this article for publication. All other authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest, or non-financial interest in the subject matter or materials discussed in this manuscript.

Author Contributions: Study conceptualization was completed by Ms. Rita Vought and Dr. Bernard Szirth. Dr. Bernard Szirth was responsible for funding acquisition. Data collection was performed by Ms. Rita Vought, Ms. Victoria Vought, and Dr. Bernard Szirth. Data analysis was performed by Ms. Victoria Vought and Mr. Megh Shah. The first draft of the manuscript was written by Rita Vought and Victoria Vought, and all authors commented on previous versions of the manuscript. Dr. Bernard Szirth and Dr. Neelakshi Bhagat were responsible for the supervision of the study. All authors read and approved the final manuscript.

Ethics Approval: This study was approved by the Rutgers University Institutional Review Board (Pro20140001070) and is in compliance with the World Medical Association’s Declaration of Helsinki.

Consent to Participate: Informed consent was obtained from all individual participants included in the study.

Consent to Publish: Not applicable.

Acknowledgements: This work was supported by the New Jersey Health Foundation.

Albright AL, Gregg EW (2013) Preventing type 2 diabetes in communities across the US: the National Diabetes Prevention Program. Am J Prev Med 44(4):S346-S351. https://doi.org/10.1016/j.amepre.2012.12.009
Control CfD, Prevention. (2014) National diabetes statistics report: estimates of diabetes and its burden in the United States, 2014. Atlanta, GA: US Department of Health and Human Services.
Shah A. (2016) Prevalence of diabetic retinopathy in the United States, 2011–2014. Value in Health 19(3):A199.
Cha AE, Villarroel MA, Vahratian A. (2019) Eye disorders and vision loss among US adults aged 45 and over with diagnosed diabetes, 2016–2017. NCHS Data Brief 344:1-8. PMID: 31442198
Solanki K, Ramachandra C, Bhat S, Bhaskaranand M, Nittala MG, Sadda SR. (2015) EyeArt: automated, high-throughput, image analysis for diabetic retinopathy screening. Investigative Ophthalmology & Visual Science. 56(7):1429-1429.
Eyenuk.com[Internet]. Eyenuk; c2010-2020 https://www.eyenuk.com/us-en/products/eyeart/. Accessed 2022.

Competing interest reported. Author Dr. B Szirth receives grant funding from New Jersey Health Foundation. The New Jersey Health Foundation was not involved in the study design, data collection, analysis, interpretation, and in the decision to submit this article for publication. All other authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest, or non-financial interest in the subject matter or materials discussed in this manuscript.

Download PDF

Journal Publication

published 17 Oct, 2023

Read the published version in International Ophthalmology →

Editorial decision: Major revision
10 May, 2023
Reviews received at journal
16 Apr, 2023
Reviewers agreed at journal
16 Apr, 2023
Reviewers invited by journal
16 Apr, 2023
Editor assigned by journal
09 Feb, 2023
Submission checks completed at journal
08 Feb, 2023
First submitted to journal
08 Feb, 2023

You are reading this latest preprint version

Artificial Intelligence Analysis of Diabetic Retinopathy in Retinal Screening Events

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1