High confidence Artificial Intelligence (AI) predictions in glaucoma detection: A RIM ONE database study

doi:10.21203/rs.3.rs-4622347/v1

Download PDF

Article

High confidence Artificial Intelligence (AI) predictions in glaucoma detection: A RIM ONE database study

https://doi.org/10.21203/rs.3.rs-4622347/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This study explores the application of deep learning to diagnose glaucoma using retinal fundus images, addressing the increasing global prevalence of this condition. Utilizing the EfficientNetV2B0 neural network model, trained on the RIM One dataset of 485 optic disc images, the study achieved an area under the curve (AUC) of 96%, with sensitivity, specificity, positive predictive value, and negative predictive value of 91%, 99%, 98%, and 95%, respectively.

A novel approach in this research involves the use of a 95% prediction probability threshold to enhance clinical relevance. For images with high predictive confidence, the AUC improved to 100%, with perfect sensitivity and specificity. This method aligns with clinical practices, ensuring further investigation only when high confidence in diagnosis is achieved.

The DeLong t-test indicated statistically significant improvements in AUC, sensitivity, and negative predictive value for high-confidence predictions compared to the broader test set. This study is the first to incorporate prediction probability into AI models for glaucoma diagnosis, suggesting a practical tool for efficient and accurate screening in clinical settings.

Biological sciences/Computational biology and bioinformatics

Health sciences/Health care/Medical imaging

Health sciences/Health care/Public health

Glaucoma is a progressive optic neuropathy which develops without noticeable symptoms, leading to gradual and irreversible deterioration in visual function, eventually resulting in total visual field loss.^1,2

Globally, glaucoma is the second most prevalent cause of blindness, affecting approximately one in every two hundred individuals below the age of 50 years, and one in ten individuals above the age of 80 years. It is projected that by 2040, around 111.8 million people globally between the ages of 40 and 80 years will be afflicted with glaucoma^3,4

The diagnosis of glaucoma is based on intraocular pressure measurement, visual field testing ^5,6, optic disc examination and imaging and, increasingly, optical coherence tomography (OCT)^9,10 to examine features of the optic nerve head. The increasing prevalence of glaucoma will correspond to an increase in healthcare costs required for disease management.

Over the last decade, there has been research focus on approaches rooted in deep learning techniques^11,12, which have demonstrated significant effectiveness in tasks such as image classification and segmentation. These methods have shown some promise in ophthalmology¹³, particularly in enhancing diagnostic capabilities

These AI technologies have the potential to reduce the burden on existing healthcare services by increasing the accuracy and efficiency of diagnosis, to intelligently target healthcare resources.

This study utilized the publicly available RIM One dataset¹⁴ of 485 optic disc images which comprises 313 normal control cases and 172 cases of glaucoma. Of the 485 total images, 248 of glaucoma and normal control images were used to train the EfficientNetV2B0 neural network model¹⁵while 63 were reserved for validation, and 174 for testing purposes.

Data augmentation was performed using the albumentations library, specifically rotation with a limit of 30 degrees. The image shape was set to 224x224 pixels with 3 channels. The last 165 layers corresponding to the model were fine-tuned using the Adam optimizer with a learning rate of 10^-4 and binary cross-entropy loss function. The inverse frequency formula was employed, and accuracy and F1 score metrics were utilized for evaluation.

Measures of model performance; area under the curve (AUC) and confusion matrix were obtained using the 174 test images. Subsequently, in a novel extension of the model, all 174 images were processed by the trained neural network to identify images with a prediction probability exceeding 95%.

Delong's t-test was then used to compare the AUC obtained from the first test with 174 images and the second test with only images with a prediction probability above 95%. Sensitivity and specificity were also compared between the two tests.

The results from the analysis of 174 test images, with 118 labelled as normal and 56 labelled as glaucoma, revealed an AUC of 96% with the newly trained EfficientNetV2B0 model. The sensitivity was 91%, specificity 99%, positive predictive value 98% and negative predictive value was 95%. The confusion matrix illustrating these findings is presented in Fig. 1.

Of the 174 test images assessed, a protocol was established to exclude patients from further examination if the neural network's prediction probability did not meet the 95% threshold for certainty. Subsequently, 152 out of the 174 images surpassed this probability threshold and were included for further analysis as image results with high predictive confidence.

For this subset of 152 images identified by the newly trained model with high predictive confidence, the AUC improved to 100%, with corresponding 100% sensitivity, specificity, positive predictive value, and negative predictive value. The confusion matrix illustrating these findings is presented in Fig. 2

The DeLong t-test revealed a statistically significant difference in the AUC between the two groups, with a p-value of 0.007. Additionally, significant differences were observed in sensitivity (p = 0.0003) and negative predictive value (p = 0.002). However, the differences in specificity (p = 0.18) and positive predictive value (p = 0.055) were not statistically significant.

Several studies have demonstrated the proficiency of neural networks in accurately distinguishing between glaucomatous and healthy using retinal fundus images, typically yielding AUC values of around 99% across a number of publicly available databases^16–19. Deep learning techniques have even proven adept at detecting glaucoma using retinal images excluding the optic nerve, achieving an AUC of 88%²⁰.

Fumero et al. achieved a 99% AUC with the RIM ONE14 database, while Phasuk et al²¹. attained 94%. The 96% AUC obtained in this study is in line with published literature.

To date, no published study according to our knowledge, has considered the use of prediction probability obtained from neural networks and its potential in clinical practice. This study focuses not only on the prediction of glaucomatous versus healthy discs of a trained neural network on the RIM-ONE test data but also on the certainty probability of each prediction. 95% was selected as a confidence level as it is a standard threshold in scientific research. If the probability exceeds 95%, we can consider the prediction of the artificial intelligence as accurate. This approach to the use of AI could be adopted in clinical practice to identify those patients who require further investigation for possible glaucoma. This mirrors current clinical practice of initial disc examination by a clinician and further investigation only if there is clinical suspicion of glaucoma.

While other neural networks may be mathematically superior to this model, this novel extension of the model to calculate certainty probability gives this model clinical relevance previously lacking in other models. In this study, in the high confidence prediction test set the AUC is 100%, whereas with the original test set, which included results with predictive probabilities under 95%, an AUC of 96% was obtained. The comparison with the De Long test yielded statistically significant results (p < 0.05).

In our analysis, we found that not only was the increase of Area Under the Curve (AUC) statistically significant, indicating an improved overall performance of the model, the sensitivity and negative predictive value also showed statistically significant improvements.

Specificity and positive predictive did not improve to a statistically significant degree, despite reaching 100%, because the baseline values for specificity and positive predictive value of the full test of 172 images were already high, at 98% and 95% respectively.

All previous studies have primarily focused on achieving a 100% AUC without considering individual prediction probabilities. This approach of assigning predictive probabilities to each outcome from the AI model takes into consideration the clinical relevance of neural networks in diagnosing glaucoma.

This study is, to our knowledge, the first to demonstrate the potential clinical relevance of incorporating high-confidence AI predictions into artificial intelligence models to assess glaucoma from fundus images.

Our study demonstrates the effectiveness of neural networks in diagnosing glaucoma from retinal fundus images. By calculating high confidence AI predictions with a probability of certainty exceeding 95%, we highlight a potential clinical application. This approach bridges the gap between AI research and clinical practice, offering a promising tool for efficient and accurate screening for glaucoma in community and hospital settings.

Author Contribution

FLY and MC wrote the main article. The other authors reviewed the paper

Data Availability

Dataset is available in this website: https://drive.google.com/file/d/1teYi_smpLiNZNJcTWdxXgKKLW2fkUQr4/view

Meeting presentation: Accepted in European Glaucoma Congress 2024

Financial support: None

Conflict of interest: No conflict of interest exists for any author

Casson RJ, Chidlow G, Wood JP, Crowston JG, Goldberg I. Definition of glaucoma: clinical and experimental concepts. Clin Exp Ophthalmol. 2012 May-Jun;40(4):341-9. doi: 10.1111/j.1442-9071.2012.02773.x. Epub 2012 Apr 5. PMID: 22356435.
Voelker R. What Is Glaucoma? JAMA. 2023;330(16):1594. doi: 10.1001/jama.2023.16311. PMID: 37801324.
Tham YC, Li X, Wong TY, Quigley HA, Aung T, Cheng CY. Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology. 2014;121(11):2081–90. doi: 10.1016/j.ophtha.2014.05.013. Epub 2014 Jun 26. PMID: 24974815.
Tham YC, Li X, Wong TY, Quigley HA, Aung T, Cheng CY. Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology. 2014;121(11):2081–90. doi: 10.1016/j.ophtha.2014.05.013. Epub 2014 Jun 26. PMID: 24974815.
De Moraes CG, Liebmann JM, Levin LA. Detection and measurement of clinically meaningful visual field progression in clinical trials for glaucoma. Prog Retin Eye Res. 2017;56:107–147. doi: 10.1016/j.preteyeres.2016.10.001. Epub 2016 Oct 20. PMID: 27773767; PMCID: PMC5313392.
Nouri-Mahdavi K. Selecting visual field tests and assessing visual field deterioration in glaucoma. Can J Ophthalmol. 2014;49(6):497–505. doi: 10.1016/j.jcjo.2014.10.002. PMID: 25433738.
Maupin E, Baudin F, Arnould L, Seydou A, Binquet C, Bron AM, Creuzot-Garcher CP. Accuracy of the ISNT rule and its variants for differentiating glaucomatous from normal eyes in a population-based study. Br J Ophthalmol. 2020;104(10):1412–1417. doi: 10.1136/bjophthalmol-2019-315554. Epub 2020 Jan 20. PMID: 31959590.
Law SK, Kornmann HL, Nilforushan N, Moghimi S, Caprioli J. Evaluation of the "IS" Rule to Differentiate Glaucomatous Eyes From Normal. J Glaucoma. 2016;25(1):27–32. doi: 10.1097/IJG.0000000000000072. PMID: 24844540.
Moradi Y, Moradkhani A, Pourazizi M, Rezaei L, Azami M. Diagnostic Accuracy of Imaging Devices in Glaucoma: An Updated Meta-Analysis. Med J Islam Repub Iran. 2023;37:38. doi: 10.47176/mjiri.37.38. PMID: 37332389; PMCID: PMC10270645.
Michelessi M, Lucenteforte E, Oddone F, Brazzelli M, Parravano M, Franchi S, Ng SM, Virgili G. Optic nerve head and fibre layer imaging for diagnosing glaucoma. Cochrane Database Syst Rev. 2015;2015(11):CD008803. doi: 10.1002/14651858.CD008803.pub2. PMID: 26618332; PMCID: PMC4732281.
Chan HP, Samala RK, Hadjiiski LM, Zhou C. Deep Learning in Medical Image Analysis. Adv Exp Med Biol. 2020;1213:3–21. doi: 10.1007/978-3-030-33128-3_1. PMID: 32030660; PMCID: PMC7442218.
Chen X, Wang X, Zhang K, Fung KM, Thai TC, Moore K, Mannel RS, Liu H, Zheng B, Qiu Y. Recent advances and clinical applications of deep learning in medical image analysis. Med Image Anal. 2022;79:102444. doi: 10.1016/j.media.2022.102444. Epub 2022 Apr 4. PMID: 35472844; PMCID: PMC9156578.
Orlando JI, Fu H, Barbosa Breda J, van Keer K, Bathula DR, Diaz-Pinto A, Fang R, Heng PA, Kim J, Lee J, Lee J, Li X, Liu P, Lu S, Murugesan B, Naranjo V, Phaye SSR, Shankaranarayana SM, Sikka A, Son J, van den Hengel A, Wang S, Wu J, Wu Z, Xu G, Xu Y, Yin P, Li F, Zhang X, Xu Y, Bogunović H. REFUGE Challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Med Image Anal. 2020;59:101570. doi: 10.1016/j.media.2019.101570. Epub 2019 Oct 8. PMID: 31630011.
F. Fumero, S. Alayon, J. L. Sanchez, J. Sigut and M. Gonzalez-Hernandez, "RIM-ONE: An open retinal image database for optic nerve evaluation," 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), Bristol, UK, 2011, pp. 1–6, doi: 10.1109/CBMS.2011.5999143.
Keras. (n.d.). EfficientNetV2B0. Retrieved from https://keras.io/api/keras_cv/models/backbones/efficientnetv2/
Velpula VK, Sharma LD. Multi-stage glaucoma classification using pre-trained convolutional neural networks and voting-based classifier fusion. Front Physiol. 2023;14:1175881. doi: 10.3389/fphys.2023.1175881. PMID: 37383146; PMCID: PMC10293617.
Ganesh SS, Kannayeram G, Karthick A, Muhibbullah M. A Novel Context Aware Joint Segmentation and Classification Framework for Glaucoma Detection. Comput Math Methods Med. 2021;2021:2921737. doi: 10.1155/2021/2921737. PMID: 34777561; PMCID: PMC8589492.
Rehman AU, Taj IA, Sajid M, Karimov KS. An ensemble framework based on Deep CNNs architecture for glaucoma classification using fundus photography. Math Biosci Eng. 2021;18(5):5321–5346. doi: 10.3934/mbe.2021270. PMID: 34517490.
Hemelings R, Elen B, Schuster AK, Blaschko MB, Barbosa-Breda J, Hujanen P, Junglas A, Nickels S, White A, Pfeiffer N, Mitchell P, De Boever P, Tuulonen A, Stalmans I. A generalizable deep learning regression model for automated glaucoma screening from fundus images. NPJ Digit Med. 2023;6(1):112. doi: 10.1038/s41746-023-00857-0. PMID: 37311940; PMCID: PMC10264390.
Hemelings R, Elen B, Barbosa-Breda J, Blaschko MB, De Boever P, Stalmans I. Deep learning on fundus images detects glaucoma beyond the optic disc. Sci Rep. 2021;11(1):20313. doi: 10.1038/s41598-021-99605-1. Erratum in: Sci Rep. 2023;13(1):21456. PMID: 34645908; PMCID: PMC8514536.
Phasuk S, Tantibundhit C, Poopresert P, Yaemsuk A, Suvannachart P, Itthipanichpong R, Chansangpetch S, Manassakorn A, Tantisevi V, Rojanapongpun P. Automated Glaucoma Screening from Retinal Fundus Image Using Deep Learning. Annu Int Conf IEEE Eng Med Biol Soc. 2019;2019:904–907. doi: 10.1109/EMBC.2019.8857136. PMID: 31946040

No competing interests reported.

Download PDF

Reviewers invited by journal
30 Aug, 2024
Editor assigned by journal
10 Aug, 2024
Editor invited by journal
26 Jul, 2024
Submission checks completed at journal
25 Jul, 2024
First submitted to journal
22 Jun, 2024

You are reading this latest preprint version

High confidence Artificial Intelligence (AI) predictions in glaucoma detection: A RIM ONE database study

Status:

Version 1

Abstract

Figures

Introduction

Material and methods

Results

Discussion

Conclusion

Declarations

Author Contribution

Data Availability

References

Additional Declarations

Status:

Version 1