Fundus Image Analysis of Retinitis Pigmentosa Using Artificial Intelligence

doi:10.21203/rs.3.rs-4851616/v1

Download PDF

Article

Fundus Image Analysis of Retinitis Pigmentosa Using Artificial Intelligence

https://doi.org/10.21203/rs.3.rs-4851616/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Retinitis pigmentosa (RP) is one group of inherited retinal diseases that are caused by genetic defects that lead to progressive photoreceptor loss and eventual blindness. Early diagnosis will helpful for a effective management of the disease, however, many patients remain unaware of eraly symptoms. Meanwhile, fundus images are widely taken for medical checkups, however, are underused in detecting RP. This study explores the potential of deep learning to identify RP from color fundus images. The dataset contained 200 color fundus images of Japanese RP patients and 121 color fundus images from non-RP subjects from Keio University Hospital. Using transfer learning, pretrained convolutional neural network models -VGG16, Resnet50, and InceptionV3- were finetuned to detect RP. As a result, Inception V3 achieved the best accuracy of 96.97%, which matches the average diagnostic accuracy of ophthalmologists. Using Gradient-weighted Class Activation Mapping (Grad-CAM), we identified peripheral pigmentation in the fundus images as a critical feature for diagnosis, aligning with the known progression patterns of RP. This confirms the robustness and validity of our model, highlighting the utility of deep learning in assisting ophthalmologists with RP screening.

Health sciences/Diseases/Eye diseases

Health sciences/Diseases/Eye diseases/Retinal diseases

retinitis pigmentosa

deep learning

fundus images

artificial intelligence

Retinitis pigmentosa (RP) is an intractable and serious eye disease that affects approximately 2 million people worldwide [1]. RP is caused by degeneration of photoreceptors in the retina, resulting in the appearance of abnormal findings on fundus images, mainly pigmentation, in the periphery. As the pigmentation progresses, visual field testing and fundus imaging are performed to monitor the progress of the disease. Although there is no effective treatment for RP at this moment, research and development are actively underway [2, 3]. Thus, it is important to diagnose and predict RP to smoothly guide potential future treatment. Early diagnosis also has the potential to reduce psychological burden by allowing more time to prepare for visual impairment in the future. However, detecting RP in its early stages poses a challenge due to its limited subjective symptoms. While color fundus images offer a cost-effective and noninvasive means of identifying RP, the scarcity of ophthalmologists makes widespread screening impractical in clinical settings. Consequently, leveraging AI for RP screening would be highly beneficial. Fundus images contain vast amounts of data, making it challenging to identify subtle abnormalities during routine examinations. Therefore, artificial intelligence can assist in detecting such anomalies within fundus images promptly and accurately, aiding in early intervention. Given the above, there would be a tremendous benefit if AI could be utilized to assist in the early diagnosis of RP.

With the rapid progress of deep learning study, research on the application of artificial intelligence to the diagnosis of various diseases, including ophthalmology diseases, has been active in recent years [4, 5]. Previous studies have investigated the application of deep learning techniques for the classification of RP using fundus images. Guo C et al. Eexplored various eye diseases, including RP, using deep transfer methods [6]. However, it is noteworthy that this study did not incorporate cross-validation techniques. Similarly, another study by Chen TC et al. reported sensitivity and specificity of 91.2% and 91.71%, respectively, for the early detection of RP using AI. While these findings demonstrate certain levels of accuracy, they also suggest the potential for further improvement in diagnostic performance [7]. In contrast, our study aims to build upon these prior works by not only employing deep learning algorithms for RP classification but also conducting thorough cross-validation procedures to ensure the robustness and generalizability of our model. Through this approach, we aim to achieve enhanced sensitivity and specificity in the early detection of RP. Furthermore, we also executed gradient-weighted class activation mapping (Grad-CAM), [8]which shows important pixels for prediction to confirm whether our program will be able to apply to unknown fundus images.

2.1 Dataset

In this study, a program for predicting RP using color fundus images is designed as shown in Figure 1. This study was approved by the Ethics Review Committee of Keio University School of Medicine (Approval No. 20210809) and conducted in accordance with the Declaration of Helsinki. Written informed consent was waived due to retrospective nature of the study. The patient's personal information was removed from the images, prior to the AI analysis.

We collected RP and non-RP images taken at Keio University Hospital during January 2012 through May 2021. The images were then reviewed by ophthalmologists and poor-quality images such as significant blur and wrong angles of views were removed from the dataset. After that, the images were cropped to the smallest long rectangle that would accommodate the oval shape of the fundus and resized to a (256, 256) size. Since the dataset was small, the data was augmented by randomly flipping and changing contrast. Augmentation was applied only for the training dataset.

Table 1 Patient demographic characteristics.

Patient demographic data were presented in Table 1.

		RP	Non-RP
Total		107	94
	Male	56	47
	Female	51	47
Age mean [range]		52.8 [17-87]	54.4 [23-81]

2.2 Model

In this study, deep learning transfer learning techniques with three convolutional neural networks, the VGG16, Inception V3, and Resnet 50 were used. VGG16 is composed of 13 convolutional layers, 5 max-pooling layers, and 3 fully connected layers [9]. InceptionV3 is 42 layers deep and it is using several approaches to reduce the total number of parameters [10]. Resnet 50 has 48 convolutional layers along with 1 maxpool layer and 1 average pool layer [11]. Those are pre-trained on more than a million images from the ImageNet dataset [12]. Adam optimizer was used for the optimization for all models. The learning rate was 0.000001, the number of training epochs was 200, and loss function was cross entropy. The used fundus images comprise a total of 201 individuals, with 107 individuals diagnosed with RP and 94 individuals classified as non-RP.

2.3 Evaluation

In this study, we adopted a two-stage evaluation process, where we first assessed the performance of our model using cross-validation (CV) and subsequently conducted a final evaluation using a separate test dataset. This approach ensures robustness and reliability in our assessment, as it allows us to validate the generalizability of our model's performance beyond the training data. We conducted a final evaluation on an isolated test dataset to provide a comprehensive and trustworthy assessment of our AI models.We utilized evaluation metrics, including the receiver operating characteristic (ROC) curve, and its area under the curve (AUC), to assess the performance of our model.

We used PyTorch to build a model and Matplotlib for visualization. The training and inference were executed with NVIDIA V100 Tensor Core GPU.

2.4 Visualization

We visualized which parts of the image had the greatest impact on the prediction using Grad-CAM, which is a method for providing an explanation for a given input and its prediction by a CNN-based image recognition model. Ophthalmologists conducted a validity assessment and error analysis of the regions of interest identified based on this map. 2.5

For further examination of the performance and usefulness of the model, a comparison was made with the performance of ophthalmologists and medical students in classification. The model, four ophthalmologists, and one medical student performed classification using the same 66 images as test data.

Figure 1. Program Flow Chart. The dataset containing 321 images was separated into the training and test dataset. We evaluated the models by 5-fold cross-validation on the training dataset and hold-out method using the test dataset.

3.1 Image collection

We collected 411 fundus images and excluded 90 images due to poor quality. As a result, our dataset contained 200 RP fundus images from 107 patients and 121 non-RP fundus images from 94 patients. The collected 321 images consisting of RP and non-RP fundus images were separated into a training set (159 RP and 96 non-RP) and a test set (41 RP and 25 non-RP). The patient demographic characteristics are shown in Table 1.

3.2 5-fold cross validation

In comparing the three CNNs used in training the model, 5-fold cross validation showed that Inception V3 attained the best performance in accuracy (Table 1). For VGG16 the accuracy was 0.749 ± 0.110, for Resnet50 0.883 ± 0.120, and for InceptionV3 0.897 ± 0.078.

After training, we conducted testing on the test dataset. Among these, the results of InceptionV3 yielded the best performance, as demonstrated below. As Tabel 2 shows, the model made wrong precisions only for 2 non-RP images.　For the rest of the images in the dataset (41 RP images and 22 non-RP images), the model made correct prediction (Table 3).

Table 2 Performance metrics of deep learning models

The accuracy and standard deviation (SD) obtained from cross-validation, presented as a percentage. Cross-validation involves dividing the dataset into multiple smaller groups, using each group as a test set to evaluate the model's generalizability and reliability. The AUROC (Area Under the Receiver Operating Characteristic Curve) value obtained on the test dataset, presented as a percentage. The AUROC value measures how well the model can distinguish between different classes, with higher values indicating better performance.

Model	Cross Validation Accuracy (± SD), %	Test Dataset AUROC, %
VGG16	0.749 ± 0.110	95.21
Resnet50	0.883 ± 0.120	97.85
Inception V3	0.897 ± 0.078	99.32

Table 3 Confusion matrix with Inception V3.

This table shows the performance of the Inception V3 model in classifying data into two categories, RP and Non-RP.

		Predicted Label
		RP	Non-RP
True Label	RP	41	0
True Label	No RP	2	23

Figure2　Receiver operating characteristic (ROC) curve of Inception V3 model with plots of the average of ophthalmologists and a student.

ROC curve of Inception V3 model, with performance comparisons to average ophthalmologists and a student, depicted by orange and green triangles respectively.

The horizontal axis shows False Positive Rate (FPR) and the vertical axis shows True Positive Rate (TPR).

Figure 2 shows the performance of the Inception V3 model along with the ophthalmologists and medical student accuracy. Inception V3 scored best in AUROC compared with two other models shown in Table 2.

3.3 Visualization

We conducted a Grad-CAM to visualize which pixels the model sees as important in the prediction process for the model based on Inception V3 to confirm the validity of the model to show that it did not rely on features unrelated to RP.

Figure 3 Grad-CAM analysis of misdiagnosed samples. Representative original fundus color images and corresponding Grad-CAM heatmaps are shown for each diagnostic category. The Grad-CAM heatmap show the regions that contributed significantly to the model's decision: red and yellow indicate high contribution, blue indicates low contribution.

Our analysis revealed that the heatmap predominantly focus on the peripheral regions. However, instances of false positives were observed to be centered around the macular region, indicating potential misfocus. The model correctly identified all RP images. Figure 4 shows two RP images that was misdiagnosed as non-RP by the ophthalmologist but was correctly predicted to be RP by this program. As shown in this example, the program was able to correctly diagnose RP images that would have been missed by an ophthalmologist in a non-RP screening.

Figure 4. RP Images that only AI could correctly diagnose

Representative original fundus color images and corresponding Grad-CAM heat maps are shown for each diagnostic category.

3.4 Performance comparison with Ophthalmologists and a Medical student

The model and ophthalmologists demonstrated comparable performance in terms of accuracy, recall, specificity, and precision. Additionally, the model outperformed medical students in RP detection. The results are shown in Table 4. The performance of the model was s comparable to that of ophthalmologists and higher than that of a medical student.

Table 4 Performance comparison of our AI model, ophthalmologist, and medical student.

The table displays accuracy, recall, specificity, and precision for four ophthalmologists (labeled 1-4), their mean values, a medical student, and a machine learning model.

	Accuracy, %	Recall, %	Specificity, %	Precision, %
Ophthalmologist 1	95.45	92.68	100	100
Ophthalmologist 2	98.48	97.56	100	100
Ophthalmologist 3	96.97	95.12	100	100
Ophthalmologist 4	96.97	95.12	100	100
Ophthalmologists Mean	96.97	95.12	100	100
Medical student	81.82	73.17	96	96.77
Model	96.97	95.35	100	100

4.1 Performance

RP is a retinal degenerative disease that can lead to loss of sight. Early detection is important in terms of reducing the psychological burden of possible future disabilities even if there is no effective treatment for now. The model using AI can be useful to help ophthalmologists find RP patients. We experimented with VGG16, Resnet50, and Inception V3 to construct the model. As a result, Inception V3 reached the highest accuracy. Since Resnet50 and Inception V3 are improved networks of VGG16, they performed better than VGG16. For an image classification task like this, features can exist anywhere in the image at various sizes. That is why choosing a kernel size is difficult. Inception V3 solves this problem by setting filters of various size at the same level at the same time instead of choosing one. This is probably the reason why Inception V3 could achieve the best performance. In comparison to the prior study conducted by Chen et al., which reported sensitivity and specificity values of 91.2% and 91.71%, respectively, we achieved higher sensitivity and specificity rates in our classification model. This improvement underscores the effectiveness of our approach in accurately identifying individuals with RP, thereby highlighting the potential clinical utility.

Our Grad-Cam analysis, as shown in Fig. 3, revealed that the heatmap predominantly highlights the peripheral regions. This inclination can be rationalized by the typical progression of RP from the periphery inward, leading to more prominent degenerative changes in peripheral areas. Consequently, the focus on these regions appears to be a logical outcome.

In the two false positive cases, the program primarily concentrated on the central area of the image. The probable cause for this misfocus might arise from the misinterpretation of the macular Tigroid pattern as retinal degenerative changes. Additionally, these errors could stem from insufficient data. These findings underscore the necessity for further exploration into the factors contributing to misfocusing, particularly in distinguishing macular patterns from degenerative changes, to refine the accuracy of the classification model.

In summary of the visualization, it can be inferred that the model predominantly directs attention to pigmentation in the periphery and the macula for prediction, mirroring the approach of ophthalmologists. This suggests the potential for accurate classification of unknown fundus images.

Going back to the performance of the model was as high as that of the ophthalmologists’ diagnoses. The reason for the high performance despite the relatively small amount of data is thought to be that the abnormal findings were relatively easy to detect.

4.2 Limitations and future research

Our work has several limitations. First, the number of images used in the study was limited, and further accumulation of training data is needed to achieve a practical level. Second, this model only detects the presence or absence of RP and cannot determine the causative gene, stage of the disease, or prognosis. Third, All images used in this study were only taken at Keio University Hospital. To increase the versatility of this model for clinical application, it is necessary to collect data not from a single institution but from multiple institutions.

This study is the first to report the development of an AI to diagnose RP from fundus images of Japanese patients. It achieved results comparable to ophthalmologists' diagnoses. Additionally, by utilizing Grad-CAM, we have confirmed that the model prioritizes the peripheral areas, aligning with the known progression pattern of RP. We also discovered that the model may mistakenly identify macular patterns as indicative of RP The results of this study suggest the usefulness of using deep learning for the automatic determination of RP or tools to reduce the burden on ophthalmologists.

AI; Artificial Intelligence, AUC; Area Under Cover, FPR; False Positive Rate, GradCAM; gradient-weighted class activation mapping, ROC; Receiver Operating Characteristic, RP; retinitis pigmentosa, TPR; True Positive Rat

Acknowledgments

The authors thank the members of the Laboratory of Photobiology, Keio University School of Medicine for their technical and administrative support. Especially, we would like to express our gratitude to those who provide fundus images and who were involved in clinical practice at photographed the data.

Author contributions

Saki Ubukata: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing - Original Draft, Visualization

Kanato Masayoshi: Conceptualization, Methodology, Software, Formal analysis, Investigation, Visualization

Yusaku Katada: Conceptualization, Methodology, Resources, Data Curation, Writing - Review & Editing, Funding acquisition

Lizhu Yang: Methodology, Resources, Data Curation, Writing - Review & Editing

Nobuhiro Ozawa: Resources, Data Curation, Writing - Review & Editing, Funding acquisition

Mari Ibuki: Resources, Data Curation, Writing - Review & Editing, Funding acquisition

Kazuno Negishi: Writing - Review & Editing, Supervision

Toshihide Kurihara: Writing - Review & Editing, Supervision

Data availability

The dataset is not publicly available due to its containing information that could compromise the privacy of research participants but are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.

Curr Genomics. 2011;12(4):260–266.
Prado DA, Acosta-Acero M, Maldonado RS. Gene therapy beyond luxturna: a new horizon of the treatment for inherited retinal disease. Curr Opin Ophthalmol. 2020;31(3):147–154.
Miraldi Utz V, Coussa RG, Antaki F, Traboulsi EI. Gene therapy for RPE65-related retinal disease. Ophthalmic Genet. 2018;39(6):671–677.
Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103(2):167–175.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402–2410.
Chen G, Yu M, Li J. Prediction of different eye diseases based on fundus photography via deep transfer learning. J Clin Med. 2021;10(230):5481.
Chen TC, Lim WS, Wang VY, Ko ML, Chiu SI, Huang YS, Lai F, Yang CM, Hu FR, Jang JSR, Yang CH. Artificial intelligence–assisted early detection of retinitis pigmentosa—the most common inherited retinal degeneration. J Digit Imaging. 2021;34(4):https://doi.org/10.1007/s10278-021-00479-6.
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 618–626.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. Preprint at https://arxiv.org/abs/1409.1556. Accessed 19 Jan 2021.
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015:2818–2826.
Targ S, Almeida D, Lyman K. Resnet in resnet: generalizing residual architectures. Preprint at https://arxiv.org/abs/1603.08029.
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR09), Miami, FL, USA, 20–25 June 2009.

No competing interests reported.

Download PDF

Reviewers agreed at journal
22 Sep, 2024
Reviews received at journal
24 Aug, 2024
Reviewers agreed at journal
11 Aug, 2024
Reviewers agreed at journal
11 Aug, 2024
Reviewers invited by journal
09 Aug, 2024
Editor assigned by journal
09 Aug, 2024
Editor invited by journal
03 Aug, 2024
Submission checks completed at journal
03 Aug, 2024
First submitted to journal
03 Aug, 2024

You are reading this latest preprint version

Fundus Image Analysis of Retinitis Pigmentosa Using Artificial Intelligence

Status:

Version 1

Abstract

Figures

1. Introduction

2. Material and Methods

2.1 Dataset

2.2 Model

2.3 Evaluation

2.4 Visualization

3. Results

3.1 Image collection

3.2 5-fold cross validation

3.3 Visualization

3.4 Performance comparison with Ophthalmologists and a Medical student

4. Discussion

4.1 Performance

4.2 Limitations and future research

5. Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1