There are many fields in ophthalmology that require improvement in visibility. This is because there are many types of imaging tests to diagnose diseases, and most surgical treatments are performed with the help of a microscope. Vitreoretinal surgery, which is performed using an additional magnifying lens to the basic microscope, is a representative ophthalmic surgery where visibility is important.
Ensuring successful vitreoretinal surgery requires skilled surgeons equipped with a clear and comprehensive view of the ocular structures. In particular, when performing procedures such as macular membrane peeling, detailed manipulation is required. As mentioned above, the 3D heads-up visualization system enables surgery to be performed using images converted via adjustment to the image parameters.6 We believe that this can be facilitated by obtaining image parameters optimized for the color of each patient’s retina and vitreous body, as well as by developing a deep learning algorithm to predict the optimal image parameters. In this study, we attempted to predict and apply parameters for optimal imaging during 3D heads-up vitreoretinal surgery using both deep learning algorithms and digital image enhancement methods.
This study aims to derive optimal parameter values applicable to 3D heads-up visualization systems for digital image enhancement. To achieve this goal, we employed a two-stage deep learning algorithm designed to predict the parameter values for optimizing surgical images. Briefly, several pairs of images were learned using a deep learning algorithm by repeating the process of obtaining an optimal surgical image from the original surgical image through manual parameter adjustments in the software. For the two-stage deep learning algorithm, first, a fake image was created via the Pix2Pix approach; next, the original and generated fake images were learned to predict the parameter values more efficiently via the ResNet architecture. Consequently, satisfactory PSNR and SSIM values were measured (34.59 ± 5.34 and 0.88 ± 0.08, respectively), which were similar to or slightly better than those yielded by the deep learning algorithm developed by other authors (Table 1).3, 22–24 The SSIM is a measurement indicator for the human visual system that considers factors such as luminance, contrast, and structure instead of simple objective differences between images.18, 19 In other words, an SSIM value of approximately 1 implies that humans, particularly vitreoretinal surgeons, do not perceive any difference between manually adjusted and optimized surgical images. Therefore, the performance of the proposed algorithm can be regarded as excellent.
The sharpness, brightness and contrast values of the RGB channels increased, with a significant difference indicated in the optimized surgical image; this is attributable to an increase in the objective clarity of the image. We speculate that this occurred because vitreoretinal surgeons adjusted the parameter values in the direction of increasing these values to clearly observe the vitreoretinal surface while creating a manually adjusted image, and that the deep learning algorithm accurately predicted the parameter values for a similar image.
In addition, we conducted a survey with seven vitreoretinal surgeons to assess the utility of adjusting the image parameters. Most reported that peeling the macular membrane without staining the original images would be difficult, although it can be attempted using the optimized images. In addition, most of them reported that the optimized images offered better visibility and expressed their preference for optimized images for performing operations.
The contrast in visual perception is the difference between two or more regions of a field.25 If the ERM is prominent in the macula, then several folds are created on the retinal surface. Clearer folds aid surgeons in peeling the membrane. Therefore, the color contrast of the folds can significantly affect the visualization of the ERM. Notably, it is generally determined by the relationship between the front and back luminances.21 In some previous studies, the visibility during vitreoretinal surgery was compared based on the CCR, where a high CCR corresponded to high visibility.10, 21 In our study, the optimized fundus images showed significantly improved CCRs. In other words, adjusting the image parameters improved the visibility of the ERM folds.
The ultimate goal of enhancing surgical images is to enable surgeons to operate effectively and safely. To evaluate this, we requested two vitreoretinal surgeons to measure and compare the ERM size for each fundus image. If the visibility of the ERM improves, then the average measured ERM area is expected to increase, and the difference in measurement between investigators is expected to decrease.26 In our study, the mean ERM area in the optimized fundus images was significantly wider than that in the original fundus images. This shows that performing surgery after optimization is more useful as it allows the ERM to be identified more easily. Additionally, the mean difference between the two measurements was closer to zero in the optimized fundus images (-2563.48 pixels) than in the original fundus images (-5878.76 pixels), which can be interpreted as a lower risk of overestimation or underestimation of the ERM in the optimized images. In summary, the ERM was verified more extensively, and the difference in measurement between the investigators decreased by using the optimized fundus images; therefore, the surgery can be rendered more efficient and safer by adjusting the parameters for optimization.
Finally, after adjusting the image parameters, the number of readable letters by the same investigators on the Pelli–Robson contrast sensitivity chart increased. This can be interpreted as the same person having a higher contrast sensitivity through image parameter adjustment.
Despite the advantages, this study presents several limitations. First, the deep learning algorithm cannot be readily applied to actual surgical scenarios. The actual application goal of our study is to convert images in real time during surgery to proceed through the most optimal surgical view. This is because of licensing issues such as technology transfer and technical limitations in performing surgery while applying optimal parameters automatically in real time. Nevertheless, given the capability of our deep learning algorithm to predict optimal parameters, coupled with the option for manual input and application by the assistant, we believe that this study sufficiently demonstrates the potential for future utilization. Second, the CCR and ERM size measurements were conducted in vitro using high-resolution fundus images with prominent ERMs instead of using actual surgical images. In the actual surgical images, the ERM was not clearly visible prior to staining with dye or triamcinolone acetonide, and the quality of the captured image was low; hence, actual surgical images are not suitable as they do not allow the necessary values to be obtained for analysis. Further studies regarding these variables based on actual surgical images are necessary to evaluate the quantitative improvement more accurately. Finally, as this study is based on a retrospective design, these limitations should be addressed in prospective studies.
Despite these limitations, there were no studies that used both methods of deep learning algorithms and digital image enhancement to improve the visibility and suitability of ophthalmic surgical images. Although the method of this algorithm does not optimize the video every moment during the surgery, it allows a good view of the vitreoretinal surface during macular membrane peeling, which is the most important moment in all processes. In general, the proposed deep learning algorithm demonstrated excellent performance, and the visibility of the predicted optimal surgical image improved objectively, thus allowing it to be used by vitreoretinal surgeons to perform vitrectomies. In conclusion, applying digital image enhancement using deep learning algorithms to actual surgeries seems promising in the near future.