The structural features of retina have been shown to be closely related to many ophthalmological diseases. For example, the thickness of the retina, especially the nerve fiber layer (NFL), has been used to indicate the progression of glaucoma . It has been reported that the junction of the Inner segment (IS) and the Outer segment junction (OSJ) can facilitate the diagnosis of retinitis pigmentosa [2, 3]. Quantitative assessment of retinal pigment epithelium (PRE) is also useful in diagnosing some age-related macular degeneration (AMD) . Thus, quantitative research on the retinal features has clinical value. Optical coherence tomography (OCT), as a noninvasive, label-free and high-resolution imaging modality, has been proven useful to diagnose various retinal diseases . The accurate and reliable segmentation of retinal layers in OCT images is a key step in the quantitative study of retinal features.
Commercial OCT devices are generally equipped with some kind of image analysis software which is able to perform retinal segmentation with varying success rates . However, the details of their design are undisclosed . In addition, most methods are designed for specific equipment . This makes them difficult to replicate in independent studies or to be improved by other scholars.
Machine learning approaches, including support vector machines , random forest , and Bayesian artificial neural networks , are proven to obtain satisfactory solutions in a noisy environment . However, the huge amount of labeled data used in machine learning is difficult to obtain. Relying solely on manual segmentation will make this task extremely difficult. Furthermore, manual segmentation is greatly affected by changes in observers . Therefore, the automatic retina segmentation algorithm based on non-machine learning still has certain practical value. It can be used alone, or it can provide a large amount of label data for machine learning under human supervision.
Anisotropic filtering seems very promising for improving contrast and boundary detection rate , but it also enhances some vertical boundary caused by vascular shadows. In order to avoid this influence, neighborhood information is often used by some algorithms. The prerequisite is that the retinal layers tend to be horizontal. So, A-lines alignment on the OCT images is often required before layer segmentation. Qi Yang et al.  used a large size gradient template to incorporate adjacent information and deal with vascular shadow and artifacts. Before that, the IS/OS boundary was first detected, and then the images were further aligned along the IS/OS boundary. Garvin et al. proposed a 3D graph search approach, which need to first align all the slices and straightened the RPE layer . Zhang et al. proposed a segmentation algorithm based on two-step predenoising filtering, which can only be executed after A-lines alignment . However, a number of severe eye diseases (AMD, choroidal neovascularisation (CNV), glaucoma, etc.) can change the structure of retina and choroid . In this case, A-lines alignment cannot be performed.
The active contour model is good at finding local optima, but its limitation is that the algorithm cannot converge to the optimal boundary unless the initial point is close enough [19, 20]. Dynamic programming algorithms, due to their significant advantages in solving optimization strategy problems, have been widely used in retina segmentation. Koozekanani et al. used A Markov boundary model to connect the rough edge . But it is sensitive to noise, so the detected layer boundaries can easily deviate from the true boundary. Mishra et al.  presented a promising two-step algorithm based on a kernel optimization scheme. First, approximate positions of the boundaries were found, and then by using dynamic programming the boundaries were refined to obtain the ideal segmentation results; however, no quantitative evaluation on a large data set was given. Stephanie J. et al. proposed an automatic initialization method that bypasses the need for manual endpoint selection of the dynamic programming algorithm . Tian et al.  proposed a shortest path based graph search method to detect retinal boundaries by searching the shortest path between nodes at both ends. The time complexity was reduced by the limitation of the search region and down-sampling. In order to prevent the algorithm from accidentally segmenting other structures in place of the target feature, it is often necessary to limit the graph to a valid search space that excludes any irrelevant content when using dynamic programming methods. However, the selection of search space will be adjusted artificially according to actual needs. Apparently, if an algorithm relies heavily on human intervention, it will be difficult to deal with large amounts of data, and more importantly, its performance is difficult to evaluate objectively. Therefore, the development of a retinal segmentation algorithm with high accuracy and robustness without human intervention has important clinical significance.
Among the fully automatic edge detection algorithms, Canny operator is recognized as the best operator so far, and is often used to obtain the initial boundary of the retina in retinal segmentation. Unfortunately, if the Canny operator is applied to retinal edge detection without any modification, it will face some limitations. First, it is susceptible to the shadow of blood vessels and detects many borders perpendicular to the retinal layer. Second, the detected boundary may be discontinuous, especially when there are many blood vessels or the image quality is not ideal. Additionally, it is sensitive to noise interference and is prone to misdetection or missed detection. Qi Yang et al. combined Canny edge maps and the axial intensity gradient maps to segment the retinal layer . The axial intensity gradient maps provide complementary search guidance where Canny edge information is missing or weak. The layer boundary is then extracted by a shortest path search applied to the graph using a dynamic programming algorithm. Although this method can alleviate part of the problem of edge discontinuity, the vertical boundary caused by blood vessel shadow can only be weakened, but cannot be completely eliminated, which increases the uncertainty of boundary selection.
In this paper, we proposed an automated retinal layer segmentation algorithm based on improved Canny operator. This method adds a multi-point boundary search step on the basis of the traditional Canny operator, and adjusts the convolution kernel function. The improved Canny operator has a dramatic improvement in the extraction of the retinal boundary. To make it easier to understand, we divide the algorithm into three parts: (1) Image enhancement, this section includes image denoising, gradient calculation and non-maximum suppression, which are similar to but slightly different from the first three steps of Canny operator. (2) Boundary search, multiple gradient peak points are selected as seed points to search the retinal boundaries and superimpose the obtained boundaries together. (3) Boundary selection, the number of superposed boundary is converted into probabilities, and the double threshold method in Canny operator is used to select and connect the edges. This method can accurately distinguish eleven retinal boundaries without additional intervention such as A-lines alignment, manual initialization, parameter adjustment or search space restriction. Quantitative and qualitative methods were used to verify the accuracy and stability of the algorithm.