Enhancing Volumetric Segmentation in Wide-Field OCT Images with a Semi- Supervised Learning Framework: Cross-Teaching CNN and Transformer Integration

doi:10.21203/rs.3.rs-4704316/v1

Download PDF

Article

Enhancing Volumetric Segmentation in Wide-Field OCT Images with a Semi- Supervised Learning Framework: Cross-Teaching CNN and Transformer Integration

https://doi.org/10.21203/rs.3.rs-4704316/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Wide-field optical coherence tomography (OCT) imaging can enable monitoring of peripheral changes in the retina, beyond the conventional fields of view used in current clinical OCT imaging systems. However, wide-field scans can present significant challenges for retinal layer segmentation. Deep Convolutional Neural Networks (CNNs) have shown strong performance in medical imaging segmentation but typically require large-scale, high-quality, pixel-level annotated datasets to be effectively developed. To address this challenge, we propose an advanced semi-supervised learning framework that combines the detailed capabilities of convolutional networks with the broader perspective of transformers. This method efficiently leverages labelled and unlabelled data to reduce dependence on extensive, manually annotated datasets. We evaluated the model performance on a dataset of 74 volumetric OCT scans, each performed using a prototype swept-source OCT system following a wide-field scan protocol with a 15x9 mm field of view, comprising 11,750 labelled and 29,016 unlabelled images. Wide-field retinal layer segmentation using the semi-supervised approach show significant improvements (P-value < 0.001) of up to 11% against a UNet baseline model. Comparisons with a clinical spectral-domain-OCT system revealed significant correlations of up to 0.91 (P-value < 0.001) in retinal layer thickness measurements. These findings highlight the effectiveness of semi-supervised learning with cross-teaching between CNNs and transformers for automated OCT layer segmentation.

Biological sciences/Computational biology and bioinformatics/Machine learning

Physical sciences/Engineering/Biomedical engineering

Health sciences/Diseases/Eye diseases

OCT layers

semi-supervised learning

CNN

transformer

cross-teaching

Optical Coherence Tomography (OCT) is crucial in the diagnosis and monitoring of various eye diseases, providing non-invasive, high-resolution, cross-sectional images of the retina [1–3]. Assessment of these layers is vital for accurate assessment of ocular health and for monitoring diseases such as age-related macular degeneration, high myopia, diabetic retinopathy, and glaucoma [2, 3, 4]. The introduction of wide-field Swept-Source OCT (SS-OCT) has further enhanced these diagnostic capabilities by providing wider fields of view to allow monitoring of posterior vitreous detachment [5] and peripheral retinal degeneration such as in diabetic retinopathy, where lesions and non-perfusion areas beyond the central region have been reported [6]. The volumetric nature of data from OCT scans make manual annotation of these layers laborious and error-prone. Consequently, automated segmentation of OCT layers is essential, not only to ease the workload of ophthalmologists but also to enhance the accuracy of diagnoses. However, the segmentation of wide-field OCT can be highly challenging due to increased variability and potentially reduced visibility of retinal features [7–8].

Early methods for segmenting retinal and choroidal layers in OCT images relied on techniques that utilized hand-crafted features, such as graph-based methods, active contour models, and conventional machine-learning algorithms [9–14]. These methods frequently faced difficulty accurately segmenting layers, especially within diseased or noisy images. The advent of deep learning, and particularly the UNet architecture, has brought transformative changes to OCT layer segmentation [15–22]. Enhancements such as the cascaded UNet [17], which refines the architecture for fully automated segmentation, and the integration of UNet with architectures like VGG16 and ResNet50 [18], represent significant advancements. Furthermore, the

shortest path framework [19], which merges graph-based algorithms with UNet, has improved segmentation accuracy. The SA-Net architecture merges spatial context from adjacent slices to reconstruct the central slice, integrating this with a UNet-based architecture for choroidal layer segmentation [21]. A comprehensive analysis of eight different UNet models across various OCT datasets [22] reveals the diverse effectiveness of these approaches, underscoring the ongoing need for refinement and innovation in this field. Further, these approaches have largely been evaluated on OCT scans with smaller fields of view.

Despite advancements in supervised deep learning methods, challenges arise primarily from their dependence on extensive, labelled datasets. This has led to a surge in research focusing on semi-supervised learning methods [15, 23–32], aimed at reducing labelling costs and enhancing segmentation efficiency. Among these developments, Huang et al. [23] introduced a pseudo-labelling method for retinal OCT that aids robotic surgical systems. Liu et al. [24] developed a GANs-based approach for segmenting retinal layers, while Sedai et al. [25] implemented a student-teacher framework for RNFL layer segmentation. Nevertheless, CNN-based methods encounter limitations in capturing the global context of features due to their localized processing nature [30, 33, 34].

Recent studies have integrated transformers into semi-supervised tasks, focusing on capturing a more comprehensive context of features [30, 31, 35–38]. These models merge supervision from both CNN and transformer pipelines to predict each other’s outputs, which aids in generating pseudo-labels. This approach has proven effective in diverse medical image segmentation tasks, including cell segmentation [28], heart MRI segmentation [30, 35], lung lesion detection [36], brain tumor identification [37], and CT spleen imaging [38]. Nevertheless, despite its proven effectiveness in various medical imaging fields, this technique has not been extensively implemented in ophthalmology, particularly in the detailed layer segmentation of OCT images. This gap in application suggests the potential for significant advancements and shifts in image analysis techniques within ophthalmology.

In this study, we build upon these semi-supervised learning (SSL) advancements by introducing a model that employs a cross-teaching framework [30] for segmenting six specific OCT layers from a 15x9 mm wide-field SS-OCT system. In experiments conducted with volumetric OCT images, our method shows superior segmentation compared to existing supervised deep learning models: UNet and Swin-UNet. The primary contributions of our study are: (1) the successful implementation of the cross-teaching framework for segmenting OCT layers, and (2) the establishment of a new benchmark for semi-supervised learning within OCT image segmentation. These achievements are significant for enhancing diagnostic efficiency and accuracy in ophthalmology.

The structure of this paper is organized as follows: Section 2 provides a review of related literature; Section 3 presents our experimental results; Section 4 discusses our finding and concluding remarks; Finally, Section 5 details our methodology.

Automated segmentation of medical images is vital for computer-aided diagnosis systems across diverse clinical applications. A common challenge in these tasks is the limited availability of annotated data. To overcome this, many researchers are turning to semi-supervised methods for medical image segmentation [39–48]. These methods combine a relatively small amount of labelled data with a much larger batch of unlabelled data. This approach reduces reliance on large labelled datasets, making it more cost-effective and reducing the risk of human error in labelling. Moreover, semi-supervised learning algorithms can leverage the inherent structure of the data, thereby improving their ability to generalize from limited labelled information. In the field of semi-supervised learning, researchers have developed various methods, including self-training [39, 40], consistency regularization [41–43], co-teaching [44], deep co-training [45], and the use of generative models [46–48], among others. Many of these techniques utilize UNet or its variations, demonstrating promising results in diverse tasks. Nonetheless, CNN-based methods have their limitations, particularly in their ability to capture the global context of features, due to their localized processing nature. Vision transformers [33] have emerged as a solution to this issue, demonstrating significant success in vision recognition tasks [33, 34] by effectively processing entire images and learning long-range spatial relationships. However, they also face significant challenges, especially in terms of data requirements and computational resources. Vision transformers typically need large datasets for training to fully leverage their self-attention mechanisms. Training transformers in a semi-supervised manner presents a complex challenge, particularly in the context of medical image analysis where data is often limited. To address this, a cross-teaching framework has recently been proposed and has proven effective in medical image segmentation tasks [30, 31, 35–38]. These techniques capitalize on the combined strengths of CNN and transformer models, merging these distinct learning paradigms. While CNNs are adept at processing local information, transformers are great at modelling global spatial relationships. Luo et al. [30] developed a cross teaching framework that integrates the principles of co-teaching [44], deep co-training [45] and cross pseudo supervision [49]. This approach implements diverse perturbations: input-level through co-teaching, supervision-level via deep co-training, and architectural-level with cross pseudo supervision, focusing on consistent predictive accuracy during training. Xiao et al. [35] presented a dual-teacher-guided architecture integrated with uncertainty perception. This framework adopts consistency regularization and pseudo-label learning methods. It features a student network with a CNN backbone for simultaneous training and prediction of each image, supported by dual-teacher networks that employ a combination of CNN and transformer backbones. Likewise, Zhao et al. [36] merged consistency regularization techniques with pseudo labelling approaches for segmenting lung lesions. Yang et al. [38] developed a cross-guidance method between CNN and hybrid Transformer. This method employs dual regularization strategies to generate broader representations for unlabelled data. It entails introducing diverse perturbations to the feature maps from the hybrid transformer encoder and preserving consistency in predictions, thereby enhancing the ability of the encoder to represent data effectively.

3.1. Data

We evaluated our proposed model on a dataset comprising glaucoma and normal images, collected at the Singapore Eye Research Institute, a specialized eye care institution in Singapore. The study adhered to the ethical guidelines of the Declaration of Helsinki and was approved by the SingHealth Centralized Institutional Review Board. All participants consented in writing before participating in the study. Wide-field OCT imaging was performed with a prototype swept-source system (PlexElite SS-OCT, Zeiss Meditech, Dublin, CA), which has a field-of-view of up to 56° with a depth of 3mm, to acquire volumetric OCT scans using a 15 mm x 9 mm scan protocol. Only OCT volumes with signal strengths 6 and above, and with no substantial artefacts as determined by a manual grader, were included. Each volume, as depicted in Fig. 1, contained 834 cross-sectional B-scan images, with each cross-sectional scan having a resolution of 500 x 512 pixels. Due to differences in retinal layer structure at the optic nerve head, B-scans that included the optic nerve area were excluded.

The acquired OCT dataset comprised of 74 volumetric scans, with 19 volumes without glaucoma from 19 healthy participants, and 55 volumes with glaucoma from 49 glaucoma patients. No other ocular diseases were present in the participants. A summary of the participant characteristics can be found in Supplementary Table S1. Of these, ground truth labels were available for the volumes from 13 healthy eyes and 9 glaucoma eyes. The ground truth for these volumes was established through manual annotation by a single grader masked to the disease status, and labelled as six distinct layers: the retinal nerve fiber layer (RNFL), the combined layers of ganglion cell and inner plexiform (GCL-IPL), inner nuclear and outer plexiform (INL-OPL), outer nuclear layer (ONL), the interface from inner segment/outer segment to retinal pigment epithelium (IS/OS-RPE), and the choroidal layer. Detailed illustrations of each layer are presented in Fig. 1(c). The labelled dataset, was further split into a training subset of 12 volumes (8 healthy, 4 glaucoma), a validation subset of 6 volumes (3 healthy, 3 glaucoma), and a test dataset of 4 volumes (2 healthy, 2 glaucoma). An illustration of the distribution is provided in Supplementary Figure S1. The cross-sectional B-scans from the labelled dataset, totalling 11,750 images, were stratified by participant, such that images from one participant remained in the same training, validation, or test subset. In total, there were 6,172 images in the training subset, 3,184 images in the validation subset, and 2,394 images in the test subset. The remaining 52 volumes formed an unlabelled dataset of 29,016 B-scans.

To compare the results of our segmentation, we acquired an independent set of 15 glaucomatous eyes and 15 normal eyes who underwent imaging with both the SS-OCT system and a clinically-used spectral-domain (SD-OCT) system (Cirrus SD-OCT, Zeiss Meditech, Dublin, CA). Characteristics of the data used for this comparison are provided in Supplementary Table S2. For the SD-OCT system, a 200x200 macular protocol, equivalent to a 6mm x 6mm region centered on the macula, was performed. Using the built-in review software in the SD-OCT system, we extracted measurements of the RNFL and GCL-IPL based on an annulus with an inner diameter of 1 mm vertically and 1.2 mm horizontally, and an outer diameter of 4 mm vertically and 4.8 mm horizontally, centered on the macula (Supplementary Figure S2) [50]. Measurements of superior, inferior, and global thicknesses were obtained. The same measurement protocol was applied to the segmented SS-OCT images of the same eyes. The location of the macula centers in the SS-OCT scans were manually determined by an experienced grader through assessing the anatomical structure in the B-scans.

3.2. Experimental results

Our study implemented a semi-supervised learning approach using cross-teaching techniques to segment OCT images into six distinct layers: RNFL, GCL-IPL, INL-OPL, ONL, IS/OS-RPE, and the choroidal layer. In this semi-supervised learning setup, OCT image segmentation is achieved using two distinct networks: a CNN-based UNet and a Transformer-based Swin-UNet, with training conducted on a Tesla V100 DGXS 32GB GPU using PyTorch. The training utilizes the SGD optimizer, with a batch size of 4, for a total of up to 200k iterations. The initial learning rate is set at 10^− 4 and is adjusted using a poly learning rate strategy. Training incorporates both labelled and unlabelled datasets, where labelled data undergoes individual ground truth supervision, and losses are computed using cross-entropy and Dice loss. For the unlabelled data, model predictions are utilized to generate pseudo labels through the argmax function, contributing to the cross-teaching loss calculation. The overall loss is modulated by a Gaussian warming-up function, which combines the supervised loss with the cross-teaching loss to optimize the training’s effectiveness.

We evaluated the efficacy of our approach by comparing it with supervised UNet and Swin-UNet baseline models. The supervised models were trained only with the labelled dataset, consisting of with 12 volumes, validated on 6, and tested on 4 volumes. To enhance the training process, we introduced an additional 60 unlabelled volumes through cross-teaching approaches. The lowest validation loss identified the best model for each layer. The performance of the segmented models was evaluated using the Intersection over Union (IoU) metric. The IoU quantifies the extent of overlap between the model predictions and the ground truth mask. It is calculated using the formula:

$$\:IoU(P,Y)\:=\:\frac{|P\:\cap\:\:Y|}{|P\:\cup\:\:Y|}$$

where: P denotes the set of pixels in the predicted segmentation and Y represents the set of pixels in the ground truth segmentation.

Figure 2 illustrates the performance evaluation of the cross-teaching SSL between UNet and Swin-UNet by varying the amounts of labelled data in training for RNFL layer segmentation. Initially, the model was trained with 3 labelled volumes and 60 unlabelled volumes, achieving an IoU score of 0.799 ± 0.046. Adding more labelled volumes in increments, specifically 6, then 9, and finally up to 12, led to a significant (P-value < 0.001) improvements in IoU scores, reaching 0.826 ± 0.042, 0.841 ± 0.037, and 0.857 ± 0.037, respectively. The increase in IoU scores on the test set as more labelled data is utilized indicates a positive impact on model generalization rather than overfitting.

Table 1 presents the results from an ablation study on RNFL layer segmentation. The study evaluates the performance of different semantic segmentation models and approaches. The baseline models used for comparison are UNet and Swin-UNet. Additionally, the study explores the use of semi-supervised learning with UNet and Swin-UNet backbones. Cross-teaching with semi-supervised learning between the UNet and Swin-UNet models was evaluated (CT-SSL). Among these, the CT-SSL, achieved the highest IoU score. This indicates its superior performance in segmenting the RNFL layer accurately.

Encouraged by these findings, we extended this cross-teaching SSL approach to segment all six OCT layers using 12 labelled datasets. The IoU scores presented in Table 2 demonstrate the effectiveness of the CT-SSL model compared to the baseline models, UNet and Swin-UNet with all layers showing significant improvements (P-values < 0.001). The choroid layer showed the largest improvement with IoU scores increasing from 0.914 ± 0.062 (UNet) and 0.918 ± 0.032 (SwinU-Net) to 0.945 ± 0.033 with our CT-SSL model. The results underscore the ability of the CT-SSL model in segmenting different retinal layers within the OCT images. Figure 3 illustrates the segmentation results achieved by our model for six distinct OCT layers: RNFL, GCL-IPL, INL-OPL, ONL, IS/OS-RPE, and the choroid. For each layer, 3D views showcased in Fig. 3(a) are generated from the segmented results derived from a consistent set of B-scans. Figure 3(b) displays a side-by-side comparison between our model's segmentations and the corresponding ground truth data. A detailed 3D rotation and orientation of the OCT scan and the segmented layers can be found in Supplementary Video S1. In this figure, the left column presents the ’Ground Truth’ for each layer, serving as a reference, while the right column demonstrates the ’Predictions’ made by our model. IoU scores are provided along with the detected layers. Additionally, the thickness maps for this volumetric scan, associated with the RNFL and GCL-IPL layers, are provided in Supplementary Figure S3.

Table 1

Ablation study for RNFL layer segmentation
Model	IoU	P-value*
UNet	0.747 ± 0.134	< 0.001
Swin-UNet	0.837 ± 0.033	< 0.001
UNet with SSL	0.839 ± 0.048	< 0.001
Swin-UNet with SSL	0.831 ± 0.035	< 0.001
Cross-Teaching (UNet and Swin-UNet) with SSL (CT-SSL)	0.857 ± 0.037	̶
SSL: Semi-Supervised Learning, IoU: Intersection over Union Values represent mean IoU and standard deviation *P-value represents significance testing using a paired t-test comparing CT-SSL with the iter models

Table 2

Performance evaluation for each layer with respect to manual ground truth annotation
Layer	CT-SSL	UNet	P-value*	Swin-UNet	P-value**
RNFL	0.857 ± 0.037	0.747 ± 0.134	< 0.001	0.837 ± 0.033	< 0.001
GCL-IPL	0.858 ± 0.043	0.782 ± 0.119	< 0.001	0.830 ± 0.036	< 0.001
INL-OPL	0.882 ± 0.029	0.824 ± 0.087	< 0.001	0.858 ± 0.022	< 0.001
ONL	0.914 ± 0.018	0.893 ± 0.053	< 0.001	0.905 ± 0.015	< 0.001
IS/OS-RPE	0.925 ± 0.025	0.890 ± 0.058	< 0.001	0.904 ± 0.019	< 0.001
Choroid	0.945 ± 0.033	0.914 ± 0.062	< 0.001	0.918 ± 0.032	< 0.001
CT-SSL: Cross-Teaching Semi-Supervised Learning Values represent mean IoU and standard deviation P-value of paired t-test comparing CT-SSL and UNet *P-value of paired t-test comparing CT-SSL and Swin-UNet

Table 3 compares the same eyes between the RNFL and GCL-IPL thicknesses obtained from our deep learning model and those from a clinical SD-OCT system. The measurements are segmented into three defined regions within the elliptical annulus: Superior, Inferior, and Global, differentiating between normal individuals and those with glaucoma. Scatter plots for our approach and the measurements from the SD-OCT system for the global RNFL and GCL-IPL thicknesses are shown in Figure 4. The left plot shows a moderate correlation (represented by a Pearson’s r of 0.67 and P-value < 0.001) for RNFL segmentation, while the right plot reveals a higher correlation for GCL-IPL segmentation (represented by Pearson’s r of 0.91 and P-value < 0.001), highlighting the model accuracy in identifying this layer, which plays an important role in the clinical diagnosis and management of glaucoma.

Table 3: Correlation of Segmented RNFL and GCL-IPL Layer Mean Thickness with Clinical Measurements from SD-OCT

RNFL: Retinal Nerve Fiber Layer; GCL-IPL: Ganglion Cell Layer to Inner Plexiform Layer

Pearson’s correlation values of the segmented mean thickness with respect to the clinical measurements in the corresponding regions are provided

This study adopted an innovative cross-teaching framework that merges the capabilities of CNN with transformer for OCT layer segmentation by leveraging a mix of labelled and unlabelled data. We assessed the model performance using a dataset of widefield OCT scans, which comprises 11,750 labeled and 29,016 unlabeled images, on a prototype swept-source wide-field OCT system following a scan protocol with a 15x9 mm field of view. A notable increase in segmentation accuracy was observed across various OCT layers, with enhancements of up to 11% for the RNFL compared to UNet model (t-statistic = 41.29, P-value < 0.001). Moreover, the model demonstrated a strong correlation of 0.91 (P-value < 0.001) in measuring the GCL-IPL layer thickness using a clinical SD-OCT system. Our results show that cross-teaching between CNN and transformer can serve as an effective technique for automated segmentation of wide-field OCT layers, which could enable analysis at the peripheral retina.

The main strength of our deep learning model is its semi-supervised learning framework, which utilizes both labelled and unlabelled data. This method addresses the common challenge of limited annotated data in medical imaging. Clinically, the model accuracy in segmenting complex OCT layers such as the RNFL and GCL-IPL can facilitate early glaucoma detection. This advancement has the potential to enhance preliminary screening processes and aid in patient management. By comparing our model performance with standard clinical measurements, we further bridge the gap between automated segmentation and practical clinical application.

Although our study presents an advance semi-supervised learning framework for OCT layer segmentation and has achieved promising results compared to the baseline models, there are some limitations to be addressed. One limitation arises from preprocessing steps such as cropping and resizing. While these methods are essential for preserving image consistency, they could lead to biases. For example, cropping might impact the depiction of OCT layers, which could affect how their thickness is perceived. In comparisons with the clinical measurements from SD-OCT, correlations between the two devices for RNFL were observed to differ from that of GCL-IPL. This could be attributed to the sensitivity of the thinner RNFL to differences in pose and alignment during acquisition at the two devices, which was performed separately. Another issue is that the datasets used here are from a single source. Considering that a single dataset typically does not offer a diverse range of images from various devices and races, there is uncertainty about whether the results could be consistently replicated across different populations without additional training.

Future work will aim to tackle these limitations by examining pre-processing methods to enhance image quality and uniformity, thereby reducing potential biases. We also aim to significantly expand our datasets to include a more diverse range of images from different devices and racial groups to prevent demographic bias. Moreover, a significant direction for future development is to advance from single-layer to multi-layer segmentation. The current study focuses on segmenting each OCT layer individually. Single-layer segmentation could be computationally intensive when applied to all OCT layers. Going forward, a multi-layer segmentation approach will be implemented. We aim to evaluate the model utility in various clinical environments, including wide-field OCT data from different platforms and the presence of different ocular pathologies, including the detection of lesions and non-perfusion beyond the central retina.

The SSL-based technique typically begins with an initial training phase using labelled data, which is crucial as it enables the model to subsequently generate pseudo-labels for the unlabelled data. In this study, we adopted a semi-supervised learning approach using a cross-teaching framework [30], as depicted in Fig. 5. The initial phase involves a preprocessing step that enhances the quality and consistency of the images. This is achieved by applying a moving average technique between the target slice and its adjacent slices, followed by standardizing the sizes of the images. Then, we combine the capabilities of two distinct networks: the CNN and the Transformer. In this setup, the labelled data undergoes simultaneous training through both the CNN and Transformer models. This dual training strategy is designed to allow each model to predict the other’s outputs, thereby facilitating the generation of accurate pseudo-labels for the unlabelled data.

5.1. Pre-processing

The pre-processing of OCT images is an essential step, aimed at improving image quality and preparing the data for effective segmentation. Initially, each OCT image undergoes a moving average filtering process. This involves averaging the pixel values of the target image with those of its adjacent images. By doing so, we reduce speckle noise and improve the overall smoothness of the image. This step is crucial for enhancing the visibility of features in the OCT layers, which are essential for accurate segmentation. After noise reduction, we crop the images to remove the black, non-informative margins, thus isolating the region of interest. This essential procedure eliminates unnecessary black space, centering the analysis on the significant portions of the images. By concentrating on the relevant areas of the retina, we ensure that the subsequent processing steps are more accurate and efficient. Finally, each cropped image is resized to a standard dimension 500 x 224 pixels. This standardization is imperative for maintaining uniformity across the dataset, as it ensures that all images are subjected to the segmentation algorithm under consistent conditions. Resizing the images to the same dimensions is also crucial for the comparability of features across different scans and facilitates the application of machine learning models, which often require input data of a uniform size.

5.2. Semi-supervised Learning Methodology

After the preprocessing of OCT images, they are utilized as inputs for the segmentation phase. Figure 5 depicts the semi-supervised learning approach that integrates cross-teaching between a CNN and a transformer. The framework for the CNN is based on the UNet architecture [51], while the transformer model employs the Swin-UNet [52]. The training collection is divided into two separate categories: the labelled dataset $\:{D}^{l}$, consisting of $\:N$ annotated images, and the unlabelled dataset $\:{D}^{u}$, containing $\:M$ images where typically $\:M$ is substantially larger than $\:N$. Collectively, the dataset is expressed as $\:D\:=\:{D}^{l}\cup\:{D}^{u}$. For any given image $\:{x}_{i}$ within $\:{D}^{l}$, there exists an associated ground truth $\:{y}_{i}$. However, for images $\:{x}_{i}$ within $\:{D}^{u}$, such ground truth data are not available.

For the labelled data, each input image, denoted as $\:{x}_{i}$, is subject to individual ground truth supervision for both the CNN and the transformer model. This dual-supervised approach yields a pair of predictive outcomes for every input as follows:

$$\:{P}_{i}^{c}\:=\:{f}_{c}\left({x}_{i}\right)$$

;

$$\:{P}_{i}^{t}\:=\:{f}_{t}\left({x}_{i}\right)$$

Here, $\:{P}_{i}^{c}$ and $\:{P}_{i}^{t}$ denote the predictions made by the CNN f_c (·) and Transformer f_t (·) correspondingly. The networks are tasked with predicting $\:{P}_{i}^{c}$ for the CNN and $\:{P}_{i}^{t}$ for the Transformer. With $\:{y}_{i}$ representing the true labels class, the total supervised loss is the aggregate of each network’s individual supervised loss as follows:

$$\:{L}_{sup}\:=\:{L}_{sup1}\:+{L}_{sup2}$$

Specifically, the losses are calculated for each network by combining the Dice Loss and cross-entropy loss:

$$\:{\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:L}_{sup1}=\:Dice({P}_{i}^{c}\:,{y}_{i})\:+\:CE({P}_{i}^{c},{y}_{i})$$

$$\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:{L}_{sup2}\:=\:Dice({P}_{i}^{t},{y}_{i})\:+\:CE({P}_{i}^{t},{y}_{i})$$

In handling unlabelled data, the predictive outputs from both the CNN and Transformer models significantly influence their respective parameter updates. This method leverages the predictions from f_c (·) and f_t (·) to form pseudo labels, a core aspect of the cross-teaching technique. These pseudo labels are determined by:

$$\:{\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:Pˆ}_{i}^{c}=\:argmax\left({P}_{i}^{c}\right)\:=\:argmax\left({f}_{c}\right({x}_{i}\left)\right)$$

$$\:{\:\:\:\:\:\:\:\:\:\:\:\:\:\:\:Pˆ}_{i}^{t}=\:argmax\left({P}_{i}^{t}\right)\:=\:argmax\left({f}_{t}\right({x}_{i}\left)\right)$$

The cross-teaching loss function is calculated by summing the Dice Loss between the CNN’s predictions and the Transformer’s pseudo labels, and vice versa. For each iteration, backpropagation of gradients between $\:{P}_{i}^{c}$ and $\:{Pˆ}_{i}^{c}$, as well as between $\:{P}_{i}^{t}$_i and $\:{Pˆ}_{i}^{t}$, is intentionally omitted (Stop-gradient). Thus, the formulas are presented as follows:

where,	$\:{L}_{ct}\:=\:{L}_{ct1}\:+\:{L}_{ct2}$	(9)
	$\:{L}_{ct1}=\:Dice({Pˆ}_{i}^{c}\:,{Pˆ}_{i}^{t})$	(10)
	$\:{L}_{ct2}\:=\:Dice\left({{Pˆ}_{i}^{t}\:,\:Pˆ}_{i}^{c}\right)$	(11)

The overall loss integrates the supervised loss, augmented by a weighting factor, with the loss derived from the cross-teaching approach:

$$\:{L}_{total}\:=\:{L}_{sup}\:+\:\lambda\:{L}_{ct}$$

The weight factor is determined by a Gaussian warming-up function, which is commonly used:

$$\:\lambda\:\left(t\right)=\:0.1·{e}^{{-5(1-\:\frac{{t}_{i}}{{t}_{total}})}^{2}}$$

where $\:{t}_{i}$ represents the current training iteration, and $\:{t}_{total}$ is the total number of iterations.

Competing interests

The authors declare no competing interests.

Supplementary Materials

All supplementary materials for this study are available in the supplementary tables and figures online.

Author Contribution

S.S., P.R., P.D.N.P., N.F.B.A.G., J.C., M.E.N., T.A., R.H., L.S. and D.W. jointly conceived and designed the research study. D.W., L.S. and J.C. take responsibility for data preparation and analysis. S.S. and P.R. contributed equally to the work by implementing the experiments and drafting the manuscript. Study concept and design: D.W., P.R., S.S., and L.S. Study supervision: D.W. and L.S. All authors have reviewed the manuscript and have given their consent to the published version

Acknowledgements

This work was funded by grants from the National Medical Research Council (OFLCG/004c/2018-00; MOH-000249-00; MOH-000647-00; MOH-001001-00; MOH-001015-00; MOH-000500-00; MOH-000707-00; MOH-001072-06; MOH-001286-00), National Research Foundation Singapore (NRF2019-THE002- 0006 and NRF-CRP24-2020-0001), Agency for Science, Technology and Research (A20H4b0141) and the Singapore Eye Research Institute & Nanyang Technological University (SERI-NTU Advanced Ocular Engineering (STANCE) Program).

Data Availability

The data supporting the findings of this study are not publicly available at this time, but may be requested upon reasonable request and with a formal research agreement from the corresponding author.

Bhende, M., Shetty, S., Parthasarathy, M. K., & Ramya, S. Optical coherence tomography: A guide to interpretation of common macular diseases. Indian Journal of Ophthalmology, 66(1), 20 (2018).
Ly, A., Phu, J., Katalinic, P., & Kalloniatis, M. An evidence-based approach to the routine use of optical coherence tomography. Clinical and Experimental Optometry, 102(3), 242–259 (2019).
Schuman, J. S., Puliafito, C. A., Fujimoto, J. G., & Duker, J. S. (Eds.). Optical Coherence Tomography of Ocular Diseases, 3rd Edition. Slack Incorporated, 2012.
Wong, D., Chua, J., Baskaran, M., Tan, B., Yao, X., Chan, S., Tham, Y.C., Chong, R., Aung, T., Lamoureux, E.L., Vithana, E.N., Cheng, C.Y., & Schmetterer, L. Factors affecting the diagnostic performance of circumpapillary retinal nerve fibre layer measurement in glaucoma. British Journal of Ophthalmology (2020).
Chiku, Y., Hirano, T., Takahashi, Y., Tuchiya, A., Nakamura, M., & Murata, T. Evaluating posterior vitreous detachment by widefield 23-mm swept-source optical coherence tomography imaging in healthy subjects. Scientific Reports, 11(1), p.19754 (2021).
Silva, P.S., Cruz, A.J.D., Ledesma, M.G., van Hemert, J., Radwan, A., Cavallerano, J.D., Aiello, L.M., Sun, J.K., & Aiello, L.P. Diabetic retinopathy severity and peripheral lesions are associated with nonperfusion on ultrawide field angiography. Ophthalmology, 122(12), pp.2465–2472 (2015).
Guo, Y., Camino, A., Zhang, M., Wang, J., Huang, D., Hwang, T., & Jia, Y. Automated segmentation of retinal layer boundaries and capillary plexuses in wide-field optical coherence tomographic angiography. Biomedical Optics Express, 9(9), pp.4429–4442 (2018).
Lezama, J., Mukherjee, D., McNabb, R.P., Sapiro, G., Kuo, A.N., & Farsiu, S. Segmentation guided registration of wide field-of-view retinal optical coherence tomography volumes. Biomedical Optics Express, 7(12), pp.4827–4846 (2016).
Zhang, L., Sonka, M., Folk, J. C., Russell, S. R., & Abramoff, M. D. Quantifying disrupted outer retinal-subretinal layer in sd-oct images in choroidal neovascularization. Investigative Ophthalmology & Visual Science, 55(4), 2329–2335 (2014).
Terry, L., Cassels, N., Lu, K., Acton, J. H., Margrain, T. H., North, R. V., Fergusson, J., White, N., Wood, A. Automated retinal layer segmentation using spectral domain optical coherence tomography: evaluation of intersession repeatability and agreement between devices. PLoS ONE, 11(9), e0162001 (2016).
Gawish, A., Fieguth, P., Marschall, S., & Bizheva, K. Undecimated hierarchical active contours for OCT image segmentation. In: 2014 IEEE International Conference on Image Processing (ICIP), IEEE, 2014, pp. 882–886.
Abramoff, M. D., Lee, K., Niemeijer, M., Alward, W. L., Greenlee, E. C., Garvin, M. K., Sonka, M., & Kwon, Y. H. Automated segmentation of the cup and rim from spectral domain OCT of the optic nerve head. Investigative Ophthalmology & Visual Science, 50(12), 5778–5784 (2009).
Mazzaferri, J., Beaton, L., Hounye, G., Sayah, D. N., & Costantino, S. Opensource algorithm for automatic choroid segmentation of OCT volume reconstructions. Scientific Reports, 7(1), 42112 (2017).
Srinivasan, P. P., Heflin, S. J., Izatt, J. A., Arshavsky, V. Y., & Farsiu, S. Automatic segmentation of up to ten-layer boundaries in sd-OCT images of the mouse retina with and without missing layers due to pathology. Biomedical Optics Express, 5(2), 348–365 (2014).
Fazekas, B., Aresta, G., Lachinov, D., Riedl, S., Mai, J., Schmidt-Erfurth, U., Bogunovic, H. Sd-layernet: Semi-supervised retinal layer segmentation in OCT using disentangled representation with anatomical priors. In: International Conference on Medical Image Computing and Computer Assisted Intervention, Springer Nature Switzerland, Cham, 2022, pp. 320–329.
Li, Q., Li, S., He, Z., Guan, H., Chen, R., Xu, Y., Wang, T., Qi, S., Mei, J., Wang, W. Deepretina: layer segmentation of retina in OCT images using deep learning. Translational Vision Science & Technology, 9(2), 61 (2020).
Yadav, S. K., Kafieh, R., Zimmermann, H. G., Kauer-Bonin, J., NouriMahdavi, K., Mohammadzadeh, V., Shi, L., Kadas, E. M., Paul, F., Motamedi, S., Brandt, A. U. Deep learning-based intraretinal layer segmentation using cascaded compressed u-net. medRxiv, 2021–11 (2021).
Man, N., Guo, S., Yiu, K. F. C., Leung, C. K. S. Multi-layer segmentation of retina OCT images via advanced u-net architecture. Neurocomputing, 515, 185–200 (2023).
Mishra, Z., Ganegoda, A., Selicha, J., Wang, Z., Sadda, S. R., Hu, Z. Automated retinal layer segmentation using graph-based algorithm incorporating deep-learning-derived information. Scientific Reports, 10(1), 9541 (2020).
Xu, X., Wang, X., Lin, J., Xiong, H., Wang, M., Tan, H., Xiong, K., Han, D. Automatic segmentation and measurement of choroid layer in high myopia for OCT imaging using deep learning. Journal of Digital Imaging, 35(5), 1153–1163 (2022).
Cahyo, D. A., Yow, A. P., Saw, S. M., Ang, M., Girard, M., Schmetterer, L., & Wong, D. Multi-task learning approach for volumetric segmentation and reconstruction in 3d OCT images. Biomedical Optics Express, 12(12), 7348–7360 (2021).
Kugelman, J., Allman, J., Read, S. A., Vincent, S. J., Tong, J., Kalloniatis, M., Chen, F. K., Collins, M. J., & Alonso-Caneiro, D. A comparison of deep learning U-net architectures for posterior segment OCT retinal layer segmentation. Scientific Reports, 12(1), 14888 (2022).
Huang, Y., Asaria, R., Stoyanov, D., Sarunic, M., & Bano, S. Pseudosegrt: efficient pseudo-labelling for intraoperative OCT segmentation. International Journal of Computer Assisted Radiology and Surgery (2023), 1–8.
Liu, X., Cao, J., Fu, T., Pan, Z., Hu, W., Zhang, K., & Liu, J. Semi-supervised automatic segmentation of layer and fluid region in retinal optical coherence tomography images using adversarial learning. IEEE Access, 7, 3046–3061 (2018).
Sedai, S., Antony, B., Rai, R., Jones, K., Ishikawa, H., Schuman, J., Gadi, W., & Garnavi, R. Uncertainty guided semi-supervised segmentation of retinal layers in OCT images. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part I. Vol. 22 of Lecture Notes in Computer Science. Springer International Publishing, 2019, pp. 282–290.
Gao, F., Hu, M., Zhong, M. E., Feng, S., Tian, X., Meng, X., Huang, Z., Lv, M., Song, T., Zhang, X., & Zou, X. Segmentation only uses sparse annotations: Unified weakly and semi-supervised learning in medical images. Medical Image Analysis, 80, 102515 (2022).
Huynh, T., Nibali, A., & He, Z. Semi-supervised learning for medical image classification using imbalanced training data. Computer Methods and Programs in Biomedicine, 216, 106628 (2022).
Yang, X., Tian, J., Wan, Y., Chen, M., Chen, L., & Chen, J. Semi-supervised medical image segmentation via cross-guidance and feature-level consistency dual regularization schemes. Medical Physics, In press (2023).
Wang, K., Zhan, B., Zu, C., Wu, X., Zhou, J., Zhou, L., & Wang, Y. Semi-supervised medical image segmentation via a tripled-uncertainty guided mean teacher model with contrastive learning. Medical Image Analysis, 79, 102447 (2022).
Luo, X., Hu, M., Song, T., Wang, G., & Zhang, S. Semi-supervised medical image segmentation via cross teaching between CNN and transformer. In: International Conference on Medical Imaging with Deep Learning, PMLR, 2022, pp. 820–833.
Joubbi, S., Ciano, G., Cardamone, D., Maccari, G., & Medini, D. CrossCT: CNN and transformer cross–teaching for multimodal image cell segmentation. In: Competitions in Neural Information Processing Systems. PMLR, 2023, pp. 1–14.
Li, K., Zhang, G., Li, K., Li, J., Wang, J., & Yang, Y. Dual cnn cross-teaching semi-supervised segmentation network with multi-kernels and global contrastive loss in ACDC. Medical & Biological Engineering & Computing (2023), 1–9.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
Li, Q., Zhong, R., Du, X., & Du, Y. Transunetcd: A hybrid transformer network for change detection in optical remote-sensing images. IEEE Transactions on Geoscience and Remote Sensing, 60 (2022), 1–19.
Xiao, Z., Su, Y., Deng, Z., & Zhang, W. Efficient combination of cnn and transformer for dual-teacher uncertainty-guided semi-supervised medical image segmentation. Computer Methods and Programs in Biomedicine, 226 (2022), 107099.
Zhao, Z. A., Feng, X. F., Ren, X. Q., & Dong, Y. Y. Uncertainty-guided cross learning via cnn and transformer for semi-supervised honeycomb lung lesion segmentation. Physics in Medicine and Biology, In press (2023).
Zhang, S., Zhang, J., Tian, B., Lukasiewicz, T., & Xu, Z. Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation. Medical Image Analysis, 83 (2023), 102656.
Yang, X., Tian, J., Wan, Y., Chen, M., Chen, L., & Chen, J. Semi-supervised medical image segmentation via cross-guidance and feature-level consistency dual regularization schemes. Medical Physics, In press (2023).
Gu, X. A self-training hierarchical prototype-based approach for semi-supervised classification. Information Sciences, 535 (2020), 204–224.
Lee, D. H. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML, Vol. 3, 2013, p. 896.
Sajjadi, M., Javanmardi, M., & Tasdizen, T. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In: Advances in Neural Information Processing Systems, Vol. 29, 2016.
Laine, S., & Aila, T. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 (2016).
Tarvainen, A., & Valpola, H. Mean teachers are better role models: Weight averaged consistency targets improve semi-supervised deep learning results. In: Advances in Neural Information Processing Systems, Vol. 30, 2017.
Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., & Sugiyama, M. Coteaching: Robust training of deep neural networks with extremely noisy labels. In: Advances in Neural Information Processing Systems, Vol. 31, 2018.
Qiao, S., Shen, W., Zhang, Z., Wang, B., & Yuille, A. Deep co-training for semi-supervised image recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 135–152.
Kingma, D. P., Mohamed, S., Jimenez Rezende, D., & Welling, M. Semi-supervised learning with deep generative models. In: Advances in Neural Information Processing Systems, Vol. 27, 2014.
Duan, J., Luo, B., & Zeng, J. Semi-supervised learning with generative model for sentiment classification of stock messages. Expert Systems with Applications, 158, 113540 (2020).
Li, D., Yang, J., Kreis, K., Torralba, A., & Fidler, S. Semantic segmentation with generative models: Semi-supervised learning and strong out of domain generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8300–8311.
Chen, X., Yuan, Y., Zeng, G., & Wang, J. Semi-supervised semantic segmentation with cross pseudo supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2613–2622.
Chua, J., Tan, B., Ke, M., Schwarzhans, F., Vass, C., Wong, D., Nongpiur, M. E., Chua, M. C. W., Yao, X., Cheng, C. Y., Aung, T., & Schmetterer, L. Diagnostic ability of individual macular layers by spectral-domain OCT in different stages of glaucoma. Ophthalmology Glaucoma, 3(5), 314–326 (2020).
Ronneberger, O., Fischer, P., & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III. Springer International Publishing, 2015, pp. 234–241.
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M. Swin-Unet: Unet-like pure transformer for medical image segmentation. In: European Conference on Computer Vision. Springer Nature Switzerland, Cham, 2022, pp. 205–218.

No competing interests reported.

Download PDF

Reviews received at journal
02 Aug, 2024
Reviewers agreed at journal
23 Jul, 2024
Reviewers agreed at journal
23 Jul, 2024
Reviewers invited by journal
23 Jul, 2024
Editor assigned by journal
22 Jul, 2024
Editor invited by journal
11 Jul, 2024
Submission checks completed at journal
09 Jul, 2024
First submitted to journal
08 Jul, 2024

You are reading this latest preprint version

Enhancing Volumetric Segmentation in Wide-Field OCT Images with a Semi- Supervised Learning Framework: Cross-Teaching CNN and Transformer Integration

Status:

Version 1

Abstract

Figures

1. Introduction

2. Related Works on Semi-Supervised Learning

3. Results

3.1. Data

3.2. Experimental results

4. Discussion

5. Methodology

5.1. Pre-processing

5.2. Semi-supervised Learning Methodology

Declarations

Competing interests

Supplementary Materials

Author Contribution

Acknowledgements

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1

where,	\(\:{L}_{ct}\:=\:{L}_{ct1}\:+\:{L}_{ct2}\)	(9)
	\(\:{L}_{ct1}=\:Dice({Pˆ}_{i}^{c}\:,{Pˆ}_{i}^{t})\)	(10)
	\(\:{L}_{ct2}\:=\:Dice\left({{Pˆ}_{i}^{t}\:,\:Pˆ}_{i}^{c}\right)\)	(11)