PAFormer: Pyramid Attention Transformer for Polyp Segmentation

doi:10.21203/rs.3.rs-2215007/v1

Download PDF

Research Article

PAFormer: Pyramid Attention Transformer for Polyp Segmentation

https://doi.org/10.21203/rs.3.rs-2215007/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Adenomatous polyps are usually the most common tumor in clinical screening for colorectal cancer. Early detection and removal of these precursor lesions have been proved to be effective in preventing many cancers and reducing the mortality rate. Therefore, intelligent polyp detection is an urgent matter, which could assist clinicians to quickly identify detected polyps at an early stage. This paper proposes a polyp segmentation method based on sequential self-attentive networks with the features such as the color and texture of images and the relationship features between them. The color and texture features of the region image are extracted and fused with the original image. The resulting feature maps are serially input to the Transformer model to learn high-level spatial and attentional features. The loss of spatial information is reduced by multiscale local aggregation. By aggregating the models from a global perspective, the transformer model can further reduce the loss of context information caused by deep convolution. Controlled experiments between the PAFormer network and the mainstream polyp segmentation networks are conducted on five datasets, including ColonDB, ETIS, CVC-ColonDB, CVC-612 and Kvasir. The results show that the segmentation accuracy of PAFormer has great improvement on the datasets and PAFormer is superior to existing mainstream baseline networks.

Polyp segmentation

Transformer

Attention

Colorectal cancer

Colorectal cancer (CRC) has been a challenging clinical problem in health and it is the third leading cause of cancer-related deaths in humans [1]. Notably, in 2012, CRC accounted for more than 1.36 million cancer cases, and was the third most common cancer worldwide, accounting for 10% of the total number of men and 9.2% of women. When CRC is diagnosed and detected at an early stage, patient survival rates can reach nearly 90%, while advanced disease drops to less than 7% [2]. Adenomatous polyps are usually the most common tumor in clinical screening for cancer. Early detection and removal of these precancerous lesions have been shown to prevent many cancers and reduce mortality rate effectively [3]. It is widely believed that more than 95% of colorectal cancers originate from adenomatous polyps. Polyps detected at an early stage are usually defined as benign lesions with dysplastic epithelium and they have the potential for malignant transformation [4]. In conventional polyp detection methods, clinicians require a high degree of hand-eye coordination. However, nearly 25% of polyps are still missed during video examinations. Therefore, it is vital to use computer-aided diagnostic techniques to help clinicians detect polyps that are easily missed by the human eye [5].

Existing technical defects: Traditional image segmentation methods include threshold segmentation methods, edge-based segmentation methods, region-based segmentation methods, etc. Most of these methods are based on manual feature extraction of images, such as color and texture information. However, the artificially designed features are often shallow, causing significant limitations and little room for performance improvement in the traditional methods [6]. Compared with traditional methods, semantic segmentation based on a framework of deep learning has significant advantages in terms of accuracy and efficiency. Although better networks have been designed for semantic segmentation, the results are still not applicable to all types of images, the diversity of images makes the amount of training data to be prepared very large, and the interference between the various categories is carried out. These issues reduce the accuracy of pixel prediction. At the same time, as the layers of the neural network deepen, the edge information of the image will be lost seriously. These factors seriously affect the effect of image segmentation [7].

The traditional image segmentation methods for colon polyps are based on the characteristics of human vision. The features are more sensitive and extracted with discriminative power. The features extracted by traditional image processing methods have specific physical meanings in each dimension, including color, texture information, location, etc[8]. In recent years, existing deep learning techniques have performed better in polyp detection than traditional methods, extracting polyp regions more comprehensively [12]. Deep neural networks learn features of high-dimensional images based on broader and deeper network hierarchies. In the public polyp segmentation dataset, there is still a lot of room for improvement in accuracy.

This paper proposes a Pyramid Attention Transformer (PAFormer) network based on the fusion of resnet50 and the Transformer encoder. The Transformer model based on the attention mechanism has achieved good results in the fields of machine translation and natural language processing (NLP) [9]. Currently, there are many works in the computer version (CV) field using Transformer for end-to-end target detection and image classification [10, 11]. We chose Transformer's encoder module with resnet50 as the backbone. The segmentation accuracy is improved on the common dataset by extracting and fusing semantic information from different layers.

The contributions of this paper are as follows.

1. In this paper, the PAFormer module was proposed, which was integrated with the encoding and decoding modules in segmentation networks and formally extracted high-level semantic features of images through global aggregation in an incremental pyramidal structure.

2. The SEAP (SEattention and Atrous Spatial Pyramid Pooling) module was proposed to reduce the spatial location information lost due to deep convolution by local aggregation and focus more on edge detection.

3. Through extensive ablation experiments, this paper demonstrates that the proposed network (PAFormer) achieves better results on all existing public datasets and is significantly competitive with existing methods.

Polyp Segmentation

In recent years, the application of deep learning techniques to the automatic measurement and classification of gastrointestinal diseases in colonoscopy videos have received much attention, especially for the detection of colon polyps [13]. Deep learning techniques are used to retrieve polyps in images automatically, and label them to assist physicians in diagnosis and treatment eventually [12]. The performance of research based on deep learning techniques has surpassed the expertise of general endoscopists.

In the deep learning-based image segmentation methods for colon polyps, convolutional neural networks are typical methods [14]. Complex features are extracted from the original image by convolution and pooling methods, and then the extracted features are fed to the subsequent classifier. The U-Net [15] network does not have any fully connected layers, and only the effective part of each convolution is used to obtain the complete context of the pixels in the input image, i.e., the segmentation map contains only pixels. U-Net++ [16] and ResUNet++ [17] are used for polyp segmentation with good performance. These methods mainly segment the whole region of polyps, but ignore the region boundary constraint which is the key to improving the segmentation performance. [18] proposed a 16-layer semantic segmentation SegNet network to improve the accuracy of polyp segmentation and the robustness of the network. [19] used convolutional neural networks to learn features at different scales, and then used a fully connected network. [20] studied intestinal polyp segmentation using DeepLabv3 combined with a long and short-term memory network. The segmentation effect was improved compared with the existing DeepLabv3 model, but the overall accuracy was still low.

PraNet [21] proposes a novel deep neural network for polyp segmentation that uses parallel partial decoders to aggregate features at a high level. The merged features use contextual information, and produce a global graph that serves as an initial guide for the regions in the subsequent steps. In particular, a backward attention mechanism is incorporated in the last three stages of ResNet50 [26] to enhance training at the boundaries. However, for application scenarios like polyp segmentation, the training at the edges and the analysis of higher-level semantics can be enhanced further. In addition, more attention should be paid to the texture and size features of the underlying semantics.

Transformer in Vision

Transformer was first applied in the field of NLP and belonged to a deep neural network mainly based on a self-attentive mechanism [9, 22]. It is used as an encoder-decoder architecture. The encoder is mainly responsible for generating the context of the source image and the representation of each pixel channel by continuously deepening the network. The decoder generates a pixel-intensity output image automatically, with one channel per pixel at each time step. The decoder generates the pixel-intensity output image in an autoregressive manner, with one channel per pixel. Due to its powerful representation capabilities, it performs semantic analysis of each word that corresponds to the analysis of each pixel in computer vision tasks. In various vision benchmarks, transformer-based models perform similarly or better than other types of networks, such as convolutional and recurrent neural networks.

Visual transformers (ViT) [11] are proposed to segment images into patches and use a linear sequence of embeddings of these patches as input to the Transformer. Segmentation into patches and using linear embedding sequences of these patches as input to the Transformer. SEgmentation TRansformer (SETR) [23] proposes a pure transformer applied to the field of semantic segmentation. This is the first architecture that departs from FCN [24] encoding and decoding completely. Instead of relying on convolutional neural networks to extract features, it abandons convolution operation altogether. This sequence-sequence prediction model always maintains the same spatial resolution. [25] proposed a novel learning framework called Hierarchical Aggregation Transformer (HAT), which aggregated features at multiple scales on top of the transformer and mined features with discriminative power. In image recognition, pure transformers are as effective as CNN-based feature extraction methods. Multiple attention modules were introduced to eliminate convolution and subsampling operators.

This study used the convolutional neural network as the encoder of the segmentation network, and used the stacked Transformer pyramid to aggregate the global information as the decoder. Combining the advantages of a convolutional neural network and Transformer, the contextual information of each stage was used to obtain a high-precision polyp segmentation network. The encoder was mainly used to identify and capture the contextual information in the input, and the network could encode higher-level semantic features of the image by continuously reducing the spatial size. The decoder used the encoded higher-level features and the detailed features conducted through the hop-over connections to decode into the semantic labels required for training. During the decoding process, the spatial information of the same spatial size class conducted from the jump connection can help the decoder precisely locate the part of the image that needs to be segmented out [27].

Overall architecture

An encoder-decoder structure is used to improve the segmentation accuracy, as well as the combination of convolutional layers and multiple scales. As shown in the Fig. 1, the convolution kernel performs layer-by-layer convolution and down-sampling of the feature map during the encoding process. The down-sampling generates contextual information in the feature map for each pixel of the input image. Such multilayer down-sampling allows for more translation invariance, making the classification more robust. However, there is some loss of spatial resolution on the feature map. Therefore, the feature maps of the coded network need to be captured, and the boundary information needs to be stored before down-sampling. Using the maximum pooling window, the location of the maximum eigenvalue of each encoder feature map can be stored. The encoder part connects the linguistic information at different depths across layers through the SEAP (SE-attention and Atrous Spatial Pyramid Pooling). To some extent, the semantic location information lost because deep convolution is maintained. The decoder up-samples the input feature maps by the index value of the maximum pooling window, stored in the feature maps of the encoded network, to obtain the sparse feature maps. The higher resolution is preserved in the second half of the network, and then the respective extracted modules are used to obtain the target information at different scales. The final segmentation map is output by directly multiplying the up-sampling back.

For the Segmentation task, a low-level feature map is also needed to provide edge feature information, so the output of the Encoder module is first up-sampling, and then a feature map with consistent resolution (like the jump layer structure) is output from the network process of ResNet. The feature maps go through the convolutional layer to expand the number of channels to make it consistent with the output of the Encoder. The feature maps of the same channel are aggregated by the multi-level aggregations (MA) module, and then the predicted masks are obtained by upsampling.

Transformer attention

We characterize the high-level semantic information as a number pyramid of stacked transformers to decode it. The self-attentive mechanism of the Transformer is able to mine global dependencies with a global perceptual field, which has achieved good results on many visual tasks. The transformer encoder consists of successive encoder layers. Each layer includes a multi-head self-attentive MSA and Feed Forward Network (FFN) [9, 40, 41]. The FFN contains a two-layer multilayer perceptron and a GELU activation function. Since the Transformer is an alignment-invariant sequence-to-sequence model, it naturally fuses the features of the template branch and the search branch by splicing. We add sinusoidal positional encoding to the input sequence of each attention layer to ensure that the model knows where to locate and branch feature markers during the computation of attention. The dependency between the template and search region features in the total feature representation are captured by the self-attention mechanism in the encoder layer. The global information is used to fuse and enhance the discriminative nature of the features effectively to help the network locate the target location more accurately.

As shown in Fig. 2, the transformer attention function can be explained by query(Q), keys (K), and values(V), where the query and keys are mapped pairs to an output. The output is calculated as the weighted sum of values. V is the vector representing the input features, and Q and K are the feature vectors for the Attention weight calculation. They are all obtained from the input features. Attention (Q, K, V) is used to multiply V by the corresponding weight according to the degree of attention [41]. The matrix of outputs can be described as [41]:

$$Attention(Q,K,V)=softmax\left(\frac{Q{K}^{T}}{\sqrt{{d}_{k}}}\right)V$$

Where the input consists of queries and keys of dimension ${d}_{k}$.

SEAP

SEAP improves on the ASPP [33] structure and SE-attention [38, 39] by extracting image features using separable convolutional layers of different channels. The spatial information of features is captured by parallel atrous convolutions at different rates in a spatial pyramid structure, and channel compression is performed by 1×1 convolution, as shown in Fig. 3.

SEAP effectively solves the loss of spatial location information caused by convolution. The labeling of local patches usually depends on the global image context. The first branch is 1×1 convolution, which aims to maintain the original perceptual field; the second to fourth branches are depth-separable convolutions with different dilution rates, which aims at feature extraction to obtain different perceptual fields; the fifth branch is to pool the input on average to obtain global features. Finally, the feature maps of the five branches are stacked on the channel dimension and undergo 1×1 convolution to fuse the different scales information. The shape information of different layers is captured, and the convolution kernel of the lower layers captures the overall shape. In contrast, the detail information for object categories is encoded in the higher-level network, taking both category detail information and shape information into account.

The multilevel aggregations (MA) module is used to realize the aggregation of three SEAP modules, and the specific implementation method is as shown in Fig. 4:

Datasets

For a fair comparison, our test set and training set are the same as PraNet. The training set contains a total of 1450 images. Five different datasets were used for the test set, which are described in detail below. The ETIS [28] dataset contains 196 images of size 1225×966 from 34 colonoscopy videos. This dataset is more challenging because the colon polyps vary in size and shape. The CVC-ColonDB [29] dataset contains 380 images of size 500×574 from 15 different colonoscopy videos. The CVC-300 [30] dataset contains 912 images from 44 colonoscopy inspection videos. The CVC-612 [31] contains 612 images from 25 colonoscopy videos with an image size of 384 × 288. The Kvasir [32] dataset consists of 1000 polyp images and their annotations. The image sizes in this dataset are not uniform.

Evaluation metrics

To evaluate the segmentation performance of the network, the common metrics are used: metric 1 mean Iou, which measures the accuracy of object detection. Metric 2 mean Dice, which measures the similarity between samples [34, 35]. Furthermore, mean absolute error (MAE) is also added to measure, reflecting the actual prediction error accurately. In order to calculate the above metrics better, it is assumed that ${t}_{p}$ (True Positive) is positive and ${f}_{p}$ (False Positive) is negative. In order to calculate the above two indicators better, it can be assumed that is the vivacious; ${f}_{p}$ is the false positive; ${f}_{n}$ (False Negative) is the false negative; n is the number of samples. The mean Iou and mean Dice are calculated as follows.

$$Mean IoU=\frac{{t}_{p}}{{t}_{p}+{f}_{p}+{f}_{n}}$$

$$Mean Dice=\frac{2\times {t}_{p}}{2\times {t}_{p}+{f}_{p}+{f}_{n}}$$

Control experiments

To verify the segmentation performance of PAFormer on the five test sets, the segmentation results of U-Net [15], UNet++ [16], and PraNet [21] were compared, as shown in Fig. 5. The red box marks the blurred part of the mask boundary of other networks. As can be seen from row 4 of Fig. 5, PAFormer finds the location of polyps more accurately than other networks in the case of large-sized images. PAFormer segments polyps more thoroughly and accurately, and could handle polyps of different sizes more precisely. In addition, the last row in Fig. 5 shows that the PAFormer network can identify the boundary information more accurately in the challenging dataset and achieve more accurate localization, and the polyp can be located more accurately during segmentation. Tables 1,2,3,4 and 5 are the comparison of PAformer and the mainstream research polyp segmentation networks on five public datasets, which prove that PAFormer can grasp the feature details better.

Table 1
Datasets	Methods	Mean Dice	Mean IoU	MAE
CVC-ClinicDB	ResUNet-mod [36]	0.779	-	-
	ResUNet++	0.796	0.796	-
	U-Net	0.823	0.755	0.019
	U-Net++	0.794	0.729	0.022
	SFA [37]	0.700	0.607	0.042
	PraNet	0.899	0.849	0.009
	PAFormer(ours)	0.914	0.863	0.010

Table 2
Datasets	Methods	Mean Dice	Mean IoU	MAE
ETIS	U-Net	0.398	0.335	0.036
	U-Net++	0.401	0.344	0.035
	SFA	0.279	0.217	0.109
	PraNet	0.628	0.567	0.031
	PAFormer(ours)	0.679	0.588	0.015

Table 3
Datasets	Methods	Mean Dice	Mean IoU	MAE
CVC-ColonDB	U-Net	0.512	0.444	0.061
	U-Net++	0.483	0.410	0.064
	SFA	0.469	0.347	0.094
	PraNet	0.709	0.640	0.045
	PAFormer(ours)	0.720	0.654	0.037

Table 4
Datasets	Methods	Mean Dice	Mean IoU	MAE
Kvasir	ResUNet-mod	0.791	-	-
	ResUNet++	0.813	0.793	-
	U-Net	0.823	0.746	0.055
	U-Net++	0.821	0.743	0.048
	SFA	0.723	0.611	0.075
	PraNet	0.898	0.840	0.030
	PAFormer(ours)	0.921	0.856	0.026

Table 5
Datasets	Methods	Mean Dice	Mean IoU	MAE
CVC-300	U-Net	0.710	0.623	0.031
	U-Net++	0.707	0.624	0.022
	SFA	0.467	0.329	0.018
	PraNet	0.871	0.797	0.031
	PAFormer(ours)	0.892	0.806	0.008

Ablation study

To further explore the effectiveness of the Transformer, SEAP module of the network, a series of ablation experiments were performed on the ColonDB and Kvasir datasets.

Table 6
Datasets	Methods	Mean Dice	Mean IoU
CVC-300	Backbone	0.726	0.631
	Backbone + SEAP	0.864	0.786
	Backbone + Transformer	0.873	0.797
	Backbone + SEAP + Transformer	0.892	0.806

Table 7
Datasets	Methods	Mean Dice	Mean IoU
Kvasir	Backbone	0.747	0.668
	Backbone + SEAP	0.874	0.792
	Backbone + Transformer	0.890	0.811
	Backbone + SEAP + Transformer	0.921	0.856

As shown in Table 6 and 7, the second row of data in the table show the ability of SEAP to characterize network performance. By extracting features of different scales, SEAP enables the network to have different receptive fields, thereby improving network performance. From the last row of data in the table, it can be seen that the decoder makes the network better represent the global features through the transformer attention mechanism, and the segmentation performance of the network is thus improved to some extent.

Semantic segmentation is dedicated to the correct classification of each pixel in an image. The two main factors affecting performance are spatial information and semantic information. In general, the feature maps with high resolution correspond to rich spatial details. At the same time, they have too much computation. On the contrary, feature maps with robust semantic information usually have a small resolution, but it lacks enough spatial details to accurately locate the target. Usually, they restrict each other.

The PAFormer was proposed by the Transformer module to model the features of each pixel. It obtains global information to make the segmentation results more consistent in semantics. In addition, the SEAP module extracts contextual information at different scales through multiple parallel null convolution branches with different null rates. Moreover, it improves feature representation by incorporating global context fusion. The SE-attention mechanism allows dynamic access to multi-scale contextual information, which improves the target boundary localization problem. The cross-layer connection of encoding and decoding networks is combined with low-level feature mapping at decoding time to capture global information and recover clear target boundary simultaneously.

However, it is necessary to clarify the limitations and the direction for further research for the PAFormer. Firstly, there is still room for improvement for datasets with more difficult cases, such as ETIS. In the future, we will focus on improving the performance of the network for difficult case segmentations. On the premise of ensuring the accuracy of the network, it is mainly to improve the generalization performance of the network. Besides, we will gradually improve our network structure, optimize parameters, and reduce the time and space of network operation. Through optimizing the network, we can make our network has more practical applications in clinical.

Despite the limitations of our network, the network proposed in this paper has greatly improved in the mainstream polyp segmentation methods. Further research can assist clinicians with the recognition rate of colonic polyps and reduce the misdiagnosis rate of missed detection. It also accelerates the detection time and treatment speed for patients with polyps, which has practical research significance and application value for clinicians and patients.

Ethical Approval and Consent to participate: Not applicable

Human Ethics：Given the purely observational nature of the study, written consent was not required.

Consent for publication: This retrospective study, which has been ethically reviewed, is exempt from informed consent for publication.

Availability of supporting data：The datasets generated or analyzed during this study are available from the corresponding author on reasonable request.

Competing interests：The authors declare that they have no conflict of interests.

Funding: The work was supported by grants from the Natural Science Foundation of Chongqing (No. cstc2020jcyj-msxmX0641), the Fundamental Research Funds for the Central Universities (No. 2020CDJ-LHZZ-025), and the Natural Science Foundation of China (No. 11902057)

Authors' contributions：Zhiyong Huang proposed the article novel idea and the article main frame, has the direct contribution; Fang Xie was responsible for detail writing and data collection; WenCheng Qing, MengYao Wang, and Man Liu provided the guidance of the analysis about deep learning models in article; Daming Sun was in charge of revising the grammar of the article and improving its content.

Acknowledgments: Not applicable

Authors’ information：Zhiyong Huang, Fang Xie, WenCheng Qing, MengYao Wang, and Man Liu are with School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044,China (email: [email protected], [email protected],[email protected],[email protected],[email protected]); Daming Sun is with Chongqing Engineering Research Center of Medical Electronics and Information Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China (email: [email protected] );

Brandao P, Mazomenos E, Ciuti G, et al. Fully convolutional neural networks for polyp segmentation in colonoscopy[C]//Medical Imaging 2017: Computer-Aided Diagnosis. SPIE, 2017, 10134: 101–107.
Frontiers of robotic endoscopic capsules: a review[J]. Journal of Micro-Bio Robotics, 2016, 11(1):1–18.
Lieberman D A, Rex D K, Winawer S J, et al. Guidelines for colonoscopy surveillance after screening and polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer[J]. Gastroenterology, 2012, 143(3): 844–857.
Corley D A, Jensen C D, Marks A R, et al. Adenoma detection rate and risk of colorectal cancer and death[J]. New england journal of medicine, 2014, 370(14): 1298–1306.
Leufkens A M, Van Oijen M G H, Vleggaar F P, et al. Factors influencing the miss rate of polyps in a back-to-back colonoscopy study[J]. Endoscopy, 2012, 44(05): 470–475.
S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz and D. Terzopoulos, "Image Segmentation Using Deep Learning: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3059968.
Brandao P, Mazomenos E, Ciuti G, et al. Fully convolutional neural networks for polyp segmentation in colonoscopy[C]//Medical Imaging 2017: Computer-Aided Diagnosis. SPIE, 2017, 10134: 101–107.
Mamonov, A.V., Figueiredo, I.N., Figueiredo, P.N., Tsai, Y.H.R.: Automated polyp detection in colon capsule endoscopy. IEEE TMI 33(7), 1488–1502 (2014)
Cambria, Erik, and Bebo White. "Jumping NLP curves: A review of natural language processing research." IEEE Computational intelligence magazine 9.2 (2014): 48–57.
Parmar, Niki, et al. "Image transformer." International Conference on Machine Learning. PMLR, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929 [cs.CV]
hang, J., et al.: UC-Net: uncertainty inspired RGB-D saliency detection via conditional variational autoencoders. In: IEEE CVPR (2020)
Wang, P., Xiao, X., Glissen Brown, J. R., Berzin, T. M., Tu, M., Xiong, F., … Liu, X. (2018). Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nature biomedical engineering, 2(10), 741–748.
Liu C, Xiao Z Y, Du N M. Application of improved convolutional neural network in medical image segmentation[J]. Journal of Frontiers of Computer Science and Technology,2019,13(9): 1593–1603.
Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241). Springer, Cham.
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018). Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support (pp. 3–11). Springer, Cham.
Jha, D., Smedsrud, P. H., Riegler, M. A., Johansen, D., De Lange, T., Halvorsen, P., & Johansen, H. D. (2019, December). Resunet++: An advanced architecture for medical image segmentation. In 2019 IEEE International Symposium on Multimedia (ISM) (pp. 225–2255). IEEE.
Wang P, Xiao X, Brown JR G, et al. Development and validation of a deep-learning algorithm for the detection of polypsduring colonoscopy[J]. Nature Biomedical Engineering,2018,2(10):741–748.
Xiao W T,Chang L J, Liu W M. Semantic segmentation of colorectal polyps with DeepLab and LSTM networks[C]/Proceedings of the 2018 IEEE International Conference on Consumer Electronics, Taiwan, China, May 19–21, 2018.Piscataway: IEEE, 2018: 1–2.
Elsken T, Metzen J H, Hutter F. Neural architecture search: a survey[J]. Journal of Machine Learning Research,2019,20(55):1–21.
Fan, D. P., Ji, G. P., Zhou, T., Chen, G., Fu, H., Shen, J., & Shao, L. (2020, October). Pranet: Parallel reverse attention network for polyp segmentation. In International conference on medical image computing and computer-assisted intervention (pp. 263–273). Springer, Cham.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., … Rush, A. M. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., … Zhang, L. (2021). Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 6881–6890).
Dai, J., Li, Y., He, K., & Sun, J. (2016). R-fcn: Object detection via region-based fully convolutional networks. Advances in neural information processing systems, 29.
Zhang, G., Zhang, P., Qi, J., & Lu, H. (2021, October). Hat: Hierarchical aggregation transformers for person re-identification. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 516–525).
Wu, Z., Shen, C., & Van Den Hengel, A. (2019). Wider or deeper: Revisiting the resnet model for visual recognition. Pattern Recognition, 90, 119–133.
Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).
Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. 2013. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery 9, 2 (sep 2013), 283–293. https://doi.org/10.1007/s11548-013-0926-3
J. Bernal, J. Sánchez, and F. Vilariño. 2012. Towards automatic polyp detection with a polyp appearance model. Pattern Recognition 45, 9 (2012), 3166–3182. https://doi.org/10.1016/j.patcog.2012.03.002 Best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA’2011).
David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Antonio M López, Adriana Romero, Michal Drozdzal, and Aaron Courville. 2017. A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering 2017 (2017).
Jorge Bernal, F. Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. 2015. WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43 (2015), 99–111. https://doi.org/10.1016/j.compmedimag.2015.02.007
Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. 2020. Kvasir-seg: A segmented polyp dataset. In International Conference on Multimedia Modeling. Springer, 451–462
Wu, H., Zhang, J., Huang, K., Liang, K., & Yu, Y. (2019). Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation. arXiv preprint arXiv:1903.11816.
Jha, D., et al.: Kvasir-SEG: a segmented polyp dataset. In: Ro, Y.M., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 451–462. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2 37
Jia, X., Xing, X., Yuan, Y., Xing, L., Meng, M.Q.H.: Wireless capsule endoscopy: a new tool for cancer screening in the colon with deep-learning-based polyp recognition. Proc. IEEE 108(1), 178–197 (2019)
Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual U-net. IEEE Geosci. Remote Sens. Lett. 15(5), 749–753 (2018)
Fang, Y., Chen, C., Yuan, Y., Tong, K.: Selective feature aggregation network with area-boundary constraints for polyp segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 302–310. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7 34
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, pages 7132–7141, 2018.
Hou, Qibin, Daquan Zhou, and Jiashi Feng. "Coordinate attention for efficient mobile network design." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097–1105.
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

PAFormer: Pyramid Attention Transformer for Polyp Segmentation

Status:

Version 1

Abstract

Figures

Introduction

Related Work

Polyp Segmentation

Transformer in Vision

Methods

Overall architecture

Transformer attention

SEAP

Results

Datasets

Evaluation metrics

Control experiments

Ablation study

Discussion And Conclusions

Declarations

References

Additional Declarations

Status:

Version 1