Segmentation of Pulmonary Nodules Using SegNet and Adversarial Networks

doi:10.21203/rs.3.rs-1658720/v1

Download PDF

Research Article

Segmentation of Pulmonary Nodules Using SegNet and Adversarial Networks

https://doi.org/10.21203/rs.3.rs-1658720/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Lung cancer is seen as one of the most common lung diseases in common people. For the patients having symptoms, the presence of lung nodule is checked by using various imaging techniques. Pulmonary nodules are detected in most of the cases having symptoms. But identifying the type of the nodule and the categorization still remains as a challenge. After confirming the presence of nodule (benign or malignant) it takes other several steps to identify its characteristics. Improved imaging methods produce results within a short span of time. Research works are going on to increase the overall efficiency of the system. The proposed system consider authentic data source for the study. The benign and malignant samples are considered for the generation of realistic large image set. The generation of large data set with the help of generative adversarial network (GAN) is the first part of the work. The generated images by GAN can’t be differentiated from the original images even by a trained radiologist. This proves the importance of images generated by GAN. GAN is able to generate 1024x1024 resolution for natural images. Real data images are used to fine tune the SegNet output. Through transfer learning these weights are transferred to the system for segmentation of the images. The training process use real and generated images, which improve efficiency of the network. The original data from LUNA 16 is used to further generate benign and malignant samples using GAN. A total of 440 images and its augmented images are used for training GAN and generated 100,1000 images. Hence the overall efficiency of the system gets improved. To verify the same various combinations and method are considered and tabulated various parameters. Methods with SegNet, GAN and other combinations are evaluated to verify the efficiency of the system. Receiver operating characteristics also plotted and compared the area under the curve for verification of the result.

Results

Proposed method having GAN along with image augmentation is compared with other methods. GAN with SegNet shows very small deviation from the proposed method. This clearly shows the advantage of using GAN for generating images for nodule segmentation. The method is also compared with other Encoder-Decoder architectures. The performance comparison is tabulated. Finally the receiver operating characteristic (ROC) curve of SegNet while training the model with GAN and without GAN is also investigated. The Area Under the Curve (AUC) for SegNet with GAN is found to be 0.851 while that of SegNet is 0.76. The proposed method stood ahead in all respect, considered.

Conclusion

The proposed work with the help of available data set manages to generate sufficient amounts of data with the help of GAN. Perfect samples are considered here for the generation of data. Hence the data generated assures the quality and that is used for training the network. For training the GAN, 440 images and augmented images are used. 100,1000 images generated and are used for training the SegNet architecture. The segmentation shows improved accuracy in the case of benign as it has a perfect shape compared to malignant samples. Through the evaluation of the parameters it is clear that the GAN along with image augmentation shows better performance.

Pulmonary Nodule

SegNet

Lung Cancer

Adversarial Network

Transfer Learning

The characteristic of cancerous cells is their growth in the tissues in an uncontrolled manner. It is a serious condition which is to be treated in no time to avoid further spread. The most common reason for lung cancer is the practice of tobacco smoking. There are also some other reasons like passive smoking, air pollution, asbestos etc. With the help of computer tomography and radiography the presence of lung cancer is identified. Through biopsy the presence of Cancer is confirmed. For giving proper medication in an effective manner to a cancer patient it is necessary to identify the disease at an early stage. But it is very difficult to identify the same from CT images even by very experienced professionals. So it is very essential to find out efficient methods to identify the disease. Various deep learning models are showing promising results and many more researches are on the way. But the difficult part here is the collection of data needed for research purposes. Unavailability of reliable data sources and lack of data sources with proper labels make research in this area very difficult. Here in the proposed paper the required data is generated with the help of a generative adversarial network. With the help of encoder and decoder network available in semantic segmentation model the lung nodule classification is done.

Detection and segmentation of lung images are aggressive research areas where many promising results are received. The research results are very much helpful in determining the medicines and to study its effectiveness [1], [2]. Textural analyses of medical images are also much helpful in identifying the tissues abnormalities. Accuracy of the results holds a very important role in the recovery of the patient. Best observations result in prescribing the best medicines.

Segmentation can be done manually, but the chances to get inaccurate results are much higher. Semi automated segmentation is also available. It is error free when compared with the former one. For fast and accurate results it is desirable to use an automatic segmentation process [3]–[8]. There are so many deep learning experiments and research going on. A drastic development has occurred in the area of classification segmentation and detection of nodules. Automatic nodule detection is an advanced method that can be used to identify various tissues of abnormalities or nodules. It can be applied to various organs such as lung, liver, brain, bones etc. Recent researches declare that they provide efficient and accurate results in the area of segmentation.

Deep learning problems need a huge data set with accurate and reliable data. Sufficient amounts of reliable data images are necessary to obtain accurate results but lack of sufficiently large data sets, high cost to get data from sources and the issues dealing with personal information create difficulties in obtaining the same.

As a solution to this problem many methods are developed for availing large amounts of data. True transferring weight of a pre-trained model is one of the popular methods for achieving the same to a certain extent [6], [7]. Another method is data augmentation. Data augmentation is the process of increasing the amount of data available by adding copies of the available data with slight modifications [8]. Using generative adversarial networks it is possible to generate data artificially. Generative adversarial networks can generate high quality images that can be used instead of real data. In the proposed model GAN [9] is used to generate CT images of lungs. Various studies related to this field use the image generation technique. CT images generated from the available authentic data resources are used for various applications. [10]- [15]

To achieve quality data images GAN is used. Authentic and accurate data samples are used here to generate the needed database. The fine tuned architecture, using real data samples make the proposed system more stable and efficient. The methods employed and other specifications are included in the following sections in detail.

Method

The data set consists of images that are fed into the GAN network. The network which is adversarial in nature processes the given data. The generator network generates realistic images from the available original data. Simultaneously the discriminator network learns from the real and generated images. The learning will result in a conclusion having a unique result, termed as prediction. Throughout the learning process different parameters are updated and fine tuned. The final segmented results are generated from the Seg Net transferring the weights.

The proposed system shown in Fig. 1 generates images using a generative adversarial network. After that the generated images are used to learn a SegNet, and the parameters are fine tuned by using real data. Through parameter transfer learning, the SegNet is updated for image segmentation. The results are verified by using a test data set.

Augmented image generation using GAN

Instead of traditional square patches we used horizontal and vertical patches. This will keep global and local appearance features in the patches. A dimension of 16x64 and 64x16 is kept for horizontal and vertical patches respectively. The mean intensity and variance of all the patches in the training set are extracted. All the patches are then normalized with zero mean and unit variance before fed into the auto encoder for training.

The idea can be implemented by using a deep learning based generative model. Generative Adversarial Networks (GANs) are considered here for improving system performance by training the system intelligently. The generative adversarial network consists of two opposing networks, one is generative and the other is discriminative in nature. These two networks race to reach the learning target for obtaining optimised results. These networks can be used to generate any distribution of data through training. Random inputs are used here to generate the initial set of images. The discriminator discriminates the images and assists the generator to reduce the discrimination. Through the feedback, the generator modifies the parameters and tries to generate better outputs. This can be achieved by updating loss functions. Generator and discriminator loss are updated throughout the course of process, and thereby achieve resultant images that highly resemble in nature. Thus the network provides data with high quality for the process. CT images can be generated for various applications using these generative models [10]-[15].

The expression for standard GAN loss function is shown here. It comprises loss function of discriminator and loss function of generator. Generator tries to improve the performance and thereby the loss function gets minimised. But the discriminator functions in the reverse direction by trying to maximise the function.

Loss function = Ex [ log ( D(x) ) ] + EZ [ log ( 1- D (G(z)) ) ] (1)

New realistic data can be generated with great accuracy using generative models. Here the probability distribution of the existing dataset is used to generate the new data set. Generative Adversarial Network comes under this category and is used here to generate realistic data. The performance can be improved by integrating multiple generators and discriminators within the system. The GAN can also be trained in a distributed fashion and then it can be presented as a Multi-Discriminator GAN.

In GAN as shown in the Fig. 2, random images are given as the input to the generator section. From the random inputs received, the generator tries to generate sample images. The purpose of the generator section is to generate images that are identical in nature with the real images. So from the random inputs, the generator generates real images like samples. These samples are fed into the discriminator. The discriminator receives real image samples also. The function of the discriminator is to discriminate generated images from the real images. The output of the discriminator section, known as loss functions, are generated and classified as generator loss function and discriminator loss function. The signal provided by the discriminator functions act as a feedback signal to the generator. This signal is used by the generator to update the weights. Through updating the weights of the generator, generates images that are more identical to the real image samples. The aim of the generator is to generate images that are very much identical to the real images. This can be achieved through the updating of weights periodically. Back propagation from the discriminator helps the generator to update the weights and reproduce images with better similarity.

SegNet

Semantic segmentation models in a broad sense comprise networks responsible for encoding and decoding. Here the encoder works as a classification network, which is pre-trained. The encoder is composed of different convolution layers. The feature extraction process is done by the encoder which is composed of convolution layers. Features are taken out of the input through the convolution process. The decoder decodes low resolution feature maps to full input resolution for the purpose of dense classification. The decoder is composed of different de-convolution layers. The aim of the de-convolution layer is up sampling. Through de-convolution the features that are reduced in dimensions are up sampled to image size.

Figure 3 shows the SegNet system flow from the input to the output. In SegNet a set of encoder layers encode the input images based on deep convolution. These images are decoded by corresponding decoder layers. Decoded images are forwarded to a classification layer. This layer acts as a pixel based classifier. Feature fusion layer combine the features that are collected from the patches. Input image sliced into patches in horizontal and vertical fashion. These are passed via the encoder and feature fusion layers. Based on the feature characteristics the decoder layers produce the output. The decoder output will be the lung nodule segmentation result as shown in the Fig. 3.

Transfer Learning Process

Accumulation of information is not happening in the case of traditional machine learning. Information that was learned previously is not considered in the case of the traditional learning process. Traditional learning is an isolated process and considered as a single task learning process. In transfer learning the learning process relies on information that is learned previously. Hence the transfer learning process can be done faster and require less amount of data for training.

Transfer learning starts from taking a previously learned network for some other process. The previously learned information is used here in the new training process which makes it faster and more accurate. The method of reusing a pre-trained model is considered here. The information gained via training can be used for improving the performance of another model by transfer learning process. In practice it is done by transferring the weights that are learned by the network. A problem can be solved easily by using the information learned from solving another problem which is related to the first one. Figure 4 explains the concept of traditional and transfer learning process.

The procedure followed throughout the investigation is described below.

A. Database

Many public databases are available with Data sets of various images of internal organs for research purposes. Lung Nodule Analysis is a data set widely known as LUNA16 with CT images of lungs. Another source is the Decathlon lung data set, having a lot of lung image data. NSCLC radio genomics is also helpful for this purpose. Data sets available with Luna16 are three dimensional CT images intended for lung nodule detection. Decathlon includes three dimensional CT images and the detail of segmentation with labels. Images without pre-processing, from the decathlon lung data set are used to prepare three sets of training data sets. The NSCLC radio genomics data set includes non-small cell lung cancer CT, PET/CT images. Images with segmentation labels can only be used as test images in this system.

B. Setup

LUNA16 lung database consist of 888 CT scans with slice thickness less than 2.5 mm. In LIDC/IDRI database the presence of nodule is categorized into three as no nodule, less than 3mm nodule, greater than 3mm nodule through annotation process. GAN can be used to generate images for lung nodule detection. The same method can be used in generating dataset for nodule segmentation. Due to the lack of labels of large true nodules, generation of data is not practical. So three dimensional CT images corresponding to small lung nodules are considered for the generation of lung images having cancerous lung nodules

C. Training and Optimization

Figure.5 shows some examples of benign and malignant nodules found in the lung tissues. The first row represents benign nodules while the second row shows the malignant nodules. All the images are taken from the LUNA16 lung database consisting of 888 CT scans. GAN is used for generating further images from the dataset.

A total of 100, 1000 data is generated for training the SegNet architecture for nodule segmentation. The real and generated images are considered for the training process. The details of training data for the GAN and SegNet are shown in Table.1.

Table 1

Training the GAN – Original and augmented images used for training
Training	Benign, Augmented	Malignant, Augmented
SegNet	450, 1280	430, 1148
GAN	225, 640	215, 574

The original data from LUNA 16 is used to further generate benign and malignant samples using GAN. A total of 440 images and its augmented images are used for training GAN and generated 100,1000 images.

D. Evaluation

There is also a significant difference in segmentation while using GAN based images. This is evident from the evaluation table in Table.2. We used mainly three parameters for evaluation namely, Dice Similarity Coefficient (DSC), Positive Predictive Value(PPV) and Sensitivity. DSC is a measure of overlap between ground truth and segmented result. Sensitivity shows how much positive proportions are measured correctly. Positive results are represented as a proportion as PPV. The parameters are evaluated from confusion matrix as follows:

DSC = 2TP / (FP + 2TP + FN) (2)

PPV = TP / (TP + FP) (3)

Sensitivity = TP / (TP + FN) (4)

Then the receiver operating characteristic (ROC) curve of SegNet, during training the model with GAN and without GAN is also evaluated. The area under curve is calculated with ROC curve.

Figure 6 shows the sample images generated by the generative adversarial networks. Benign nodules are more round in shape compared to malignant images generated with spicula and withdrawal into the pleura. The generated images by GAN can’t be differentiated from the original images even by a trained radiologist. This proves the importance of images generated by GAN.

Benign b. Malignant

Figure.6. Images generated by GAN (a. Benign, b. Malignant)

But the generated images are low resolution 64x64 in size. GAN is able to generate 1024x1024 resolutions for natural images. The improvement in GAN based architectures will improve the resolution of the medical images which will further enhance the accuracy in detection and segmentation of lung nodules.

Figure 7.a represents the SegNet segmentation on pulmonary nodules for benign cases along with its ground truth counterpart. Since the benign nodules are more round in shape more accurate segmentation is possible compared to Malignant case shown in Fig. 7.b. In Malignant case the nodules extend to the lung bones and it is difficult to differentiate the difference. This reduces the segmentation accuracy considerably compared to benign nodules.

Different methods and the performance results are tabulated in the Table.2. It is evident from the table that GAN along with image augmentation is clearly the winner. Simultaneously GAN with SegNet shows very small deviation from the winner. This clearly shows the advantage of using GAN for generating images for nodule segmentation. The method is also compared with other Encoder-Decoder architectures.

Table.2. Performance evaluation of SegNet + GAN

Method	DSC	PPV	Sensitivity
SegNet + GAN + Augmentation	0.87	0.86	0.89
SegNet + GAN	0.86	0.85	0.87
SegNet + Augmentation	0.80	0.79	0.82
SegNet	0.78	0.77	0.80

The performance comparison is given in Table 3. Finally the receiver operating characteristic ( ROC) curve of SegNet while training the model with GAN and without GAN is given in Figure.8. The Area Under the Curve (AUC) for SegNet + GAN is found to be 0.851 while that of SegNet is 0.76

Table 3

Performance comparison of Segnet with GAN with other Encoder-Decoder architectures
Method	DSC	PPV	Sensitivity
SgNet + GAN	0.87	0.86	0.89
U-Net + GAN	0.84	0.86	0.87
Autoencoder	0.75	0.74	0.77
Variational Autoencoder(VAE)	0.80	0.81	0.82

Different encoder - decoder combinations are also considered here and the comparison of parameters tabulated in Table 3. SegNet with GAN performed well in comparison with VAE, Auto encoder and U-Net. The proposed model outperformed the VAE scheme by 7% and U-Net with GAN by 2%.

Receiver operating curve also plotted here for validating the result. SegNet and SegNet with GAN plotted for comparing the AUC. The AUC of the proposed scheme outperformed the former.

I have the pleasure of sending you the manuscript entitled “Segmentation of Pulmonary Nodules Using SegNet and Adversarial Networks“ authored by Vinod C to be considered for publication as a research article in your prestigious journal BMC Medical Imaging. Paper is containing original research and has not been submitted / published earlier in any journal and is not being considered for publication elsewhere. All authors have seen and approved the manuscript and have contributed significantly for the paper.

Ethics approval and consent to participate

Ethics approval

The research meets all applicable standards with regard to the ethics of experimentation and all methods were carried out in accordance with relevant guidelines and regulations and research integrity, and the following is being certified/declared true.

As a research scholar and along with co-authors of concerned field, the paper has been submitted with full responsibility, following due ethical procedure, and there is no duplicate publication, fraud, plagiarism, or concerns about animal or human experimentation.

Consent to Participate

Not Applicable

Consent for Publication

Not applicable

Conflicts of Interest

The authors report no conflicts of interest. None of the authors of this paper has a financial or personal relationship with other people or organizations that could inappropriately influence or bias the content of the paper.

Availability of data and materials

The data acquired from a publically available data set

https://zenodo.org/record/3723295#.YpBzg2hBxEZ

The datasets and materials used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests

Funding

The authors declare no funding was accepted any public or private body for this research work

Authors Contributions

Vinod C designed the research study and provided research ideas, read the literature and collected medical records. Dr.D.Menaka was a major contributor in writing the manuscript. All authors read and approved the final manuscript.

Acknowledgements

Not applicable.

Author Details

Corresponding Author: Vinod C¹

Affiliation: Noorul Islam Centre for Higher Education

E-mail: [email protected]

Phone: 9947360452

Dr. D.Menaka²

Associate professor

Dept. of Applied Electronics

Noorul Islam Centre for Higher Education

P. D. Mozley et al., “Measurement of Tumor Volumes Improves RECIST-Based Response Assessments in Advanced Lung Cancer,” Transl. Oncol., vol. 5, no. 1, pp. 19–25, Feb. 2012.
S. A. Hayes et al., “Comparison of CT volumetric measurement with RECIST response in patients with lung cancer,” Eur. J. Radiol., vol. 85, no. 3, pp. 524–533, Mar. 2016.
F. Isensee et al., “nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation,” Sep. 2018.
W. Chen, H. Wei, S. Peng, J. Sun, X. Qiao, and B. Liu, “HSN: Hybrid Segmentation Network for Small Cell Lung Cancer Segmentation,” IEEE Access, vol. 7, pp. 75591–75603, 2019.
Y. Gordienko et al., “Deep Learning with Lung Segmentation and Bone Shadow Exclusion Techniques for Chest X-Ray Analysis of Lung Cancer,” 2019, pp. 638–647.
H.-C. Shin et al., “Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning,” IEEE Trans. Med. Imaging, vol. 35, no. 5, pp. 1285–1298, May 2016.
P. Tschandl, C. Sinz, and H. Kittler, “Domain-specific classification-pretrained fully convolutional network encoders for skin lesion segmentation,” Comput. Biol. Med., vol. 104, pp. 111–116, Jan. 2019.
M. Chen, X. Shi, Y. Zhang, D. Wu, and M. Guizani, “Deep Features Learning for Medical Image Analysis with Convolutional Autoencoder Neural Network,” IEEE Trans. Big Data, 2017.
M. Nishio et al., “Attribute-guided image generation of three-dimensional computed tomography images of lung nodules using a generative adversarial network,” Comput. Biol. Med., vol. 126, p. 104032, Nov. 2020.
Jin, D., Xu, Z., Tang, Y., Harrison, A. P., and Mollura, D. J. (2018). CT-realistic Lung Nodule Simulation from ds Conditional Generative Adversarial Networks for Robust Lung Segmentation. Lect Notes Computer Science (Including Subser Lect Notes Artif Intell Lect Notes Bioinformatics).;11071, 732–740. Available at: http://arxiv.org/abs/1806.04051
Han, C., Kitamura, Y., Kudo, A., Ichinose, A., Rundo, L., Furukawa, Y., et al. (2019). “Synthesizing Diverse Lung Nodules Wherever Massively: 3D MultiConditional GAN-based CT Image Augmentation for Object Detection,” in Proceedings - Int Conf 3D Vision. 3DV, 729–737. Available at: http://arxiv.org/ abs/1906.04962.
Onishi, Y., Teramoto, A., Tsujimoto, M., Tsukamoto, T., Saito, K., Toyama, H., et al. (2019). Automated Pulmonary Nodule Classification in Computed Tomography Images Using a Deep Convolutional Neural Network Trained by Generative Adversarial Networks. Biomed. Res. Int. 2019, 1–9. doi:10.1155/ 2019/6051939.
Yang, J., Liu, S., Grbic, S., Setio, A. A. A., Xu, Z., Gibson, E., et al. (2019). “Classaware Adversarial Lung Nodule Synthesis in Ct Images,” in Proceedings - International Symposium on Biomedical Imaging. IEEE Computer Society, 1348–1352. doi:10.1109/ISBI.2019.8759493.
Yi, X., Walia, E., and Babyn, P. (2019). Generative Adversarial Network in Medical Imaging: A Review. Med. Image Anal. 58, 101552. doi:10.1016/ j.media.2019.101552
Armanious, K., Jiang, C., Fischer, M., Küstner, T., Hepp, T., Nikolaou, K., et al. (2020). MedGAN: Medical Image Translation Using GANs. Comput. Med. Imaging Graphics 79, 101684. doi:10.1016/ j.compmedimag.2019.101684

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Segmentation of Pulmonary Nodules Using SegNet and Adversarial Networks

Status:

Version 1

Abstract

Figures

I. Introduction

Ii. Review Of Literature

Iii. Methodology

Iv. Results And Discussion

V. Conclusion

Declarations

References

Additional Declarations

Status:

Version 1