Early detection of tongue cancer using a convolutional neural network and evaluation of the effectiveness of EfficientNet

doi:10.21203/rs.3.rs-1628071/v1

Download PDF

Research Article

Early detection of tongue cancer using a convolutional neural network and evaluation of the effectiveness of EfficientNet

https://doi.org/10.21203/rs.3.rs-1628071/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Objectives

Early detection of oral cancer is critical because the survival rate is reported to be 90%, whereas 45.5% for advanced oral cancer occur with neck metastasis. This study assessed the detection potential of oral tongue cancer and precancerous lesions using a convolutional neural network (CNN) and evaluated the effectiveness of EfficientNet in a situation with limited datasets.

Materials and Methods

1,810 tongue images organized into four categories (malignant tumors, precancerous lesions, benign or inflammatory lesions, and normal) were used for model training. Data augmentation, transfer learning, and fine-tuning have been used to overcome the problems associated with limited datasets. Also, the weight balancing method was introduced to mitigate class imbalance by assigning different weights to each class. VGG16, Inception-ResNet-V2, and EfficientNet models were used and compared to each other.

Results

This study evaluated the possibility of detecting tongue cancer and precancerous lesions using CNN. The final model achieved an accuracy of 0.9167, a precision of 0.9212, a recall of 0.9167, and an F₂ score of 0.9176 with the test dataset.

Conclusions

Our proposed model can detect potential oral cancer and precancerous lesions. EfficientNet was effective through data augmentation and weight balancing in detecting the tongue lesions with a limited dataset. The results of this study may help patients and general practitioners in tongue lesion diagnosis.

Clinical relevance

In many cases, tongue cancer is initially confused with glossitis, even by general practitioners. Thus, screening of tongue cancer/precancerous lesions through an artificial neural network will help detect tongue cancer early.

convolutional neural network

CNN

oral cancer

tongue cancer

EfficientNet

With the development of artificial intelligence (AI), image processing technology in many fields has been developed and is being applied industrially. In the medical field, AI is often used to assist doctors by finding areas that doctors cannot see [1]. However, unlike other industrial fields, AI application in the medical field has a limitation in that the number of input data is small. Therefore, it is imperative to obtain optimal accuracy and recall using a small dataset.

It has been reported that the accuracy of using AI for diagnosing skin lesions is as high as that of an experienced dermatologist [2, 3]. Also, several studies have reported the early detection of oral cancer using AI [4]. In most cases, a convolutional neural network (CNN) is used. There are several popular CNN models, such as VGG16 [5], Inception-ResNet-V2 [6], and EfficientNet [7]. They are the mainly used CNN-based models so far in this field of research.

Thus, this study evaluated the detection potential of oral cancer and precancerous lesions using a CNN and evaluated the effectiveness of the EfficientNet network in overcoming the limitations of a small dataset in the medical field.

Dataset

Table 1

Classes and number of tongue lesion samples
Class	Number of samples
Class 0 (Cancer)	372
Class 1 (Precancer)	141
Class 2 (Benign or Inflammation)	144
Class 3 (Normal)	1,153
Total	1,810

In this study, 1,810 tongue images were collected from adult patients over age 20 who visited the Department of Oral and Maxillofacial Surgery from Jan 2006 to Dec 2020. The dataset was approved by the Institutional Review Board of the Seoul National University Dental Hospital (ER121036). Images were divided into four categories: malignant tumors, precancerous lesions, benign or inflammatory lesions, and normal (Table 1). Two board-certified surgeons with more than five years of clinical experience performed the classification of tongue images. For the test set, 10% (180 images) of the whole dataset was randomly selected by stratified sampling. The remainder were used as training and validation sets. Of these, 70% (1,141 images) were used for training and 30% (489 images) for validation. We defined the base model and tuned hyperparameters by comparing the results of the validation set. Results of the test set were used only for evaluating the final model.

Models

EfficientNet [7] served as the backbone model for the classification. It has the advantage of saving a significant number of parameters and computational costs while showing similar accuracy to several conventional CNN models, such as ResNet50, DenseNet201, and Inception-ResNet-v2. Additionally, it allows us to use the transfer learning approach by providing the pretrained models from a large-scale dataset (e.g., ImageNet).

VGG16 and Inception-ResNet-V2 were used as comparative models, both of which are representative CNN-based models and have been widely adopted in medical image classification tasks. Also, they provide pretrained models for transfer learning, likewise the EfficientNet.

Among the EfficientNet models, the most basic EfficientNetB0 model was adopted in this study because of the small datasets used. The number of parameters was 14,718,788 for VGG16 and 54,349,028 for Inception-ResNet-V2, respectively. In comparison, EfficientNetB0 had the lowest number, which was 4,059,815. The input shape of all models is fixed at 224 × 224 for a fair backbone model comparison (Fig. 1).

Transfer Learning And Fine Tuning

Transfer learning is effective in image classification tasks since many datasets share low-level spatial characteristics, such as line, point, and texture. These low-level characteristics are better learned by training a network with a big dataset, for example, ImageNet. Therefore, transfer learning can provide performance gains for medical deep learning applications prone to struggle with limited datasets [8]. Transfer learning was conducted using an Adam optimizer on a pretrained EfficientNetB0 model. The maximum epoch of the learning parameters, the mini-batch, and the learning rate initial value were 60, 8, and 1e^− 2, respectively. After transfer learning, fine-tuning was performed to update the model to suit the current dataset, starting from the last-block part in a bottom-up manner. Finally, we decided that there was a big difference between the ImageNet dataset and tongue cancer images. Therefore, the feature extractor part was retrained (Fig. 2).

Data Augmentation

ImageDataGenerator of the Keras framework was used to generate batches of tensor image data with real-time data augmentation. Inputs were rotated with random angles between − 30° and 30°. They were randomly zoomed by 80–100% and flipped horizontally and vertically. They are randomly shifted to the left or right between − 10% and 10% of the total width and vertically in the same way. Points outside the input boundaries are filled according to the nearest pixels.

Weight Balancing

The amount of data per class was unbalanced. Classes 0, 1, 2, and 3 consisted of 372 (20.55%), 141 (7.79%), 144 (7.95%), and 1,153 (63.7%) images, respectively. When unbalanced data are used for deep learning, learning on data with low observations will not work well because learning is focused on data with high observations [9]. This study used the weight balancing method to mitigate class imbalance by assigning different weights to each class. The model sets higher weights for classes with fewer samples so that they could receive a stronger penalty for misclassification of classes with small samples and vice versa for classes with many samples. The class weights used are specified according to the number of samples. Values are as follows: class 0, 0.15; class 1, 0.4; class, 2, 0.4; and class 3, 0.05.

Performance Evaluation

To evaluate the screening performance of the model, accuracy, precision, recall, and F₂ score were calculated. During early diagnosis and screening, missing a tumor is worse than giving a false alarm for a nonexistent tumor. Thus, the F₂ score is used because it lowers the importance of precision and increases the importance of recall. The F₂ score was computed using weighted average precision and weighted average recall. Accuracy, precision, recall, and F₂ scores are defined as follows (Table 2).

Table 2

Metrics used for performance evaluation and the corresponding definition
Metric	Definition
True Positive (TP_i)	the number of correctly recognized observations for class C_i
True Negative (TN_i)	the number of correctly recognized observations that do not belong to the class C_i
False Positive (FP_i)	the number of observations that were incorrectly assigned to the class C_i
False Negative (FN_i)	the number of observations that were not recognized as belonging to the class C_i

$$accuracy= \frac{TP+TN}{TP+TN+FN+FP}$$

$$precision= \frac{TP}{TP+FP}$$

$$recall= \frac{TP}{TP+FN}$$

$${F}_{2} score=\left(1+{2}^{2}\right)* \frac{precision*recall}{\left({2}^{2}*precision\right)+recall}$$

Grad-cam

Grad-CAM (gradient-weighted class activation mapping) is a post-hoc model-explanation tool [10]. It produces a heatmap that indicates which part of the input image is considered important for the classification, making CNN-based models more transparent and explainable. Grad-CAM uses class-specific gradient information flowing into the last convolutional layer of the CNN model to understand each neuron for a particular decision of interest. The input image (left) and the heatmap (right) are shown. The red area in the heatmap (right) is the most important part of the model to classify the input image (Fig. 3).

Comparison of backbone models

Table 3

Classification results of backbone models on the validation set
Model	Accuracy	Precision	Recall	F₂ Score
VGG16	0.8364	0.8173	0.8364	0.8325
Inception-ResNet-V2	0.8446	0.8247	0.8446	0.8405
EfficientNetB0	0.8609	0.8444	0.8609	0.8575

The performances of backbone models are listed in Table 3. EfficientNetB0 reached the highest F₂ score of 0.8575, while VGG16 achieved the lowest F₂ score of 0.8325. EfficientNetB0 also achieved the highest accuracy of 0.8609, precision of 0.8444, and recall of 0.8609. This supports the validity of our choice of EfficientNet as the backbone model.

Classification Experiments

Table 4

Classification results of EfficientNet-based models using the validation set (DA: data augmentation, WB: weight balancing, FT: fine-tuning)
Model	Accuracy	Precision	Recall	F₂ Score
EfficientNet	0.8609	0.8444	0.8609	0.8575
EfficientNet + DA	0.8630	0.8523	0.8630	0.8608
EfficientNet + DA + WB	0.8098	0.8588	0.8098	0.8191
EfficientNet + DA + WB + FT	0.8978	0.8957	0.8978	0.8974

Data Augmentation

The performance of the models that applied data augmentation techniques was slightly improved. The accuracy, precision, recall, and F₂ score were 0.8609–0.8630, 0.8444–0.8523, 0.8609–0.8630, and 0.8575–0.8608, respectively (Table 4).

Data Augmentation + weight Balancing

Table 5

Classification results of the EfficientNet model that applied data augmentation and weight balancing using the validation set (*: Minority samples)
Class	Precision	Recall	F₂ Score	Support
1. Cancer	0.7417 → 0.8353	0.8812 → 0.7030	0.8493 → 0.7260	101
2. Precancer*	0.6667 → 0.3770	0.1579 → 0.6053	0.1863 → 0.5399	38
3. Benign or Inflammation*	0.7500 → 0.4923	0.6154 → 0.8205	0.6383 → 0.7240	39
4. Normal	0.9238 → 0.9712	0.9743 → 0.8682	0.9638 → 0.8870	311

The amount of data per class was unbalanced. Thus, the weight balancing method was applied to mitigate this issue. The model's performance was degraded compared to the model without weight balancing. However, the recall and F₂ scores of Classes 2 and 3, which contained minority samples, were considerably improved. Recall was 0.1579–0.6053 and 0.6154–0.8205 in Classes 2 and 3, respectively. Their F₂ scores improved from 0.1863 to 0.5399 and from 0.6383 to 0.7240, respectively (Table 5).

Data Augmentation + weight Balancing + fine-tuning

Table 6

Classification results of the EfficientNet model that applied data augmentation, weight balancing, and fine-tuning techniques using the test set
Model	Accuracy	Precision	Recall	F₂ Score
EfficientNet + DA + WB + FT	0.9167	0.9212	0.9167	0.9176

The model that applied data augmentation, weight balancing, and fine-tuning techniques achieved the highest performance, with an accuracy of 0.8978, a precision of 0.8957, a recall of 0.8978, and an F₂ score of 0.8974. The final model achieved an accuracy of 0.9167, a precision of 0.9212, a recall of 0.9167, and an F₂ score of 0.9176 with the test set (Table 6).

This study evaluated the possibility of detecting tongue cancer and precancerous lesions using CNN. Then, EfficientNet’s effectiveness was evaluated to overcome the limitation of a small dataset inherent in the medical field.

Thus, our proposed model can detect potential oral cancer and precancerous lesions. EfficientNet was effective through data augmentation and weight balancing in detecting tongue lesions with a limited dataset. Finally, results of this study may help patients and general practitioners in tongue lesion diagnosis.

VGG16 is a CNN model proposed by Simonyan and Zisserman [5], and it improves AlexNet by individually replacing large kernel-sized filters with multiple 3 × 3 kernel-sized filters one after another. The model achieved 92.7% top-5 test accuracy on ImageNet. Inception-ResNet-V2 is a CNN model proposed by Christian Szegedy and Sergey [6]. The network has an image input size of 299 × 299, 164 layers deep, and 95.3% top-5 test accuracy on ImageNet. VGG16 and Inception-ResNet-V2 have been widely adopted in medical image classification tasks. Also, EfficientNet [7] is a CNN architecture that uniformly scales network width, depth, and resolution with an asset of fixed scaling coefficients, unlike conventional CNN architectures. Through uniform model scaling, EfficientNet improves accuracy and efficiency. Therefore, it is expected to serve as a new foundation for future CNN-based classification tasks.

Limited dataset is a prevalent challenge in medical image analysis. Thus, the transfer learning technique is often required. All models used in this study have been trained on a large-scale dataset (ImageNet) so that a transfer learning approach is available for a new classification task. EfficientNet transfers well and has achieved a state-of-the-art performance. Here, EfficientNet provided the best capability for feature extraction with a limited training dataset.

The diagnosis rate of oral cancer lesions using visual inspection and palpation by an oral specialist treating cancer is 90%. The general practitioner's diagnostic rate of oral cancer lesions is 75% [11]. In this study, our final model for oral cancer diagnosis had an accuracy of 0.9167, a precision of 0.9212, a recall of 0.9167, and an F₂ score of 0.9176. These scores were based on a test dataset not used for training. Therefore, results suggest that comparable diagnostic performance can be achieved using this model. Furthermore, this model can assist general practitioners in diagnosis before referring a patient to an oral specialist treating cancer.

Using the Grad-CAM post-hoc tool, researchers and clinicians can determine whether AI makes a diagnosis by looking at the same part as human sees. Furthermore, there is a possibility that AI can extract new information from the area where doctors conventionally think that it has no informational value. For instance, a recent study has found that AI can extract useful information that ophthalmologists could not discover by fundoscopy in ophthalmology [1]. We expect that Grad-CAM can also be used to help the struggle with the limited dataset. By extracting representative regions of an image calculated from Grad-CAM and combining four different images from each class, an image with four different contexts can be generated. We plan to extend the work on model training with newly generated images to see if it helps with class imbalance.

Here, AI was trained to distinguish cancer, precancerous lesions, and normal tissues through pictures of the tongue. It is expected that the scope of this study can be extended to the entire oral cavity, such as the buccal mucosa, retromolar trigon, gingiva, and palate once we have enough datasets for each case.

Additionally, AI can be utilized by the public for building Android applications and web APIs. This can provide auxiliary help in identifying oral cancer/precancerous lesions on their own, even for those who have low access to tertiary center hospitals. Anyone with a smartphone camera can take their tongue picture and upload it to the web APIs and application. This is expected to contribute to an increase in the early detection rate of oral cancer and an increase in the survival rate of oral cancer patients in the long term.

Author Contributions

S-J LEE and I-J Kwon equally contributed as primary authors, while J-H Lee and H-K Kim equally contributed as corresponding authors. Conceptualization, H-K Kim and J-H Lee; methodology, S-J LEE and I-J Kwon; validation, Y-D Son, J-H Kim, and D Kwon; data curation, M-S Lee, E-S Song, and B Kim; prepared figures, S-J Lee, D Kwon, and H-K Kim; review and editing, S-J Lee, I-J Kwon, J-H Lee, H-K Kim, Y-D Son, J-H Kim, D Kwon, M-S Lee, E-S Song, and B Kim; funding acquisition, H-K Kim. All authors have read and agreed to the published version of manuscript.

Acknowledgments

This research was supported by grants from the Technology Innovation Program (Grant number: 20006105) funded by the Ministry of Trade, Industry & Energy, Republic of Korea and the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea (Grant number: HI20C2114).

Conflicts of Interests

The authors declare that they have no conflicts of interest.

Funding

This research was supported by grants from the Technology Innovation Program (20006105) funded by the Ministry of Trade, Industry & Energy, Republic of Korea and the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea (Grant number: HI20C2114).

Ethics approval and consent to participate

All the participants’ guardians’ informed consents were obtained. The present study was approved by the Institutional Review Board of Seoul National University Dental Hospital (IRB No. ERI21036).

Kim, Y. D., Noh, K. J., Byun, S. J., Lee, S., Kim, T., Sunwoo, L., Lee, K. J., Kang, S. H., Park, K. H., & Park, S. J. (2020). Effects of hypertension, diabetes, and smoking on age and sex prediction from retinal fundus images. Scientific reports 10: 4623. https://doi.org/10.1038/s41598-020-61519-9
Esteva A, Kuprel B, Novoa RA, et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118. https://doi.org/10.1038/nature21056
Maron RC, Utikal JS, Hekler A, et al (2020) Artificial intelligence and its effect on dermatologists’ accuracy in dermoscopic melanoma image classification: Web-based survey study. J Med Internet Res 22:1–9. https://doi.org/10.2196/18091
Welikala RA, Remagnino P, Lim JH, et al (2020) Automated detection and classification of oral lesions using deep learning for early detection of oral cancer. IEEE Access 8:132677–132693. https://doi.org/10.1109/ACCESS.2020.3010180
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Szegedy C, Ioffe S, Vanhoucke V et al (2017) Inception-v4, inception-ResNet and the impact of residual connections on learning. 31st AAAI Conf Artif Intell AAAI 2017 4278–4284
Tan M, Le Q V. (2019) EfficientNet: Rethinking model scaling for convolutional neural networks. 36th Int Conf Mach Learn ICML 2019 2019:10691–10700
Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22:1345–1359. https://doi.org/10.1109/TKDE.2009.191
Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks 106:249–259. https://doi.org/10.1016/j.neunet.2018.07.011
Selvaraju RR, Cogswell M, Das A, et al (2017) Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proc IEEE Int Conf Comput Vis 2017:618–626. https://doi.org/10.1109/ICCV.2017.74
Downer MC, Evans AW, Hallett CMH, et al (1995) Evaluation of screening for oral cancer and precancer in a company headquarters. Community Dent Oral Epidemiol 23:84–88. https://doi.org/10.1111/j.1600-0528.1995.tb00206.x

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Early detection of tongue cancer using a convolutional neural network and evaluation of the effectiveness of EfficientNet

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Dataset

Models

Transfer Learning And Fine Tuning

Data Augmentation

Weight Balancing

Performance Evaluation

Grad-cam

Results

Comparison of backbone models

Classification Experiments

Data Augmentation

Data Augmentation + weight Balancing

Data Augmentation + weight Balancing + fine-tuning

Conclusion

Discussion

Declarations

References

Additional Declarations

Status:

Version 1