A trial deep learning-based model for four-class histologic classification of colonic tumor from narrow band imaging

doi:10.21203/rs.3.rs-2497829/v1

Download PDF

Article

A trial deep learning-based model for four-class histologic classification of colonic tumor from narrow band imaging

https://doi.org/10.21203/rs.3.rs-2497829/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 09 May, 2023

Read the published version in Scientific Reports →

You are reading this latest preprint version

Narrow band imaging (NBI) has been extensively utilized as a diagnostic tool for colorectal neoplastic lesions. This study aimed to develop a trial deep learning (DL) based four-class classification model for low-grade dysplasia (LGD); high-grade dysplasia or mucosal carcinoma (HGD); superficially invasive submucosal carcinoma (SMs) and deeply invasive submucosal carcinomas (SMd) and evaluate its potential as a diagnostic tool. We collected a total of 1390 NBI images as the dataset, including 53 LGD, 120 HGD, 20 SMs and 17 SMd. A total of 59,8801 patches were trimmed from the lesion and background. A patch-based classification model was built by employing a residual convolutional neural network (CNN) and validated by three-fold cross-validation. The patch-based validation accuracy was 0.876, 0.957, 0.907 and 0.929 in LGD, HGD, SMs and SMd, respectively. The image-level classification algorithm was derived from the patch-based mapping across the entire image domain, attaining accuracies of 0.983, 0.990, 0.964, and 0.992 in LGD, HGD, SMs, and SMd, respectively. Our CNN-based model demonstrated high performance for categorizing the histological grade of dysplasia as well as the depth of invasion in routine colonoscopy, suggesting a potential diagnostic tool with minimal human inputs.

Biological sciences/Computational biology and bioinformatics

Biological sciences/Computational biology and bioinformatics/Computational models

Although it is relatively simple for human observers to recognize and describe the visual elements in empirical terms, it has been remarkably difficult to accurately define and analyze them with a computer. The electronic endoscope has allowed us to quantify any element making up a digitized endoscopic image through mathematical processes. Several studies have evaluated the effectiveness of feature extraction for computer-aided diagnosis (CAD) to classify the endoscopic severity of ulcerative colitis ^1,2 and to assess the risk of developing gastric cancer among Helicobacter pylori-positive patients ³. However, the diagnostic accuracy of feature engineering was limited due to the challenges in extracting features for image analysis in gastrointestinal diseases.

With the proliferation of convolutional neural network (CNN), the task of classifying objects in natural images can be solved simply by presenting examples of images and the names of the objects to a neural network that acquired all its knowledge from the training data ⁴. This groundbreaking technology has freed engineers from feature engineering and endoscopists from knowledge-based image interpretation. The CNN-based supervised learning has been applied in the automated localization of gastric cancer in routine gastroscopies ⁵ and the automated detection of colon polyps ⁶. Several studies have reported the utilization of a CNN-based model for distinguishing adenomatous from non-adenomatous polyps ⁷, adenomatous from hyperplastic diminutive colorectal polyps ⁸, and neoplastic polyps from non-neoplastic polyps ⁹.

However, to the best of our knowledge, a multi-class model for evaluating the grade of histologic dysplasia along with the depth of invasion has not yet been developed. The NBI international colorectal endoscopic (NICE) classification applicable to with or without magnification was proposed for diagnosing submucosal invasive colon cancer ¹⁰. However, criteria for classification described in empirical terms (Table 1) may inevitably suffer from a variety of biases in evaluation leading to different accuracy varying with endoscopists and disturb comparison of accuracy among different endoscopist communities. This study aimed to develop a trial CNN-based supervised learning model for evaluating histologic atypism or invading depth from NBI images of detected colonic neoplastic lesions and evaluate the potential as a diagnostic tool.

Table 1

The NBI international colorectal endoscopic (NICE) classification. NBI, narrow band imaging.

Source: DOI: 10.1016/j.gie.2013.04.185.
	Type 1	Type 2	Type 3
Color	Same or lighter than background	Browner relative to background (verify color arises from vessels)	Brown to dark brown relative to background; sometimes patchy whiter areas
Vessels	None, or isolated lacy vessels coursing across the lesion	Brown vessels surrounding white structures	Has area(s) of disrupted or missing vessels
Surface pattern	Dark or white spots of uniform size, or homogeneous absence of pattern	Oval, tubular or branched white structure surrounded by brown vessels	Amorphous or absent surface pattern
Most likely pathology	Hyperplastic	Adenoma with either low or high grade dysplasia, or with superficial submucosal carcinoma	Deep submucosal invasive cancer

Preparation of endoscopic images. NBI images of neoplastic lesions from patients who underwent endoscopic or surgical resection at Sendai City Medical Center Sendai Open Hospital from April 2017 to December 2019 were used for this single center retrospective study. Characteristics of collected NBI images are summarized in Table 2. A total of 1390 NBI images were sampled from a total of 210 lesions with definite histologic diagnosis ¹¹: 53 low-grade dysplasia (LGD); 120 high grade dysplasia or mucosal carcinoma (HGD); 20 superficially invasive (< 1000 mm) submucosal carcinoma (SMs) and 17 deeply invasive (> 1000 mm) submucosal carcinomas (SMd). Sampled picture number per lesion was 5.5 to 7 samples with an averaged image capturing conditions: no magnification 41.0%; low magnification 37.9%; high magnification 21.1%. The video endoscopes CF-HQ290ZI, PCF-H290ZI, PCF-H290TI and video endoscopy system EVIS LUCERA ELITE CV-290/CLV-290SL (Olympus Medical Systems, Co., Ltd., Tokyo, Japan) were used.

Table 2

Collected NBI images for dataset. NBI, narrow band imaging; HGD, high grade dysplasia; LGD, low grade dysplasia; SMs, superficially invasive submucosal carcinoma; SMd, deeply invasive submucosal carcinoma.
Histology	Number of lesions	Number of still pictures	Averaged picture number per lesion	Magnification (picture counts/ %)
Histology	Number of lesions	Number of still pictures	Averaged picture number per lesion	None	Low	High
LGD	53	294	5.5	126/42.9	120/40.8	48/16.3
HGD	120	840	7.0	345/41.1	305/36.3	190/22.6
SMs	20	138	6.9	59/42.8	46/33.3	33/23.9
SMd	17	118	6.9	40/33.9	56/47.5	22/18.6
total	210	1390	6.6	570/41.0	527/37.9	293/21.1

Preparation of dataset. NBI images (Fig. 1A) were manually partitioned into the lesion (Fig. 1B) and background (Fig. 1C) from which the patch images (128 × 128 pixels) were cropped starting from the left upper corner (white dotted patch), rightwards (white solid patch), then downwards (red solid patch) at every 32-pixel-strides (white and red arrows) over the entire effective region of interest. The patches including blackouts with more than 10% of the effective region were automatically excluded from analysis. Blackouts were defined as regions with the intensity of red component lower than 50. Similarly, the patches with halations exceeding 5% of the effective region were also excluded. Halations were defined as regions with the intensity of green component higher than 250. In this study, the patches were further classified into in-focus patches and out-of-focus ones according to the amount of spatial high frequency area extracted by high pass filter with a cut-off of 6.25% Nyquist frequency. The in-focus patches were classified into 0) background (BG), 1) LGD, 2) HGD, 3) SMs and 4) SMd, and the out-of-focus ones into 5) background (BG-oof) and 6) lesion (L-oof). A total of 59,8801 patches were classified into 7 categories (Table 3). The study did not have any inclusion or exclusion criteria for pictorial quality of the patches by endoscopists. As stated, the patches with excessive blackout or halation were automatically excluded before entry. The study aimed to establish an effective histologic classifier that can be used in any common shooting conditions of NBI.

Table 3

Quantity of clipped patches within each category. BG, background; HGD, high grade dysplasia; LGD, low grade dysplasia; SMs, superficially invasive submucosal carcinoma; SMd, deeply invasive submucosal carcinoma; BG-oof, out-of-focus background; L-oof, out-of-focus lesion.
Category	The number of patches
BG	91571
LGD	52184
HGD	158187
SMs	28049
SMd	20882
BG-oof	167081
L-oof	80847

Evaluation method. We employed cross-validation to obtain more accurate results with less bias in the machine learning studies. In this study, the dataset is randomly partitioned into three equal sized folds, one fold of which is for validation and the other folds are for training. The proportion of labels was equal in each fold. The training and validation processes were repeated three times using different folds each time. The three validation results could then be averaged to produce a single estimation.

Architecture of the CNN. ResNet50 (a CNN) and Pytorch were utilized ¹². ResNet50 without pretraining was imported from Pytorch library (torchvision.models). The original patches with 128 × 128 pixels were converted into images with 224 × 224 pixels. We tuned hyper parameters, which were set by a human, as follows: optimizer, Adam; loss function, cross entropy loss; number of training epochs, 50; batch size, 256; learning rate, 0.00005 via trial and error; and number of the outer layers, 7 classes.

Image-level classification. An exemplification of SMd and the annotation mask without blackout or halation (denoted by X) are depicted in Fig. 2-A and I, respectively. The patches classified into BG, LGD, HGD, SMs, SMd, BG-oof and L-oof, by the trained CNN, are illustrated by white (Fig. 2-B), green (Fig. 2-C), yellow (Fig. 2-D), magenta (Fig. 2-E), red (Fig. 2-F), dark gray (Fig. 2-G) and cyan (Fig. 2-H) open squares, respectively, and the corresponding union masks in Fig. 2-J, K, L, M, N, O and P, respectively. Classification algorithms must be developed by utilizing in-focus patches, without sacrificing pictorial information. Here, the union masks of the patches classified into labels BG, LGD, HGD, SMs and SMd are designated by M0, M1, M2, M3 and M4, respectively. Intersection over union between X and Mi (IoUi) are given by X∩Mi/ X∪Mi (i = 0,1,2,3,4). The lesion was classified into the argmax among IoUi (i = 0,1,2,3,4), with the IoUi values of 0.12, 0.05, 0.21, 0.04 and 0.57, respectively, leading to label 4 or histologic classification SMd.

Ethics approval and consent to participate. This study was approved by the Committee of Medical Ethics of Hirosaki University Graduate School of Medicine (Aomori, Japan; reference no. 2019-1099) and Sendai City Medical Center (Sendai, Japan: reference no. 2019-0029). Informed consent was obtained in the form of opt-out on our website (https:// www. https://www.med.hirosaki-u.ac.jp/hospital/outline/resarch.html), with the approval of the Committee of Medical Ethics of Hirosaki University Graduate School of Medicine. This study was designed and carried out in accordance with the Declaration of Helsinki.

Consent for publication. Informed consent was obtained in the form of opt-out on our website (https://www.med.hirosaki-u.ac.jp/hospital/outline/resarch.html), with the approval of the Committee of Medical Ethics of Hirosaki University Graduate School of Medicine.

Accuracy of patch-level and image-level classification. An averaged validation accuracy in the patches with label 0, 1, 2, 3, 4, 5 and 6 was 0.938, 0.876,0.957, 0.907, 0.929, 0.966 and 0.904, respectively (Table 4). Table 5 displays the confusion matrix diagram depicting the outcomes of the image-level classification from the patch-based mapping across the entire image area using the trained CNN. The ground truth histology was located on the vertical axis, and the predicted histology was situated on the horizontal axis. Out of 1390 pictures, 1371 pictures were correctly classified into the correct histology with a total accuracy of 0.986.

Table 4

Patch-level three-fold validation accuracy. BG, background; HGD, high grade dysplasia; LGD, low grade dysplasia; SMs, superficially invasive submucosal carcinoma; SMd, deeply invasive submucosal carcinoma; BG-oof, out-of-focus background; L-oof, out-of-focus lesion.
Category	Fold 1			Fold 2			Fold 3			Averaged accuracy
Category	total patches	correct patches	accuracy	total patches	correct patches	accuracy	total patches	correct patches	accuracy	Averaged accuracy
BG	30524	28634	0.938	30524	28625	0.938	30523	28637	0.938	0.938
LGD	17395	15011	0.863	17395	15044	0.865	17394	15644	0.899	0.876
HGD	52729	50383	0.956	52729	50327	0.954	52729	50703	0.962	0.957
SMs	9350	8616	0.921	9350	8657	0.926	9349	8163	0.873	0.907
SMd	6961	6115	0.878	6961	6632	0.953	6960	6659	0.957	0.929
BG-oof	55694	54108	0.972	55694	53855	0.967	55693	53386	0.959	0.966
L-oof	26949	24664	0.915	26949	23758	0.882	26949	24695	0.916	0.904

Table 5

Confusion matrix and image-level accuracy. BG, background; HGD, high grade dysplasia; LGD, low grade dysplasia; SMs, superficially invasive submucosal carcinoma; SMd, deeply invasive submucosal carcinoma.
	Predicted histology
Ground truth	LGD	HGD	SMs	SMd	total	accuracy
LGD	289	4	0	1	294	0.983
HGD	6	832	0	2	840	0.990
SMs	2	2	133	1	138	0.964
SMd	0	1	0	117	118	0.992

Examples of the patch-based mapping and image-level classification. Figure 3 illustrates the examples of input images, patch-level prediction map and bar graph of IoU. In cases 1, 2, 3 and 4, the ground truth histology was consistent with the predicted histology with the maximum intersection over union. In cases 5 and 6, HGD and SMd were misclassified as SMd and HGD, respectively. In these cases, misclassification of the surrounding background into the true lesion resulted in a lower intersection over union of the true lesion compared to the misclassified ones.

In this study, we developed a trial CNN-based multi-class histology classifier model for detected colorectal neoplastic lesion in routine colonoscopy with NBI mode in common shooting condition. The diagnosis process includes patch-level histology mapping over the entire in-focus region of NBI image, trained on ResNet50 and the calculation of argmax among intersections over union between annotation mask and patch-level union masks for image-level histology. This model achieved an image-level accuracy of 0.986, suggesting its potential as a diagnostic tool.

The advancement of machine learning using CNN has enabled physicians to apply CAD of medical images in their specialized field. The American Society for Gastrointestinal Endoscopy AI Task Force ¹³ stated that CAD plays a crucial role in screening and surveillance colonoscopy for colorectal cancer prevention. Similarly, a European Society of Gastrointestinal Endoscopy mentioned to the capability of AI for accurately predicting the histology of polyps from endoscopic images and improving the cost-efficiency and safety of colonoscopic colorectal cancer screening and surveillance ¹⁴. The supervised learning of a CNN has enabled the development of a model for automated detection of colon polyps ⁶, as well as binary classification models for distinguishing adenomatous from non-adenomatous polyps (with a ten-fold validation accuracy of 0.751) ⁷, adenomatous from hyperplastic diminutive colorectal polyps (with an accuracy of 94%) ⁸, and neoplastic polyps from non-neoplastic polyps (with a high confidence rate of 0.85) ⁹.

However, a CNN model for multiclass differentiation among low-grade dysplasia, high-grade dysplasia, and carcinoma with superficial or deep submucosal invasion has not been developed. Although invading depth is determinant for therapeutic intervention (endoscopic resection or surgery), it has been evaluated so far by endoscopists with the use of knowledge-based criteria ^10,15 which inevitably suffers from a variety of biases in evaluation. One study has reported a CNN-based binary class prediction model for deeply submucosal invasive carcinoma with an overall accuracy of 85.5%, which is comparable to that of expert endoscopists ¹⁶. Although it must be done with caution, comparison of model accuracy between studies with different designs revealed a prediction accuracy of 0.991 for carcinoma with deeply submucosal invasion in this study. To accurately compare the accuracy of ML models regardless of algorithm and class numbers, a benchmark data set library with annotation masks is essential.

A patch-based CNN has been utilized for automated detection of a target area within a whole slide image in digital pathology ^17,18. This method has been recently applied for automated severity mapping along the entire colorectum in patients with ulcerative colitis from capsule endoscopy video files ¹⁹, as it is considered to have an advantage when the object for classification is composed of topographically varying elements, such as severity or atypism. When reconstructing histologic maps in resected specimens, one often encounters topographic heterogeneity in the grade of dysplasia as well as invading depth. In this study, a single histology or label was assigned to a single lesion image, thus resulting in similar labels across the entire lesion area, which may have impacted the outcomes. None the less, a multi-class classification model developed from a trained patch-level classifier has achieved a high image-level accuracy of over 0.96, which may provide a potential diagnostic tool with minimal human input in routine colonoscopy.

Data availability

The data generated or analyzed during this study are included in this published article. Some datasets generated and/or analyzed during the current study are not publicly available due to privacy but are available from the corresponding author on reasonable request.

Acknowledgements

We would like to thank Takashi Sawai, MD, PhD, Department of Pathology, Sendai City Medical Center, Toru Furukawa, MD, PhD, Fumiyoshi Fujishima, MD, PhD, Satoko Sato, MD, PhD, Department of Pathology, Tohoku University School of Medicine, and Miwa Uzuki, MD, PhD, Department of Nursing, Faculty of Medical Science and Welfare, for histologic evaluations.

Author contributions

Y.S. and T.S. contributed to the study conception and design. T.S. and K.I. collected the clinical data and tumor annotation. Y.S. established the prediction model. Y.S. and T.S. wrote the first draft of the manuscript paper. M.M., H.S. and S.F. contributed to the review, and/or critical revision of the manuscript. All authors have approved the final article.

Competing interests

The authors declare no competing interests.

Sasaki, Y., Hada, R. & Munakata, A. Computer-aided grading system for endoscopic severity in patients with ulcerative colitis. Dig. Endosc. 15,206–209, DOI: https://doi.org/10.1046/j.1443-1661.2003.00246.x (2003).
Sasaki, Y., Fukuda, S., Mikami, T. & Hada, R. Endoscopic Quantification of Mucosal Surfaceroughness for grading severity of ulcerative colitis. Dig. Endosc. 20,2891–2898, DOI: https://doi.org/10.1111/j.1443-1661.2008.00778.x (2008).
Sasaki, Y. et al. Computer-aided estimation for the risk of development of gastric cancer by image processing. Artif. Intell. Theory Pract III. 197–204, DOI: https://doi.org/10.1007/978-3-642-15286-3_19 (2010).
Krizhevsky, A., Sutskever, I. & Hinton, G.E. ImageNet classification with deep convolutional neural networks, Communications of the ACM. 60,84–90, DOI: https://doi.org/10.1145/3065386 (2017).
Hirasawa, T. et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images, Gastric Cancer. 21,653–660, DOI: https://doi.org/10.1007/s10120-018-0793-2 (2018).
Misawa, M. et al. Artificial intelligence-assisted polyp detection for colonoscopy: initial experience, Gastroenterology. 154,2027 – 2029. DOI: https://doi.org/10.1053/j.gastro.2018.04.003 (2018).
Komeda, Y. et al. Computer-Aided Diagnosis Based on Convolutional Neural Network System for Colorectal Polyp Classification: Preliminary Experience. Oncology. 93,30–34. DOI: https://doi.org/10.1159/000481227 (2017).
Byrne, M.F. et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Gut. 68,94–100, DOI: http://dx.doi.org/10.1136/gutjnl-2017-314547 (2019).
Rodriguez-Diaz, E. et al. Real-time artificial intelligence-based histologic classification of colorectal polyps with augmented visualization. Gastrointest Endosc. 93,662–670, DOI: https://doi:10.1016/j.gie.2020.09.018 (2021).
Hayashi, N. et al. Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification. Gastrointest Endosc. 78,625–632, DOI: https://doi.org/10.1016/j.gie.2013.04.185 (2013).
Bosman, F.T. et al. WHO classification of tumors of the digestive system.160–165 (IARC Press, 2010).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition. DOI: https://doi.org/10.48550/arXiv.1512.03385 (2016).
Berzin, T.M. et al. Position statement on priorities for artificial intelligence in GI endoscopy: a report by the ASGE Task Force. Gastrointest Endosc. 92,951–959, DOI: https://doi.org/10.1016/j.gie.2020.06.035 (2020).
East, J.E. et al. Advanced endoscopic imaging: European Society of Gastrointestinal Endoscopy (ESGE) Technology Review. Endoscopy. 48,1029 – 1045. DOI: 10.1055/s-0042-118087 (2016).
Kobayashi, S. et al. Diagnostic yield of the Japan NBI Expert Team (JNET) classification for endoscopic diagnosis of superficial colorectal neoplasms in a large-scale clinical practice database. United European Gastroenterol J. 7,914 – 923. DOI: https://doi.org/10.1177/2050640619845987 (2019).
Lui, T.K.L. et al. Endoscopic prediction of deeply submucosal invasive carcinoma with use of artificial intelligence. Endosc Int Open. 7,E514–E520, DOI: 10.1055/a-0849-9548 (2019).
Hou, L. et al. Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification. 2016 IEEE Conference on Computer Vision and Pattern Recognition. DOI: https://doi.org/10.48550/arXiv.1504.07947 (2016).
Roy, K., Bani, D., Bhattacharjee, D. & Nasipuri, M. Patch-based system for Classification of Breast Histology images using deep learning. Computerized Medical Imaging and Graphics. 71,90–103, DOI: https://doi.org/10.1016/j.compmedimag.2018.11.003 (2018).
Higuchi, N. et al. Automated evaluation of colon capsule endoscopic severity of ulcerative colitis using ResNet50. PLoS ONE. 17,e0269728, DOI: https://doi.org/10.1371/journal.pone.0269728 (2022).

No competing interests reported.

Download PDF

Journal Publication

published 09 May, 2023

Read the published version in Scientific Reports →

Editorial decision: Major revision
14 Mar, 2023
Reviews received at journal
25 Feb, 2023
Reviewers agreed at journal
18 Feb, 2023
Reviewers invited by journal
18 Feb, 2023
Editor assigned by journal
18 Feb, 2023
Editor invited by journal
20 Jan, 2023
Submission checks completed at journal
20 Jan, 2023
First submitted to journal
20 Jan, 2023

You are reading this latest preprint version

A trial deep learning-based model for four-class histologic classification of colonic tumor from narrow band imaging

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Declarations

References

Additional Declarations

Status:

Journal Publication

Version 1