Development of Pathological Super-resolution Images using Artificial Intelligence based on Whole Slide Image

Background: Pathology plays a very important role in the cancer diagnosis, as the gold standard for the identification of tumors. The rapid development of digital pathology (DP) which based on Whole Slide Image (WSI) has led to many improvements in telepathological consultation, digital management, and computer-assisted diagnosis by artificial intelligence (AI). In DP, the common digitization strategy is to scan the pathology slice with X20 or X40 objective. Usually, the X40's WSI is 4 times bigger than the X20's, and obviously, the storage space and transmission time of the data should be 4 times. These increased costs will be great negative factor in the popularization of DP. But at the same time, some cases have to use the high magnification WSI for reliable diagnosis. Methods: In this article, we present a novel super-resolution process which could be used for WSI through Deep Learning. This process powered by AI, have the ability to switch X20 WSI to X40 without loss of whole and locally features. Furthermore, we collect the examples of WSI data of patients with 100 uterine leiomyosarcoma and adult granulosa cell tumor (AGCT) of ovary respectively, which are used to test our super-resolution process. Results: We used the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM), and the Blind/Referenceless Image Spatial QUality Evaluator (BRISQUE) to test the resulting X40 WSI synthesized by the super-resolution (SR), which were 42.03, 0.99 and 49.22 . Then, we tested our SR images from subjective evaluation of the pathologist's perspective, and tested that if the pathologists could objectively distinguish the images between SR and high-resolution (HR), to further confirm the consistency between our SR images and the real HR images. Conclusions: The testing results indicate that the X40 WSI synthesized by the super-resolution matches the performance of the one generated from the X40 objective in of both We that this is a reliable method be used in of digital slides, and will in clinical pathology


Introduction
Pathology is an important way to diagnosis of diseases, in particular in malignant tumours, the whole slide image, which meet the characteristics of high definition, high speed, and high throughput screening, provides the possibility for digitization of traditional pathology slice, which lays a good foundation for the development and application of digital pathology (DP) 1− 2 . With the development of DP, the storage and transmission of the pathological image gets easier, which has led to many improvements in telepathological consultation, digital management, and computer-assisted diagnosis by artificial intelligence (AI). There have been reportedly more than 1,000,000 cases of telepathological consultation in China of the last 10 years. 3 In digital pathology, the common digitization strategy is to scan the pathology slice with X20 or X40 objective, the data generated by scan is called WSI (or digital slide). Usually, X40's WSI is 4 times bigger than the X20's which from the same slice, and obviously, the storage space and transmission time of the data should be 4 times. These increased costs will be great negative factor in the popularization of DP. Although many lesions can be diagnosed at low or medium magnification, in some cases, even experienced pathologists need to observe cells and cell nuclear morphology at high magnification to further confirm the diagnosis. For example, the dusty cytoplasm is often characterized by small cell neuroendocrine carcinoma, the longitudinal nucleus has a great role in the diagnosis of Langerhans histiocytosis and ovarian granulosa cell tumor, the obvious mitotic has a good predictive effect on the diagnosis and differential diagnosis of uterine leiomyosarcoma. In order to provide more clear and identifiable evidence for the diagnosis of tumors, whether it is an optical image or a digital image at a high magnification (x40), it is necessary to meet the high requirements of the pathologist to see the details in the nucleus. In this article, we present a novel super-resolution(SR) process which could be used for WSI through Deep Learning.
For super-resolution imaging, deep learning techniques were recently developed for CT and MRI with a great success 4 , there were some early attempts with a 3D convolutional neural network, generative adversarial network (GAN), and densely connected network 5− 7 . Thus, the super-resolution techniques have been used and shown super values in imaging 8 , and we expect the same in digital pathology.

Materials And Methods
In the present study, we use adult granulosa cell tumor (AGCT) of ovary and uterine leiomyosarcoma as two application scenarios to test our resolution process. AGCT is a lowgrade malignant neoplasm with a significant propensity for late recurrence and Hospital, Sichuan University. All cases with no history of cancer and did not receive radiotherapy before surgery. Specimens were fixed with 4% neutral formaldehyde, conventional paraffin-embedded, 4 µm thick sections, HE staining, and light microscopy examination. All the HE slides were reviewed by 2 senior pathologists. At last, a total of 100 HE slides from 45 cases of ovarian AGCT and 100 HE slides from 32 cases of uterine leiomyosarcoma were included. All the 200 HE slides were made into full digital scanning section WSI by Hamamatsu Optics' NanoZoomer 2.0HT digital section scanner, the scanning magnification was 20 times objective and 40 times objective.
With the development of AI, deep learning with deep convolutional neural networks (CNNs) 12 has been shown to be a powerful algorithm for advancing biomedical image analysis 13 − 14 . By studying the development and progress in the field of Single Image Super-Resolution(SISR), we find that in these two years deep learning methods are obviously superior to the traditional ones in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). Our goal is to zoom in 40 × from 20×, so we can adopt a single-scale model. For further considering the effect and computation comprehensively, EDSR is a kind of generative network based on sample learning. The general process is that firstly to get the low-resolution(LR) image down-sampled by interpolation from highresolution (HR) image, as the LR input of a convolutional neural network(CNN), and the original image will be as the HR input of a CNN; then a large number of such paired samples are used to train the network so as to establish the end-to-end mapping relationship between the LR image and its corresponding HR image; and finally the mapping relationship is used to create a super-resolution(SR) image from LR image.
The author of EDSR constructs the model baseline with residual blocks, whose structure is similar to that of SRResNet 15 . However, the EDSR does not have the Rectified Linear Unit  16 applied convolution layers on the pre-upscaled LR image. It is inefficient because all convolutional layers have to compute on highresolution feature space, yielding much more computation than on low-resolution space.
To accelerate processing speed without loss of accuracy, Fast Super-Resolution Convolutional Neural Network (FSRCNN) 17 utilized parametric deconvolution layer at the end of SR network, making all convolution layers compute on LR feature space. Another non-parametric efficient alternative is pixel shuffling 18 (a.k.a., sub-pixel convolution).
Pixel shuffling is also believed to introduce less checkerboard artifacts than the deconvolutional layer. We also used the pixel shuffling as the up-sampling operation.
We used the adult granulosa cell tumor images and the leiomyosarcoma images as our datasets ( Fig. 1a-b). We get a large of training patches with size of 512 × 512 as the HR images randomly spliting from each training Whole-Slide Imaging(WSI) and validating patches are from each validating WSI and testing patches are from each testing WSI.
Due to computational resources, we just use the baseline model of EDSR, and the specific parameters in training are listed in Table 1. When training, we use the RGB input patches of size 256 × 256 from LR image with the corresponding HR patches. The 512 × 512 RGB input patches from HR image and its bilinear down-sampled image will be as the training output-input pairs. We pre-process all the images by subtracting the mean RGB value based on the default settings. We also train our networks using L1 loss instead of L2, because L1 loss can provides better convergence than L2 as far as we know. We used both the granulosa cell tumor and the leiomyosarcoma output-input pairs. After a few training sessions, the final training loss curve is shown in Fig. 1c and the PSNR curve in validating dataset is shown in Fig. 1d. The last training used the best model before as the pretrained model.

Results
We applied our trained model to the testing dataset including 2000 images with benchmark. We test two kinds of LR input with size of 256 × 256, one is bilinear downsampled images from 40 × HR images (named sr_bilinear_as_lr_input), and another is directly from the 20 × images in WSI (named sr_self_as_lr_input). The mean PSNR and SSIM values between the SR images and HR images from testing dataset are listed in tabel 2. In general, the larger the PSNR or SSIM value is, the better quality the image has. PSNR value takes the infinite and SSIM value tasks 1.0 when two compared images are just the same. Some images results are shown in Fig. 2. We can conclude that our SR results are extremely similar to the HR images. To a large extent, we have achieved super-resolution reconstruction of the images.
We also applied our model to another dataset including 1000 images with approximate benchmark in which LR images with size of 256 × 256 are from the true 20 × WSI and HR images with size of 512 × 512 are from other true 40 × WSI. Some images results are shown in Fig. 3. The LR images are not down-sampled from HR images and they come from different WSI. Although we get the similar-pair images based on manual registration method as possible as we can, they do not have one-to-one correspondence in pixel spatial position. We couldn't use the PSNR or SSIM metrics to evaluate the results. We used another evaluation method named Blind/Referenceless Image Spatial QUality Evaluator (BRISQUE) 22 , which is a kind of no-reference image quality assessment. That is to say, it just needs one image as input and the image quality score as the output. In general, the score is between 0 and 100, and the smaller the score is, the better quality the image has. The mean BRISQUE scores of the HR images and the SR images related to Bicubic and ours on this testing dataset are listed in tabel 3. The average calculation time of our model for each image is about 2.5 milliseconds on GPU. From the results, we can see that the quality of our results are better than the SR images up-sampled by Bicubic, even better than the HR images. To a large extent, we have achieved super-resolution reconstruction of the images and our model can be applied to the true 20 × WSI to reconstructing 40 × WSI.
In order to further observe the texture details of SR images, we create two new test datasets, including 192 nuclear division images and 546 nuclear groove images with size of 32 × 32, which are annotated by doctors. Some images results are shown in Fig. 4 and the mean BRISQUE scores of the HR images and the SR images with Bicubic and ours are listed in tabel 4. From the results, we can also see that the quality of our results are better than the SR images up-sampled by Bicubic, even better than the HR images. To a large extent, our results can be used directly for subsequent tasks in these datasets.
Then, we tested the subjective evaluation of our SR images from the pathologist's perspective. We selected 1000 images from each of uterine leiomyosarcoma and adult granulosa cell tumor of ovary, and provided HR images and our SR images respectively.
Two pathologists (intermediate titles) evaluated the SR images' authenticity subjectively, and considered whether the quality of the SR images are adequate for routine usage (diagnostic confidence), the results are shown in Table 5. In addition, we extracted 200 images from each of the two tumors, provided HR images and our SR images respectively but without any tags or comments, and let the pathologist to determine which are the true images (HR images). The test results showed that the probability of the two pathologists could accurately select the real images was 51.75% and 54.25% respectively (Table 6), which objectively proved that the pathologists could hardly distinguish the difference between these two kinds of images, and further confirmed the consistency between our SR images and the real HR images.

Discussion
The testing results indicate that the quality of our SR images reconstructed from true LR images are better than the SR images up-sampled by Bicubic, even better than the HR images whether from the perspective of vision or evaluation values, and the X40 WSI synthesized by the super-resolution matches the performance of the one generated from the X40 objective in diagnosis of both tumors. Based on this, we believe that the superresolution technology is a reliable method in telepathological consultation, to improve diagnostic accuracy, reducing time and storage cost, and play an irreplaceable role in diseases which must be diagnosed at high magnification.
However, in this article, the super-resolution technology still has some problems and limitations. First, all we did is double zoom, if the magnification factor is larger such as × 4 or × 8, the restoration of details is not so good. Second, we just applied the baseline model to our task due to limited computing resources. The last, the model we have trained only applies to uterine leiomyosarcoma and adult granulosa cell tumor (AGCT) of ovary, and we need more data for transfer learning if we want to apply this model to other diseases.
Inspiringly, the super-resolution algorithms are also improving, the authors 15 of the EDSR proved experimentally that the processing effect can be improved if the network is widened and deepened, some research 23 about fractional magnification has been done and it's working very well. We believe that methods based on deep learning will be more and more applicable to our real tasks. And it should be a reliable method can be used in a variety of tumors' digital slides, and will be available for a large scale in clinical pathology as an innovative technique.

Acknowledgments
We thank all the pathologists and software engineers for participating in this study.

Author contributions
Ji Bao, Hong Bu designed the study.

Availability of data and materials
The datasets used during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate
Data used in this study was collected as part of medical records. Institutional and/or national research ethic committee has approved the data collection and management process.       Table 6 The objective test of the SR images from the pathologists HR: high-resolution, SR: super-resolution Result samples of nuclear division and nuclear groove. The a row are nuclear division images, the b row are nuclear groove images, the first column are LR images, the second column are HR images, the third column are the SR images up-sampled by Bicubic, the last column are our SR images. LR: low-resolution, SR:

Tables
super-resolution, HR: high-resolution