DOI: https://doi.org/10.21203/rs.3.rs-31153/v1
Purpose The purpose of this study is to evaluate the application efficiency of artificial intelligence (AI) image-assisted diagnosis system in chest CT examination of corona virus disease 2019 (COVID-19).
Methods A total of 33 cases of COVID-19 patients who underwent chest CT in Hefei Binhu Hospital between January 2020 and March 2020 were retrospectively included. All patients were tested positive for novel coronavirus nucleic acid by fluorescent reverse transcription-polymerasechain reaction (RT-PCR). The pneumonia screening function of the AI image-assisted diagnosis system was employed for the 103 chest CT examinations of the 33 cases. The diagnosis of four senior radiologists were used as the standard for synchronous under blind state. The sensitivity, specificity, misdiagnosis rate, missed diagnosis rate and other evaluation indexes of the COVID-19 performed by the AI image-assisted diagnosis system were analyzed, and an dynamic evaluation on the CT reexamination was conducted.
Results Out of the 103 chest CT examinations, there were 88 cases of true positive, 1 case of false positive, 12 cases of true negative and 2 cases of false negative. The sensitivity was 97.78% (88/90); the specificity was 92.31% (12/13); the positive predictive value was 98.88% (88/89); the negative predictive value was 85.71% (12/14); the accuracy was 97.09% (100/103); the Youden index was 90.09%; the positive likelihood ratio was 12.711 and the negative likelihood ratio was 0.024. There were 790 identified lesions in these CT examinations in total, of which 569 were true positive and 221 were false positive. There were also 64 missed diagnosis markers. The detection rate of all lesions was 89.89% and the rate of false positives was 27.97%. In the last CT scan, the lesion size were smaller and the percentage of lesions in total lung volume along with the mean density of lesions was lower than that of the first CT scan.
Conclusion The AI image-assisted diagnosis system has certain clinical application value in the early diagnosis and follow-up evaluation of chest CT examination of COVID-19.
As the corona virus disease 2019 (COVID-19) pandemic spreads around the world, confirmed cases have been reported on all six continents except Antarctica. Up to April 4, 2020, more than one million confirmed cases have been reported both in China and abroad. The virus has strong infectivity and a long infectious incubation period. The virus is mainly transmitted by droplets and contact[1]. On February 8, 2020, new clinical cases were added in the fifth edition of COVID-19 diagnosis and treatment plan issued by the National Health Commission of the People’s Republic of China, which included "suspected case with imaging characteristics of pneumonia" as a diagnostic criterion. Although it was removed from the sixth edition of the protocol released on February 18, 2020, "pulmonary imaging manifestation with obvious lesion progression within 24-48 hours (more than 50%)" was added as a diagnostic criterion for severe patients. It is clear that chest CT examination plays an important role in diagnosis and treatment of COVID-19.
Artificial intelligence (AI) refers to the intelligence shown by machines made by human beings[2]. It is a discipline on how to make computers simulate certain thinking processes and intelligent behaviors of human beings, such as learning, reasoning, thinking and planning, etc.. AI has been applied in almost all subjects of natural science and social science. The application of AI in the medical field has also been reported[3]. It has made important progress in the screening, positioning and analysis of pulmonary nodules[4], thyroid nodules[5], intracranial aneurysms[6] and other diseases. During the in-depth cooperation between Hefei Binhu Hospital and IFLYTEK CO.,LTD., the AI image-assisted diagnosis system was used in the early screening and review evaluation of COVID-19. An applied energy efficiency assessment of the system has been conducted. After detailed summarization and analysis, the specific results are reported as follows.
General information
A total of 33 COVID-19 patients were admitted from January 2020 to March 2020 in Hefei Binhu Hospital, including 15 males and 18 females. The patients are aged between 13 and 71, with an average age of 37.45 ± 13.63 years. All the patients were tested positive for novel coronavirus nucleic acid by fluorescence reverse transcription-polymerasechain reaction (RT-PCR), and underwent at least one chest CT scan after treatment.
Instruments and methods
All patients received multislice spiral CT (SIMENS SOMATOM Definition and SIMENS Spirit) in supine position with advanced head scan, which should cover from the tip of the lung to the bottom of the lung. The pneumonia screening function of the AI image-assisted diagnosis system developed by IFLYTEK CO.,LTD. was used to analyze the data of 103 chest CT examinations. When the system showed "high proportion of abnormalities, suspected pneumonia", the patient was considered positive. Otherwise, the patient was deemed negative. At the same time, four senior radiologists would make a joint diagnosis based on their own diagnostic experience. They would reach a consensus on whether there were signs of pneumonia. Their diagnosis was regarded as the "gold standard".
Statistical methods
The measurement data is expressed as mean±standard deviation (). The counting data is expressed in relative number composition ratio (%). The sensitivity, specificity, misdiagnosis rate, missed diagnosis rate, positive predictive value, negative predictive value, positive likelihood ratio, negative likelihood ratio, detection rate and false alarm rate of a single lesion of COVID-19 are calculated by the AI image-assisted diagnostic system. The system also carried out dynamic analysis on the volume and density of the lesion, which can produce a detailed review assessment.
Detection efficiency analysis of AI image-assisted diagnosis system
All 103 chest CT scans were performed and screened by AI system. Among them, there were 88 true positive cases, 1 false positive case, 12 true negative cases and 2 false negative cases (Table 1 and Figure 1). The evaluation indexes of the AI image-assisted diagnostic system were as follows: sensitivity was 97.78% (88/90); specificity was 92.31% (12/13); positive predictive value was 98.88% (88/89); negative predictive value was 85.71% (12/14); accuracy was 97.09% (100/103); Youden index was 90.09%; positive likelihood ratio was 12.711 and negative likelihood ratio was 0.024.
Table 1 AI image-assisted diagnosis system and comparison of diagnosis results of 4 senior radiologists (case)
|
Results of AI image-assisted diagnosis system |
Diagnosis of 4 senior radiologists |
Total |
|
|
+ |
- |
||
|
+ |
88 |
1 |
89 |
|
- |
2 |
12 |
14 |
|
Total |
90 |
13 |
103 |
Basic conditions of pneumonia lesions in the AI image-assisted diagnosis system
Altogether 790 lesions were identified, including 569 true positive markers and 221 false positive markers. There were also 64 missed diagnosis markers. The detection rate of a single pneumonia lesion was 89.89%, and the rate of false positives was 27.97%. The average lesion volume was about (259.11±323.12) cm3, the percentage of lesions in the total lung volume was about (8.11±9.81) %, and the average lesion density was (-595.24±140.33) HU. The lesions were distributed in all lobes, and some lesions were marked across two or even three lobes (Table 2).
Table 2 Recognition of pneumonia lesions in the AI image-assisted diagnosis system
|
Imaging findings and diagnosis |
number |
Proportion (%) |
|
Lesion distribution |
|
|
|
superior lobe of right lung |
114 |
20.04 |
|
middle lobe of right lung |
55 |
9.67 |
|
inferior lobe of right lung |
143 |
25.13 |
|
superior lobe of left lung |
146 |
25.66 |
|
inferior lobe of left lung |
190 |
33.39 |
|
Features of lesion imaging |
|
|
|
ground glass opacity |
371 |
65.20 |
|
ground glass opacity with consolidation |
191 |
33.57 |
|
fibrosis |
18 |
3.16 |
|
air bronchogram |
18 |
3.16 |
|
pulmonary vessel thickening |
16 |
2.81 |
|
False positive result |
|
|
|
pulmonary vascular shadow |
114 |
51.58 |
|
pulmonary respiratory movement artifact |
55 |
24.89 |
|
pleural artifacts |
39 |
17.65 |
|
other organizational motion artifacts |
13 |
5.88 |
Application of AI image-assisted diagnosis system in reexamination of CT scan
Twenty-four COVID-19 patients underwent several chest CT examinations during hospitalization (Fig 2). The lesions of all patients eventually were absorbed, though some lesions had progressed in the course of disease. The changes of the mean lesion volume, the percentage of the lesion in the total lung volume and the mean density of the lesion in the first and last chest CT examination are shown in Table 3.
Table 3 Application results of AI image-assisted diagnosis system in review and evaluation
|
group |
Mean lesion volume (cm3) |
Lesion percentage of total lung volume (%) |
Mean density of lesions (HU) |
|
First CT |
5.72±7.31 |
189.75±246.66 |
-551.93±144.82 |
|
Last CT |
4.94±6.38 |
170.27±241.71 |
-614.78±162.59 |
|
difference |
-0.77±5.31 |
-19.49±126.02 |
-62.85±114.12 |
At current stage, the COVID-19 epidemic has affected the global social, economic and livelihood development. The prevention and control work of COVID-19 epidemic is very important. "Early detection, early report, early isolation, early diagnosis and early treatment" is the key to the prevention and treatment of COVID-19. As a routine examination of pulmonary lesions, CT plays an important clinical role in the early diagnosis and dynamic assessment of COVID-19[7].
CT image is a special 3D image, so its focus detection task also requires the output of 3D external compact box. For 3D data, 3D convolution has more advantages over traditional 2D convolution[8]. The convolution kernel of 3D convolution can be traversed in 3D space to obtain 3D spatial features, and it has a stronger ability to represent lesions, whereas 2D convolution can only extract features in a plane. This system takes 3D convolution as the basic unit when building the model. While improving representational ability, 3D convolution also raises the amount of computation. In order to ensure the training efficiency of the model and the real-time performance of the application, the number of channels in 3D convolution layer was reduced and the step size of the spatial pooling layer was increased to meet the requirement of efficiency when designing the model. The recognition and segmentation of COVID-19 lesions can be achieved through the deep learning of AI. The entire sequence of a CT examination was put into the detection and segmentation models respectively to obtain the pneumonia lesion detection box and the pneumonia lesion segmentation mask. The two sets of results were then fused to obtain the final detection results. On the basis of the detection and segmentation of lesions, the volume and average density of lesions can be obtained through cumulative calculation based on DICOM information, including pixel spacing, layer thickness and layer spacing.
During the operation of the system, the diagnostic physician can directly locate the lesion at the image level after selecting the lesion in the list of screening results. Based on the location and attributes of the lesions, the system can generate a report of its imaging manifestations. When the scope of the pneumonia lesions exceeds a specific threshold of the total lung volume, a warning of "high proportion of abnormalities and suspected pneumonia" will be issued. The diagnostic physician can also remove the false positive lesions in the results, and give a more realistic lesion size and volume.
During the study, the AI image-assisted diagnosis system performed 97.78% sensitivity and 92.31% specificity for the diagnostic efficiency of COVID-19. Its accuracy rate was up to 97.09%, proving its application value in the early diagnosis of COVID-19. In the chest imaging analysis of COVID-19, it was found that most lesions were multiple ground glass opacity in both lungs. Some lesions showed consolidation and fibrosis in the course of disease, which was consistent with the report of Salehi S et al[9].
Due to the wide distribution and rapid progression of most pneumonia lesions, the detection rate of the system for a single lesion was only 89.89%, and the false alarm rate was 27.97%. The 64 missed diagnosis markers were all small ground glass opacity. Although these lesions were not marked in the pneumonia list, they appeared in the nodule screening function list. This is related to the algorithm model and parameter setting of the system which can distinguish between nodules and pneumonia by the size of the lesions. With the continuous development of deep learning increasing clinical application data, the AI image-assisted diagnosis system will be continuously improved and upgraded[10].
Pulmonary vascular shadow and respiratory movement artifact accounted for more than 80% of the 221 misdiagnosed markers. This is related to the quality of the image data, which is affected by the parameter setting of image equipment and algorithm reconstruction. In this study, the slice thickness of most CT scans was 3mm, and the scanning time was about 45s. The breathing movement due to the patient's lack of breath control affected the image quality, thus affecting the recognition efficiency. Some studies suggested that thin CT slice will contribute to lesion recognition efficiency. The recommended thickness is 1.25mm[11]. Further clinical trials should be conducted to verify its specific impact.
Through comparing the CT reexamination of 24 patients, it is found that under the condition that all patients showed recovery during hospitalization, the lesions in the last CT scan all showed different degrees of absorption compared with the first scan, which is demonstrated in the reduction in lesion size and lesion percentage of total lung volume and the decrease in mean lesion density. The AI image-assisted diagnosis system is able to present the dynamic changes of the patient's CT results with higher accuracy and shorter response time. Therefore, image diagnostician and clinicians can efficiently assess the clinical efficacy of treatment, and provide standards as well as basis for disease evaluation, isolation removal and discharge. At the same time, the system can quickly process massive data and make rapid and accurate comparison among multiple reviews, which can reduce the workload of doctors and avoid the subjective bias of doctors.
In addition, 7 patients in this study were negative on chest CT. Three of them underwent two chest CT examinations. However, all the patients were tested positive by fluorescence RT-PCR with nucleonuclear acid of novel coronavirus. This indicates that negative chest CT results are not the absolute standard to exclude COVID-19. Therefore, suspected patients with negative CT results should be treated with caution in clinical practice to prevent missed diagnosis. The differential diagnosis of COVID-19 with other viral pneumonia, bacterial pneumonia and mycoplasma pneumonia is also a key and difficult area in clinical work, which will also be a new direction of AI image-assisted diagnosis system.
The AI image-assisted diagnosis system is conducive to improve the accuracy and efficiency of COVID-19 for diagnosticians. The system can be further promoted in primary hospitals with shortage in diagnosticians and extended to the diagnosis of more infectious diseases. A network diagnosis and early warning system can be built to further strengthen the disease prevention and control system. The AI medical image-assisted diagnosis system has certain clinical application value in the early diagnosis and review evaluation of COVID-19. However, due to the small sample size, there is still a large research space of this system.
Funding
This work was funded by COVID-19 prevention and control of emergency science and technology project of Hefei Municipal Bureau of Science and Technology (Project No. J2020YJ03).
Conflicts of interest/Competing interests
The authors declare that they have no conflict of interest.
Ethical approval
All dates performed in the study involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This research study was conducted retrospectively from data obtained for clinical purposes, approved by ethics committee of Hefei Binhu Hospital.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Consent for publication
All individuals consent to having their data published in a journal article.
Availability of data and material
All dates are reliable, transparent and accurate.
Code availability
Not applicable.