COVID-19 Pneumonia: Inter-Observer Agreement and Diagnostic Accuracy of Chest CT in Patients with Intermediate Clinical Probability

Objectives To assess inter-reader agreements and diagnostic accuracy of chest CT to identify COVID-19 pneumonia in patients with intermediate clinical probability during an acute disease outbreak. Methods: From March 20 to April 8, consecutive patients with intermediate clinical probability of COVID-19 pneumonia underwent a chest CT scan. Two independent chest radiologists blinded to clinical information and RT-PCR results retrospectively reviewed and classied images on a 1-5 condence level scale for COVID-19 pneumonia. Agreements between radiologists were assessed with kappa statistics. Diagnostic accuracy of chest CT compared to RT-PCR assay and patient outcomes was measured using receiver operating characteristics (ROC). Positive predictive value (PPV) and negative predictive value (NPV) for COVID-19 pneumonia were calculated. Results:

If de nite diagnosis of COVID-19 infection requires real-time reverse transcription polymerase chain reaction (RT-PCR) of viral nuclei acids, chest CT scan has proved to be of clinical importance and the main tool for screening and primary diagnosis [3][4][5][6][7][8][9], bearing in mind that RT-PCR results can be falsely negative [10][11][12]. The Fleischner Society recently recommended the use of imaging -including chest X ray and CT scan-in patients with suspected COVID-19, regardless of the results or availability of RT-PCR testing, given the impact of imaging in both circumstances. In a resource constrained environment where access to CT is limited, chest X ray may be preferred for patients with COVID-19 unless features of respiratory worsening warrant the use of CT [13]. A recent meta-analysis by Kim et al. showed that the diagnostic value of chest CT depends on the prevalence of COVID-19 infection in the studied population [14]. In areas where the prevalence is low, chest CT screening of patients with suspected disease has a low positive predictive value [14]. On the other hand, in case of epidemic surge of patients at the emergency department, the clinicians will face a di cult challenge of rapid triage depending on disease presentation and severity. Patients with typical clinical symptoms and bilateral radiographic opacities may be hospitalized without a diagnostic CT scan. Patients presenting with fever and mild or no respiratory symptoms and a normal chest radiograph may leave the hospital after RT-PCR testing without chest CT scan and with surveillance by phone call during the following days. The patients for whom the diagnosis is unclear, represent the group with intermediate probability and may bene t from a chest CT scan, looking for evidence of COVID versus other pathologies.
The objective of this study was to evaluate the inter-observer agreement and diagnostic accuracy including positive and negative predictive values of chest CT to identify COVID-19 pneumonia in patients with intermediate clinical probability during an acute disease outbreak in a European country.

Materials And Methods
This retrospective monocentric study was approved by the Ethics Committee of Foch Hospital a liated to Paris Ouest University, and the requirement for informed consent was waived.

Patients
Three hundred and nineteen consecutive patients examined at the Emergency department between March 20 and April 8 2020 underwent a CT scan of the chest because of intermediate clinical probability of COVID-19 pneumonia. Intermediate clinical probability was de ned by typical or atypical clinical presentation (no typical nding at auscultation despite respiratory symptoms), but a normal or equivocal chest X ray. In the same period of time, 225 patients were hospitalized without a CT scan because of typical clinical and radiographic ndings of COVID-19 pneumonia, and 645 patients were not on the basis of clinical evaluation and chest X ray.

CT scans
A CT room was fully dedicated for patients suspected of having COVID-19 accessible from the Emergency department through a well delineate route. CT gantry and every potentially contaminated surface was disinfected after examination. Unenhanced low dose volumetric acquisition was performed over the entire chest at full inspiration in all patients. Contrast injection was used in 53 patients because of suspicion of pulmonary embolism associated with COVID-19. Contiguous 0.6 mm axial images were reconstructed with high resolution and soft tissue kernels.

Radiologist readings
Two independent chest radiologists (ALB and PAG) with 15 and 30 years of experience reviewed CT examinations being blinded for RT-PCR results and any clinical information except patient sex and age. They read the images using lung window and mediastinal settings on the PACS system (GE).
The radiologists were asked to classify the CT scans as: high probable, probable, low probable or improbable diagnosis of COVID-19 pneumonia, or normal. They established their diagnosis as high probable and probable on the basis of the recent publications from China illustrating typical and atypical patterns observed in patients with COVID-19 pneumonia [4][5][6][7][8][9][15][16][17][18][19]. Low probable diagnosis was de ned by the presence of parenchymal abnormalities, which did not t with the diagnosis of COVID-19 pneumonia nor with any other alternative diagnosis. Improbable diagnosis included typical patterns of infective or non infective lung diseases that might explain the patient's symptoms (ie. lobar pneumonia , bronchopneumonia, infectious bronchiolitis, infected bronchiectasis, interstitial lung disease, heart failure or malignant tumor). Pulmonary emboli was considered an alternative diagnosis if not associated with any parenchymal abnormalities suggestive or COVID. If associated with evidence of COVID, the CT was considered high probable or probable depending on the associated pattern. A normal chest CT scan was the last possible option, keeping in mind it does not exclude early COVID-19 pneumonia.

RT-PCR
Viral ARN was detected in nasal swabs. The real-time reverse transcription-polymerase chain in reaction (RT-PCR) of viral nucleic acid was used as the reference standard for the diagnosis of COVID-19 infection. In 296 patients RT-PCR assay was performed in our hospital by nasal swab with a technique based on a reagent containing three viral targets (PCR Seegene Allplex. Eurobio). In ten patients RT-PCR was performed in a different laboratory before patients arrived on site. In 13 patients RT-PCR was not available because of an alternative diagnosis suggested by CT scan or because the patient was transferred to another hospital.

Statistical analysis
Concordances and discordances between the two radiologists for the diagnosis of COVID-19 pneumonia were assessed using kappa statistics. The values of kappa can range from +1 (almost perfect agreement) to -1, where 0 represents the amount of agreement than can be expected from random chance [20].
The result of RT-PCR was used as the ground truth for measuring CT diagnostic accuracy. Because of the limited sensitivity of RT-PCR reported in the literature, clinical management and outcomes of patients with a negative RT-PCR and a CT scan classi ed by both radiologists as «high probable or probable» for COVID-19 pneumonia were reviewed. On the basis of this analysis, patients were considered as de nite COVID negative or reclassi ed COVID positive.
Receiver operating characteristics (ROC) was plotted and the area under the curve (AUC) was calculated with 95% con dence intervals for each radiologist.
Sensitivity, speci city, positive predictive value (PPV) and negative predictive value (NPV) were calculated for both radiologists PPV and NPV were calculated with the following formula [21]: The concordances and discordances between radiologists to classify the patients using the 5-con dence level scale for diagnosing COVID-19 pneumonia are reported in Table 1 with a kappa value of 0.61 (p< 0.01). The concordances between radiologists to consider CT patterns as high probable of COVID-19 pneumonia were observed in 127 of 319 (39.8%) patients with a kappa value of 0.83 (p <.001). The concordances between radiologists to consider CT pattern as high probable or probable of COVID-19 pneumonia were observed in 159 (49.8%) patients with a kappa value of 0.82 (p <.001).
The diagnostic accuracy was measured in 307 patients having available RT-PCR test results that were positive in 173 patients and negative in 134, leading to a disease prevalence in the population of 56.4% (173/307). The radiologists' rankings and RT-PCR results are reported in Table 2 (Figure 1).
The review of clinical data and outcomes for 16/307 (5.2%) patients classi ed high probable or probable by both radiologists and having a negative RT-PCR result allowed to reclassify them as true positive cases. This lead to increase the disease prevalence to 61.6% (189/307) and the AUC values for radiologists were 0.94 (IC95%: [0.91-0.97]) and 0.92 (IC95%: [0.89-0.95]) respectively ( Figure 2). Sensitivity, speci city, PPV and NPV for both radiologists are reported in Table 3. With a 61.6% prevalence of COVID-19 infection in the studied population, the PPV and NPV of chest CT for diagnosing COVID-19 pneumonia was 96.6% and 84.3% respectively for the rst radiologist and 94.4% and 78.2% for the second radiologist.

Discussion
Our study showed a very good inter-observer agreement and diagnostic accuracy of chest CT for the diagnosis of COVID-19 pneumonia in patients with intermediate clinical probability during an acute COVID-19 outbreak. This con rms the results by Prokop et al. who obtained substantial inter-observer agreement and very good performance for predicting COVID-19 in patients with moderate to severe symptoms using a categorical assessment scheme (CO-RADS) [22].
One limitation of our study could have been to use strictly the result of RT-PCR as ground truth with the risk of underestimation of CT diagnostic accuracy. The retrospective analysis of outcomes observed in our patients classi ed by both radiologists with a high probable or probable diagnosis at CT despite negative RT-PCR result permitted to reclassify them as COVID-19 positive. Actually, RT-PCR has demonstrated a limited sensitivity for diagnosing COVID-19 infection [10][11][12]. In a series by Fang et al. including 51 patients with chest CT and RT-PCR assay performed within three days the sensitivity of CT for COVID-19 infection was 98% compared to RT-PCR sensitivity of 71% [12]. In a larger series by Ai et al., 59% of 1014 patients had positive RT-PCR results, and 88% had positive (highly probable or probable) chest CT scan, and 30% of patients had negative RT-PCR and positive CT [11]. In our series, the percentage of patients presenting with a highly suggestive or suggestive CT scan according to both radiologists despite a negative RT-PCR was lower than previously reported. This difference might be related to the use of differences in quality of reagent used for RT-PCR technique, given the great majority of our patients were tested using a reagent able to assess three viral targets. In addition, according to Zou et al. [23], viral load in our study was detected in nasal swabs rather than in throat swabs like in the studies by Xie et al. [10] and Ai et al. [11].
On the other hand, a negative result of RT-PCR could be associated with a non SARS-CoV-2 viral pneumonia. In a study by Bai et al., radiologists had high speci city and moderate sensitivity in differentiating COVID-19 from other viral pneumonias on chest CT [24]. In their study, the most discriminating features for COVID-19 pneumonia included a peripheral distribution, ground glass opacity and vascular thickening, ndings that t very well with the criteria we used to classify our patients as high probable diagnosis of COVID-19 pneumonia.
The main limitation of our study was to be retrospective on data collected during the ascending phase of outbreak. The results depend totally on the prevalence of the COVID-19 in our population and consequently are not generalizable. Actually, the results of our study would have been very different if performed during the descending phase when the prevalence of disease dropped down. In a recent report, Eng and Bluemke emphasized in the context of COVID-19 pandemic, the need to move beyond sensitivity and speci city to predictive values according the differences in disease prevalence between studied populations [21]. The Fleischner society COVID-19 consensus statements acknowledged differences in patient populations and provide guidelines strati ed according to disease risk and severity [13].
Despite recent recommendations there is no consensus yet upon the integrated use of chest radiography or CT in case of massive arrival of patients suspected of COVID-19 pneumonia [25]. Our strategy was based on the principle that chest radiography remains helpful at the rst-line evaluation of patients with a high clinical probability of overt COVID-19 pneumonia. Because it allows detecting subtle diffuse ground glass opacities that are variably detected by radiography, CT scan is helpful for the management of patients with equivocal clinical symptoms and chest radiographs.
To our knowledge this is the rst study to evaluate agreements between radiologists and diagnostic accuracy of chest CT in patients selected on the basis of an intermediate clinical probability during an acute phase of outbreak.
In conclusion, during an acute COVID-19 outbreak CT scan may be used in the context of rst-line dedicated triage of patients presenting an intermediate clinical probability with very good inter-observer concordances and diagnostic accuracy. Declarations 2. Con ict of Interest: The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.

Statistics and Biometry:
One of the authors (Julie Trichereau) has signi cant statistical expertise.

Informed Consent:
Written informed consent was waived by the Institutional Review Board.

Ethical Approval:
Institutional Review Board approval was obtained.

Cohorts overlap:
Chest CT scans of two hundreds patients from the three hundred and nineteen patients included in this study were also used after removal of identifying details, to train or validate a deep learning-based tool for automatic quanti cation of abnormalities associated with COVID. The article describing the development of this new method has been submitted to Radiology: Arti cial Intelligence and is currently under revision. The radiologists who analyzed the CT scans included in the presently submitted article did not participate to the selection of the cases used for the development of the automatic quantitative tool.    CT scan of a 46 yo male patient with negative RT-PCR test but classi ed as high probable diagnosis of COVID-19 pneumonia by both radiologists. Bilateral and peripheral areas of ground glass opacities within the upper lobes ( gure_4a.tif, gure_4b.tif) and in the right lower lobe ( gure_4c.tif, gure_4d.tif).

Figure 5
CT scan of a 34 yo female patient with positive RT-PCR test classi ed as probable diagnosis of COVID pneumonia by one radiologist and as improbable diagnosis (bronchopneumonia) by the other. Presence of two rounded areas of ground glass opacities (yellow arrows) within the right upper lobe ( gure_5a.tif) and multiple small nodular opacities of ground glass attenuation disseminated in the right lower lobe ( gure_5a.tif, gure_5b.tif, gure_5c.tif) and to a lesser extent the left upper lobe ( gure_5a.tif). Bronchial wall thickening, tree-in-bud sign and subpleural focal area of consolidation are also visible in the left lower lobe ( gure_5c.tif).

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.