Study design
This prospective-retrospective observational multicentric study was approved by the French Ethics Committee for the Research in Medical Imaging (CERIM) review board (IRB CRM-2007-107) according to good clinical practices and applicable laws and regulations. The written informed consent was waived due to the nature of the analysed data, which were anonymized healthcare data. All methods were performed in accordance with the relevant guidelines and regulations. The need for written informed consent was waived because of its retrospective nature.
IMADIS Teleradiology is a French company dedicated to remote interpretation of emergency CT and MRI examinations. As of March 2020, IMADIS Teleradiology had partnerships with the emergency and radiological departments of 69 hospitals covering all French regions. The panel of IMADIS teleradiologists consisted of 109 senior radiologists with at least 5 years of emergency imaging experience (mean length of practice, 7 years) and 55 junior radiologists (i.e., residents) with 3–5 years of emergency imaging experience (mean length of practice, 4 years). Teleradiologists were on-call in groups of at least two teleradiologists per night in each of the two interpretation centres (Bordeaux and Lyon, France). All radiological reports involving COVID-19 made by junior teleradiologists were collegially validated.
Our study included all consecutive adult patients from 03/13/2020 to 04/14/2020 from 15/69 (21.7%) partner hospitals that regularly provided the RT-PCR results to IMADIS, as these patients fulfil the following inclusion criteria: need for chest CT due to suspicion of COVID-19 according to a board-certified emergency physician, available chest CT, and available RT-PCR status (Figure 1, Supplemental Data 1).
Teleradiological Interpretation Protocol
The IMADIS teleradiology interpretation protocol met the French recommendations for teleradiology practice 15. Reports and requests with clinical data for COVID-19 Chest CT image interpretation were received from the client hospitals at the IMADIS Teleradiology centres by using teleradiology software (ITIS; Deeplink Medical, Lyon, France). The images were securely transferred over a virtual private network to a local picture archiving and communication system for interpretation (Carestream Health 12, Rochester, NY). Images were immediately interpreted by OCTRs.
CT examinations were systematically reviewed within one week following each on-call session by another senior teleradiologist (n = 15; mean length of practice, 12.1 years; mean number of reviews, 34 CTs) who was not involved during the on-call duty period, blinded to the RT-PCR result and the first reader’s report. All senior radiologists had a 2-hour-long e-learning session on CT-Chest findings in COVID-19, which became publicly available on April 7 (Web-based e-learning, developed by IMADIS Radiologists, Deeplink Medical, Lyon, France and RiseUp, Paris, France: https://covid19-formation.riseup.ai/).
Clinical Data
Clinical information was prospectively provided by emergency physicians and collected through the teleradiology software in a standardized COVID-19 CT request form (ITIS; Deeplink Medical, Lyon, France), as follows: age; gender; active smoking; medical history, recent anti-inflammatory drugs intake; delay since onset of symptoms (categorized as: <1 week, 1-2 weeks, ≥ 2 weeks); oxygen saturation (categorized as: ≥ 95%, 90-95% and < 90%); dyspnoea; fever (³ 38°C); cough; asthenia; headache; and ear, nose and throat symptoms.
The RT-PCR results from throat swab samples contemporary of the emergency room visit were retrospectively collected from the patients’ electronic medical records from each partner hospital.
Radiological Data
At the end of the report, the OCTR had to propose a conclusion adapted from the SFR classification, as follows: (1) normal, (2) abnormalities inconsistent with pulmonary infection; (3) abnormalities consistent with a non-COVID-19 infection; (4) indeterminate/compatible abnormalities; and (5) findings strongly suspicious of COVID-19.
The 2nd reading assessed the following radiological features: (a) underlying pulmonary disease (categorized as: emphysema, lung cancer, interstitial lung disease, pleural lesions, or bronchiectasis); (b) GGO pattern (categorized as: rounded or non-rounded GGO); (c) consolidation pattern (categorized as: rounded or non-rounded consolidations and fibrotic bands); (d) predominant pattern (categorized as: GGO or consolidation); (e) distribution pattern of lesions (categorized as: peripheral predominant, central predominant, or mixed); (f) bilateral lesions; (g) diffuse lesions (i.e., five lobes involved); (h) basal predominant lesions; (i) pleural effusion (categorized as: uni- or bilateral); (j) adenomegaly (defined as lymph node with short axis > 10 mm); (k) bronchial wall thickening (further categorized as lobar/segmental or diffuse); (l) airway secretions; (m) tree-in-bud micronodules, and (n) pulmonary embolism. Images for each radiological feature can be found in Supplemental Data 2.
Statistical Analysis
Statistical analyses were performed using R (version 3.5.3, R Foundation for Statistical Computing, Vienna, Austria). A p-value of less than 0.05 was deemed significant.
Univariate associations between clinical-radiological categorical variables and RT-PCR status were evaluated with Pearson Χ2 or Fisher exact tests, except for age, which was compared between the two groups with Student’s t-test. Classification and regression models are negatively affected by high correlations between explanatory variables; hence, correlations between variables were evaluated with Spearman’s test. For each significantly correlated pair of dummy variables extracted from the same initial multilevel categorical variable, the variable with the lowest p-value at univariate analysis was selected for the multivariable modelling.
Next, the study population was randomly partitioned into a training cohort (n = 412/513, ≈ 80%) and a validation cohort (n = 101/513, ≈ 20%), with a same prevalence of RT-PCR positivity. We focused on two simple classifiers that do not require any computing interface to extract the probability for a positive RT-PCR, namely: classification and regression tree (CART, “rpart” package) and stepwise backward-forward binary logistic regression (Step-LR – minimizing the Akaike information criterion, “MASS” package). The models were built on the training cohort based on (i) either all dichotomized radiological variables or (ii) all dichotomized clinical + radiological variables - with a p-value < 0.05 at univariable analysis. The CART algorithm has a hyperparameter (i.e., a parameter that is set before the model building, while classical parameters are derived during the model building), named ‘complexity’, which controls the size of the tree and was selected following a cross-validation step in the training cohort as minimizing the classification error rate. Next, the tree was pruned following this optimal complexity hyperparameter. The minimal number of observations in the terminal node and the splitting criteria were set to 3 and the Gini index, respectively.
Models were evaluated and compared between themselves and the prospective conclusions made by the OCTRs on the validation cohort, according to AUC. Accuracy (number of correctly classified patients divided by the total number of patients), sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV) were estimated after dichotomizing predicted probabilities per a cut-off of 0.5. All results were given with a 95% confidence interval (95%CI). AUCs were compared using the pairwise Delong test (‘pROC’ package).
Finally, we applied a decision curve analysis (DCA) to assess the clinical usefulness of the final models in the validation cohort. DCA consists of plotting the net benefit of applying the model for clinically reasonable risk thresholds compared with two alternative strategies: (i) to treat all patients as affected by COVID-19 or (ii) to treat none of the patients 16. Herein, the net benefit of our models refers to the correct identification of patients with a positive or a negative RT-PCR, and the risk threshold can be seen as the harm-to-benefit ratio or the risk at which patients are indifferent about COVID-19 17. Hence, a low risk threshold would correspond to patients who are particularly worried about the disease 18.