Development and Validation of a Novel Risk Score for Predicting Clinical Deterioration in COVID-19 Patients: The ABCD Risk Score

Kazuo Imai (  k_imai@saitama-med.ac.jp ) Saitama Medical University: Saitama Ika Daigaku https://orcid.org/0000-0002-8416-7371 Yutaro Kitagawa Saitama Ika Daigaku Byoin Sakiko Tabata Japan Ground Self Defence Force: Rikujo Jieitai Masaru Matsuoka Saitama Ika Daigaku Byoin Mayu Nagura-Ikeda Japan Ground Self Defence Force: Rikujo Jieitai Ai Fukada Saitama Ika Daigaku Byoin Kazuyasu Miyoshi Japan Ground Self Defence Force: Rikujo Jieitai Yoshiro Saito Kokuritu Iyakuhin Shokuhin Eisei Kenkyujo Norihito Tarumoto Saitama Ika Daigaku Takuya Maeda Saitama Ika Daigaku Byoin


Introduction
Novel coronavirus disease 2019 (COVID- 19), which is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), is an ongoing global pandemic (1). The scienti c community is actively exploring biomarker and prognostic models of COVID-19 deterioration and mortality to allow for the appropriate allocation of medical resources to patients who are at increased risk of disease progression. A crucial issue in clinical practice is identifying people who are at risk of clinical deterioration from those without symptoms as well as from patients with mild/moderate symptoms who do not require hospitalization and those with severe/critical symptoms who need oxygen therapy, systemic glucocorticoids (2), and anti-viral therapy (3).
To this end, we conducted a multicenter retrospective cohort study enrolling 751 hospitalized patients with COVID-19 in Japan to develop and validate a new simple and accurate model for predicting the deterioration of COVID-19 patients at the early stage (within 10 days) of symptom onset.

Study design and patients
The study design is shown in Fig. 1. We conducted a retrospective, multicenter cohort study at Saitama Medical University Hospital and Self-Defense Forces Central Hospital in Japan, which are designated medical institutions for infectious diseases under the Infectious Disease Control Law, Japan. We enrolled adult patients (≥ 18 years old) with con rmed COVID-19 by molecular diagnostic methods (quantitative reverse transcription polymerase chain reaction [RT-qPCR] or loop-mediated isothermal ampli cation [LAMP]) and hospitalized for isolation and treatment under the Infectious Disease Control Law. Patients who did not undergo routine blood examinations (complete blood count, serum biochemical tests, and coagulation tests) within 10 days of initial symptom onset were excluded from this study. We also excluded patients who were treated with oxygen therapy before hospitalization.
First, patients who were hospitalized from February to October 2020 were enrolled for a derivation dataset. Then, patients who were hospitalized from November 2020 to March 2021 were enrolled for a temporal validation dataset. In Japan, four waves of COVID-19 have occurred from February 2020 to May 2021. The patients admitted in the rst and second waves and in the third and fourth waves were included in the derivation and temporal validation datasets, respectively (Appendix 1). A comparison dataset for comparing risk scoring models included all patients admitted during the study period. Clinical information was retrospectively collected from the hospital electrical medical records and included clinical records and laboratory ndings. The primary outcome was in-hospital clinical deterioration within 14 days of hospitalization.

De nitions
Clinical characteristics and laboratory ndings at admission were used to derive and validate the risk scoring model. Clinical deterioration was de ned as administration of oxygen therapy with SpO 2 < 93% on room air during the hospitalization. The observation period was de ned as the period from patient's admission to patient's discharge or 14 days after the admission, whichever came rst. The day of initial symptom onset was de ned as the day of symptom appearance according to the patients or their family members. For asymptomatic patients, initial symptom onset was determined as the day of hospitalization. Disease severity was classi ed by a clinician with 8 years' experience in infectious disease physician (KI) according to the 8-category ordinal scale recommended by the World Health Organization (30).

Statistical analysis
Continuous variables are expressed as the mean and standard deviation or median and interquartile range (IQR) and were compared using a t-test or Wilcoxon rank-sum test for parametric or non-parametric data, respectively. Categorical variables are presented as frequency and percentage (%) and were compared using a chi-square test or Fisher's exact test, as appropriate. A twosided p value < 0.05 was considered statistically signi cant. All statistical analyses were conducted using R (v 4.0.2; R Foundation for Statistical Computing, Vienna, Austria; http://www.R-project.org/).

Candidate predictor selection and model development
Based on the literature, 12 candidate predictor variables were selected from clinical characteristics and potential biomarkers associated with clinical deterioration. Self-reported clinical symptoms were excluded for better objectivity. Values unavailable for at least 25% of the patients in the derivation dataset were also excluded. Finally, 9 candidate predictor variables-age, sex at birth, body mass index [BMI], comorbidities of diabetes mellitus and hypertension, NLR, BUN, LDH, and CRP-were selected for analysis by consensus at a team meeting during the derivation phase. There were no missing values for these 9 candidate predictor variables in the derivation dataset (Appendix 2).
The model building process for developing the risk score was conducted according to the method reported by Knight et al. (31) with minor modi cations. In the rst step, generalized additive model (GAM) t to a Cox regression models were built by incorporating continuous variables with P-spline smoothers in combination with categorical variables as linear components. A criterion-based approach to variable selection was applied based on the deviance explained and restricted maximum likelihood.
Second, optimal cutoff values for continuous variables were selected from visually inspected plots of component continuous variables with P-spline smoothers. Third, nal models using categorized variables were speci ed with least absolute shrinkage and selection operator (LASSO) Cox regression. L1-penalized coe cients were derived using 10-fold cross-validation to select the value of lambda (minimized cross-validated sum of squared residuals) in the derivation dataset. Shrunk coe cients were converted to a point with appropriate scaling to create the risk scoring model. Discrimination of the developed risk scoring model-named the Age, BMI, CRP, LDH [ABCD] Risk Score-was evaluated using the area under the receiver operating characteristic (ROC) curve and concordance statistics (C-statistics) in the derivation dataset. The 95% con dence interval (95% CI) of the C-statistics was calculated by bootstrapped resampling (2000 samples). Calibration of the ABCD Risk Score was assessed by using a calibration plot and Brier score.

Model validation
A temporal validation dataset of patients was used for validation of the ABCD Risk Score obtained in the derivation phase. The same clinical and laboratory data were available for analysis in both cohorts. There were no missing values for the ABCD Risk Score in the temporal validation dataset. Discrimination and calibration were evaluated in a validation dataset. The cutoff values of the ABCD Risk Score for three risk groups-low, intermediate, and high-were determined by consensus at a team meeting.
Kaplan-Meier survival curves for the patients in each risk group were generated to illustrate the partitioning of the risk of disease deterioration, and differences in clinical deterioration between risk groups were assessed by log-rank test.
Comparison with other risk scoring models of clinical deterioration in COVID-19 The ABCD Risk Score was compared within the comparison dataset with previously reported risk scoring models. Sixteen risk scoring models for clinical deterioration were extracted from the literature; 12 were excluded due to a lack of clinical symptoms, CT ndings, or ultrasound ndings in the comparison dataset (13-15, 17-19, 21-24, 27, 28). Finally, four risk scoring models were selected for evaluation in this study (16,20,25,26). Discrimination, calibration, and decision curve analysis of each risk scoring model was evaluated in the comparison dataset (Fig. 1). Because the rate of missing values was 20% for D-dimer in the comparison of the risk scoring models, the missing values were imputed by a random forest imputation method.

Results
Patients' characteristics in the derivation cohort Between February and October 2020, 636 patients with laboratory-con rmed COVID-19 were hospitalized at Saitama Medical University Hospital and Self-Defense Forces Central Hospital in Japan. A total of 190 people were excluded according to the exclusion criteria of the study, leaving 446 participants for the nal analysis (Fig. 1). The median patient age was 48 years (IQR, , and 265 (59.4%) were male at birth ( Table 1). The median period from initial symptom onset to serum collection was 4 days (IQR, 3-6). At the end of the observation period, 90 patients (20.2%) were con rmed to have had clinical deterioration (Table 1). Of these 90 patients, 20 (22.2%) were supplied oxygen by high-ow nasal cannula (HFNC) and noninvasive positive pressure ventilation (NPPV), 8 (8.9%) were incubated and treated with invasive mechanical ventilation (IMV), and 7 (7.8%) died. A statistical comparison of the nonclinical deterioration and deterioration groups con rmed that the 9 selected factors were associated with clinical deterioration (Table 1).  Total score --+ 12 The penalized coe cient was derived from a least absolute shrinkage and selection operator (LASSO) Cox regression model.

Model validation
The validation dataset included 305 patients referred to the study hospitals from November 2020 to March 2021 (Fig. 1). The median age of the patients in the cohort was 65 years (IQR, 47-79) and 183 (60.0%) were male at birth ( Table 1). The median period from initial symptom onset to serum collection was 5 days (IQR, 3-8). The overall disease deterioration rate was 39.3% (120 patients). The ABCD Risk Score showed good discrimination performance in the validation dataset (C-statistics, 0.86; 95% CI: 0.82-0.90; Fig. 2). The discrimination of the ABCD Risk Score was better than that of the single predictors: age (0.78, 0.69-0.80), BMI (0.51, 0.44-0.57), CRP (0.81, 0.76-0.85), and LDH (0.78, 0.72-0.83) (Fig. 2). A calibration plot of the ABCD Risk Score showed an intercept of 0.01 and slope of 0.99 (Brier score, 0.147), suggesting good calibration (Fig. 2) Table 3 and Fig. 3). Comparison of the ABCD Risk Score with other risk scoring models The ABCD Risk Score was compared with four previously reported risk scoring models-CALL score (16), N/L*CRP*D-dimer score (20), EWAS score (25), and HNC-LL score (26)-in the comparison dataset (Fig. 1). The overall disease deterioration rate was  Table 4 and Appendix 8). In addition, the calibration plot of the ABCD Risk Score showed good calibration, with the lowest Brier score of the risk scoring models (Table 4 and Appendix 8). Decision curve analysis indicated that the ABCD Risk Score had better clinical utility across a wide range of threshold risks than the other risk scores (Fig. 4).

Discussion
The results of this retrospective multicenter cohort study revealed that our simple ABCD Risk Score has good performance for the risk strati cation of clinical deterioration in COVID-19 patients at an early stage of symptom onset (≤ 10 days).
In clinical practice, risk strati cation of the clinical deterioration of COVID-19 patients is paramount because the need for oxygen therapy in patients is strongly associated with decision-making regarding hospitalization and systemic treatment with dexamethasone (2) and remdesivir (3) to decrease mortality. Therefore, all risk scoring models for the risk strati cation of clinical deterioration should be simple and based on rapid tests that can be performed in both in-hospital and outpatient clinic settings. The ABCD Risk Score uses only clinical characteristics and routine laboratory tests that can be collected rapidly and that have been implicated in previous studies as potential risk factors for the clinical deterioration and mortality of COVID-19. Older age has been included in many models for predicting COVID-19 and pneumonia mortality (6-8, 31,32). A higher BMI has been strongly associated with the clinical deterioration and mortality of COVID-19 (33,34). CRP is widely used as a marker of in ammation in the clinical setting. CRP is secreted into the circulation by the liver in response to circulatory in ammatory mediators such as IL-6, and elevated serum CRP levels re ect the clinical activity of pneumonia (9, 10) and cytokine storm of COVID-19 (35). Elevated serum LDH levels are associated with lung tissue damage (36), and serum LDH is considered a marker of disease activity and progression in COVID-19-related pneumonia (37). Risk scoring models with the same prediction variables and weight as the ABCD Risk Score have not been reported. The ABCD Risk Score showed higher discrimination and calibration performance than other risk scoring models that used clinical characteristics and routine laboratory tests (16,20,25,26). In addition, decision curve analysis determined that the ABCD Risk Score had better clinical utility than the other risk scores. Because of its simplicity and performance, the ABCD Risk Score can be a broadly applicable tool in both in-hospital and outpatient clinics, even in regions with limited medical resources.
A previous large-scale multicenter registry study that enrolled inpatients at health care facilities from January to July 2020 in Japan with a median age of 52 years (IQR, 34-68) determined a clinical deterioration rate of 27.8% (1443 of 5194 patients) (38).
The derivation dataset seems to be representative of the target population. However, the clinical deterioration rate and age distribution were higher in our temporal validation datasets than in the above registry study and in our derivation dataset ( Table 1).
The difference in clinical characteristics between the temporal validation and derivation datasets may be due to sampling bias. Local governments are responsible for the assignment of all patients with COVID-19 to health care facilities. A higher number of patients with COVID-19 per day was evident during the time period of the temporal validation dataset than during that of the derivation dataset (Appendix 1). So, it is possible that the local government preferentially assigned patients at risk of clinical deterioration, such as elderly patients, to hospitals during the period of the temporal validation dataset rather than during that of the derivation dataset. The ABCD Risk Score showed good discrimination and calibration performance in both datasets, and therefore it may have the potential to cover people with a wide range of clinical deterioration rates.
This study has several limitations. The head-to-head comparison between the ABCD Risk Score and existing risk scoring models used clinical signs and CT imaging that were not included in this study because of missing values from datasets. In addition, the vaccination strategy for COVID-19 has been started worldwide and several studies have shown that vaccination can reduce the severity and mortality of COVID-19 (39,40). The ABCD Risk Score does not consider the effects of vaccination because COVID-19 vaccination was not available during our study period in Japan. Finally, the ABCD Risk Score was developed and validated in a dataset including patients from a single country. Although the clinical utility of the ABCD Risk Score will need to be assessed in an externally validated implementation study prior to multicountry adoption in routine practice, the ABCD Risk Score has the potential to individualize patients with COVID-19 through optimal risk strati cation of clinical deterioration.    points; and high-risk group, ≥7 points.