External Validation of the 4C Mortality Score among COVID-19 Patients Visiting the Emergency Department or admitted to Hospital in Ontario, Canada

Objectives: Risk prediction scores are important tools to support clinical decision-making for patients with coronavirus disease (COVID-19). The objective of this paper was to validate the 4C mortality score, originally developed in the United Kingdom, for a Canadian population. Methods: We conducted an external validation study within a registry of COVID-19 positive emergency department visits and hospital admissions in the Kitchener-Waterloo and Hamilton regions of southern Ontario between March 4 and January 9, 2020. We examined the validity of the 4C score to prognosticate in-hospital mortality using the area under the receiver operating characteristic curve (AUC) with 95% condence intervals calculated via bootstrapping. Results: The study included 560 individuals, of whom 115 (20.5%) died in-hospital. Median age was 69 years and 281 individuals (51%) were male. The AUC of the 4C score was 0.83, 95% condence interval 0.79-0.87. Mortality rates across the pre-dened risk groups were 0% (Low), 3.2% (Intermediate), 25.9% (High), and 59.5% (Very High). The AUC was 0.80 (0.76-0.85) among hospital inpatients. Interpretation: The 4C score is a valid tool to prognosticate mortality from COVID-19 in Canadian emergency departments and hospitals.


Introduction
Coronavirus disease (COVID-19) caused by the Severe Acute Respiratory Syndrome Virus 2 (SARS-CoV-2) can progress to acute respiratory distress syndrome, multiorgan failure, and death in some individuals 1 . The COVID-19 infection-fatality risk varies widely with age, ranging from < 0.1% for children and younger adults to greater than 10% for adults 75 and older 2 . The clinical presentation and progression of COVID-19 in patients is highly variable 3 , which makes it di cult for clinicians to triage patients and determine their risk of poor outcomes. A simple, validated prognostic tool utilizing data that is available at presentation can help clinicians better prognosticate and make clinical decisions. Numerous tools to predict mortality in COVID-19 patients have been developed, but many are limited due to small derivation cohort sizes and/or inadequate validation 4 .
The 4C mortality score is an accessible risk strati cation score developed by the International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) 5 . It was derived and internally validated on a large, diverse cohort within the United Kingdom but needs external validity to con rm its generalizability. The objective of this study was to validate the ability of the 4C score to prognosticate mortality in COVID-19 patients presenting to the emergency department or admitted to hospital in Ontario, Canada.

Study design and setting
We conducted a validation study using records from the McMaster Multi-Regional Hospital Coronavirus Registry (COREG). COREG is a multi-center data registry collecting information on positive COVID-19 cases in the Kitchener-Waterloo and Hamilton regions of southern Ontario, Canada. The registry includes COVID-19-related emergency department (ED) visits and hospital admissions from six hospitals across the regions, including both academic and community centres.

Participants
We selected all patients admitted up to four weeks before time of analysis, which included admissions between March 4th, 2020 and January 7 th , 2020 (n=596). We excluded patients with no clinical data collected at presentation (n=36), for a nal sample size of 560.

Measures
Data in COREG were abstracted from electronic medical records using a modi ed case report form published by ISARIC and the World Health Organization. We utilized demographic and clinical data at presentation, typically from the emergency department. For patients who were directly admitted, we utilized the rst day of inpatient records.

4C mortality score
The 4C mortality score was derived and validated within the ISARIC World Health Organization Clinical Characterisation Protocol UK study 6 . The score was derived from a population of over 35,000 hospital inpatients validation on over 22,000 inpatient records indicated good discriminability (area under the receiver operating characteristic curve (AUC)=0.77) 5 .
The 4C score incorporates age, sex, comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale, blood urea nitrogen, and C-reactive protein. We adapted the score to match our available data as the Glasgow Coma Scale was not collected at presentation we replaced this risk factor with the documented presence of altered consciousness or confusion (Supplementary Table S1). The 4C score ranges from 0 to 21 with risk groups de ned as Low (0-3), Intermediate (4-8), High (9)(10)(11)(12)(13)(14), and Very high (≥15).

Outcome
The primary outcome was in-hospital mortality.

Statistical Analysis
We validated the 4C score using the AUC, with 95% con dence intervals calculated via bootstrapping with 2000 resamples. Missing data was treated using multiple imputation with chained questions.
We plotted the proportion of patients who died in hospital by 4C score and risk group and compared them to the initial derivation work in the UK. As a sensitivity analysis, we also examined the discriminability of the 4C for only hospital inpatients (n=484). All analysis was done in R 4.0.3. 7

Ethics approval
Our study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board, who waived the requirement for informed consent as the data for this study was retrospectively collected from hospital medical records. All methods were performed in accordance with relevant guidelines and regulations.

Results
Our study included 560 patients, of which 484 (86%) were admitted to hospital, and 115 (21%) died (Table 1). Median age was 69 with patients split equally between males (51%) and females. After imputation, the low risk group contained 13.5% of patients, the intermediate group contained 25.3%, the high risk group contained 49.4% and the very high risk group contained 11.6%.

Interpretation
We found that the 4C mortality score is a valid tool to prognosticate mortality among COVID-19 patients presenting to the emergency department or upon admission to hospitals in a Canadian population. We observed an AUC of 0.83 in our data, which is higher than was reported in the original derivation research in the United Kingdom but similar to validation conducted in Europe 8 .
Clinical risk prediction models are critical tools that can be used to identify and prioritize high-risk patients. The proliferation of COVID-19 risk models is evidence of the demand for an accurate, accessible, and generalizable tool 9 . Our nding that the 4C mortality score is valid in a jurisdiction distinct from the location of the original derivation and subsequent validation provides evidence for the generalized prognostic ability of the score.
We have demonstrated that it is an informative clinical tool within academic and community hospital care settings in Ontario, which are similar to care settings across Canada.
The AUC we observed was higher than in the original research, which is unusual for external validation, but this was also true in other validation studies done in the United Kingdom and the Netherlands 8, 10 .
The higher discriminability in our study may in part because we examined a more heterogeneous cohort that included patients who visited the emergency department but were not admitted to hospital. The 4C score was likely able to identify that these patients were at low risk and assign them lower scores. Overall, the AUC was only slightly lower in the sensitivity analysis that only examined hospital inpatients and inclusion of outpatients into the validation population adds further value to the study.

Limitations
A key limitation of our study is that we were only able to include data from two regions within southern Ontario. While similarities between health systems across Canada suggest our ndings will have excellent generalizability to other Canadian provinces and territories, our results may not generalize to geographically remote settings or to jurisdictions with substantially different health systems.

Conclusion
The 4C mortality score is an adaptable, valid prognostic tool for use in Canadian emergency departments and hospitals. It can be used to identify and prioritize care for COVID-19 patients at high risk of death.

Funding Statement
This study leveraged data from the COREG registry, which is supported by a grant from the Canadian Institutes of Health Research (CIHR) (172754) and from the Hamilton Academic Health Sciences Organization (HAHSO) (HAH-21-04).

Data Sharing Statement
The data used in this study can be accessed for research purposes by submitting a request through the COREG data access portal. (https://www.coregontario.ca/info-data-access)

Code availability
Code is available upon request

Con icts of interest
None