DOI: https://doi.org/10.21203/rs.3.rs-268410/v1
Objectives: Risk prediction scores are important tools to support clinical decision-making for patients with coronavirus disease (COVID-19). The objective of this paper was to validate the 4C mortality score, originally developed in the United Kingdom, for a Canadian population.
Methods: We conducted an external validation study within a registry of COVID-19 positive emergency department visits and hospital admissions in the Kitchener-Waterloo and Hamilton regions of southern Ontario between March 4 and January 9, 2020. We examined the validity of the 4C score to prognosticate in-hospital mortality using the area under the receiver operating characteristic curve (AUC) with 95% confidence intervals calculated via bootstrapping.
Results: The study included 560 individuals, of whom 115 (20.5%) died in-hospital. Median age was 69 years and 281 individuals (51%) were male. The AUC of the 4C score was 0.83, 95% confidence interval 0.79-0.87. Mortality rates across the pre-defined risk groups were 0% (Low), 3.2% (Intermediate), 25.9% (High), and 59.5% (Very High). The AUC was 0.80 (0.76-0.85) among hospital inpatients.
Interpretation: The 4C score is a valid tool to prognosticate mortality from COVID-19 in Canadian emergency departments and hospitals.
Coronavirus disease (COVID-19) caused by the Severe Acute Respiratory Syndrome Virus 2 (SARS-CoV-2) can progress to acute respiratory distress syndrome, multiorgan failure, and death in some individuals 1. The COVID-19 infection-fatality risk varies widely with age, ranging from < 0.1% for children and younger adults to greater than 10% for adults 75 and older2. The clinical presentation and progression of COVID-19 in patients is highly variable3, which makes it difficult for clinicians to triage patients and determine their risk of poor outcomes. A simple, validated prognostic tool utilizing data that is available at presentation can help clinicians better prognosticate and make clinical decisions. Numerous tools to predict mortality in COVID-19 patients have been developed, but many are limited due to small derivation cohort sizes and/or inadequate validation4.
The 4C mortality score is an accessible risk stratification score developed by the International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) 5. It was derived and internally validated on a large, diverse cohort within the United Kingdom but needs external validity to confirm its generalizability. The objective of this study was to validate the ability of the 4C score to prognosticate mortality in COVID-19 patients presenting to the emergency department or admitted to hospital in Ontario, Canada.
We conducted a validation study using records from the McMaster Multi-Regional Hospital Coronavirus Registry (COREG). COREG is a multi-center data registry collecting information on positive COVID-19 cases in the Kitchener-Waterloo and Hamilton regions of southern Ontario, Canada. The registry includes COVID-19-related emergency department (ED) visits and hospital admissions from six hospitals across the regions, including both academic and community centres.
We selected all patients admitted up to four weeks before time of analysis, which included admissions between March 4th, 2020 and January 7th, 2020 (n=596). We excluded patients with no clinical data collected at presentation (n=36), for a final sample size of 560.
Data in COREG were abstracted from electronic medical records using a modified case report form published by ISARIC and the World Health Organization. We utilized demographic and clinical data at presentation, typically from the emergency department. For patients who were directly admitted, we utilized the first day of inpatient records.
The 4C mortality score was derived and validated within the ISARIC World Health Organization Clinical Characterisation Protocol UK study6. The score was derived from a population of over 35,000 hospital inpatients validation on over 22,000 inpatient records indicated good discriminability (area under the receiver operating characteristic curve (AUC)=0.77)5.
The 4C score incorporates age, sex, comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale, blood urea nitrogen, and C-reactive protein. We adapted the score to match our available data as the Glasgow Coma Scale was not collected at presentation we replaced this risk factor with the documented presence of altered consciousness or confusion (Supplementary Table S1). The 4C score ranges from 0 to 21 with risk groups defined as Low (0-3), Intermediate (4-8), High (9-14), and Very high (≥15).
The primary outcome was in-hospital mortality.
We validated the 4C score using the AUC, with 95% confidence intervals calculated via bootstrapping with 2000 resamples. Missing data was treated using multiple imputation with chained questions.
We plotted the proportion of patients who died in hospital by 4C score and risk group and compared them to the initial derivation work in the UK. As a sensitivity analysis, we also examined the discriminability of the 4C for only hospital inpatients (n=484). All analysis was done in R 4.0.3.7
Our study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board, who waived the requirement for informed consent as the data for this study was retrospectively collected from hospital medical records. All methods were performed in accordance with relevant guidelines and regulations.
Our study included 560 patients, of which 484 (86%) were admitted to hospital, and 115 (21%) died (Table 1). Median age was 69 with patients split equally between males (51%) and females. After imputation, the low risk group contained 13.5% of patients, the intermediate group contained 25.3%, the high risk group contained 49.4% and the very high risk group contained 11.6%.
The AUC of the 4C score was 0.83, 95% CI (0.79–0.87). Mortality rates within the risk groups were 0% (Low), 3.2% (Intermediate), 25.9% (High), and 59.4% (Very High) (Fig. 1). These mortality rates are slightly lower the rates reported in the original research (1.2%, 9.9%, 31.4%, and 61.5%). The performance of the model was slightly lower when only hospital inpatients were considered (AUC = 0.80, 95% CI: 0.76–0.85).
|
Characteristic |
N (%)1 n=560 |
|---|---|
|
Age, yrs, (median Q1, Q3) |
69 (54, 82) |
|
Sex, males |
283 (51) |
|
Region |
|
|
Kitchener-Waterloo |
405 (62) |
|
Hamilton |
109 (27) |
|
Presenting Symptoms |
|
|
Temperature (C), (median Q1, Q3) |
37.1 (36.7–38.1) |
|
Cough |
352 (63) |
|
Shortness of breath |
319 (57) |
|
Fatigue |
269 (48) |
|
Altered Consciousness / Confusion |
106 (19) |
|
Heart Rate, per minute, (median Q1, Q3) |
91 (78, 104) |
|
RR rate, per minute, (median Q1, Q3) |
20 (18, 24) |
|
Oxygen saturation, (median Q1, Q3) |
95 (93, 97) |
|
Comorbidities |
|
|
Chronic lung disease2 |
82 (15) |
|
Chronic cardiac disease |
86 (15) |
|
Diabetes |
178 (32) |
|
Chronic liver disease |
14 (3) |
|
Chronic kidney disease |
61 (11) |
|
Chronic neurological disease |
65 (10) |
|
Cancer |
69 (12) |
|
Obesity3 |
55 (10) |
|
Rheumatologic disease |
30 (5) |
|
Dementia |
81 (14) |
|
HIV /AIDs |
< 6 |
|
Number of comorbidities, (median Q1, Q3) |
1 (0, 2) |
|
Biomarkers |
|
|
Blood Urea Nitrogen, mmol/L, (median Q1, Q3) |
6.6 (4.6, 10.6) |
|
C-reactive protein, mg/L, (median Q1, Q3) |
66 (19,147) |
|
White blood cells, x109/L, (median Q1, Q3) |
6.6 (5.1, 9.5) |
|
Platelets, x109/L, (median Q1, Q3) Q3) |
211 (162, 282) |
|
Neutrophils, x109/L, (median Q1, Q3) |
5.0 (3.4, 7.3) |
|
Lymphocytes, x109/L, (median Q1, Q3) |
1.0 (0.7, 1.4) |
|
1. Percentages are denominated by number of non-missing cases |
|
|
2. Does not include asthma |
|
|
3. As defined by staff |
|
We found that the 4C mortality score is a valid tool to prognosticate mortality among COVID-19 patients presenting to the emergency department or upon admission to hospitals in a Canadian population. We observed an AUC of 0.83 in our data, which is higher than was reported in the original derivation research in the United Kingdom but similar to validation conducted in Europe 8.
Clinical risk prediction models are critical tools that can be used to identify and prioritize high-risk patients. The proliferation of COVID-19 risk models is evidence of the demand for an accurate, accessible, and generalizable tool 9. Our finding that the 4C mortality score is valid in a jurisdiction distinct from the location of the original derivation and subsequent validation provides evidence for the generalized prognostic ability of the score.
The AUC we observed was higher than in the original research, which is unusual for external validation, but this was also true in other validation studies done in the United Kingdom and the Netherlands 8,10. The higher discriminability in our study may in part because we examined a more heterogeneous cohort that included patients who visited the emergency department but were not admitted to hospital. The 4C score was likely able to identify that these patients were at low risk and assign them lower scores. Overall, the AUC was only slightly lower in the sensitivity analysis that only examined hospital inpatients and inclusion of outpatients into the validation population adds further value to the study.
A key limitation of our study is that we were only able to include data from two regions within southern Ontario. While similarities between health systems across Canada suggest our findings will have excellent generalizability to other Canadian provinces and territories, our results may not generalize to geographically remote settings or to jurisdictions with substantially different health systems.
The 4C mortality score is an adaptable, valid prognostic tool for use in Canadian emergency departments and hospitals. It can be used to identify and prioritize care for COVID-19 patients at high risk of death.
This study leveraged data from the COREG registry, which is supported by a grant from the Canadian Institutes of Health Research (CIHR) (172754) and from the Hamilton Academic Health Sciences Organization (HAHSO) (HAH-21-04).
The data used in this study can be accessed for research purposes by submitting a request through the COREG data access portal. (https://www.coregontario.ca/info-data-access)
Code is available upon request
None
This study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board.
Study conception and design: AJ, TP, MD, TH, MB, AC, RK
Acquisition of data: TP, MJ, JK, RP, EF, LD
Analysis and Interpretation: AJ, TP, MJ, JK, JT, MD, MB
AJ wrote the initial draft of the manuscript and all authors critically revised it for important intellectual content. All authors agree to act as guarantors of the work.
This study leveraged the McMaster Coronavirus (COVID-19) Registry (COREG) led by Drs. Andrew Costa (NPI), Marla Beauchamp (co-PI), MyLinh Duong (co-PI), Rebecca Kruisselbrink (co-PI), Terence Ho (co-PI), and Jennifer LY Tsang (co-PI) and included the following institutions: Grand River Hospital, St. Mary’s General Hospital, Hamilton Health Sciences, St. Joseph’s Healthcare Hamilton, and the Niagara Health System. The COREG acknowledges the following individuals for their contributions: Muneeb Ahmed, Darly Dash, Megan Donaghy-Hughes, Hannah Farnworth, Stefan Jevtic, Catherine Lee, Candice Luo, Sarah MacGregor, Michael Mallender, Adhora Mir, Vivek Patel, Nivedh Patro, Noam Raiter, Pranali Raval, Brittany Salter, Adib Shamsuddin, Laura Spatafora, Xinxin Tang, Cooper Webb, Kristin Wright, Grace Xu, and Zaka Ahmad Zia.