External Validation of the 4C Mortality Score among COVID-19 Patients Visiting the Emergency Department or admitted to Hospital in Ontario, Canada

doi:10.21203/rs.3.rs-268410/v1

Download PDF

Research Article

External Validation of the 4C Mortality Score among COVID-19 Patients Visiting the Emergency Department or admitted to Hospital in Ontario, Canada

https://doi.org/10.21203/rs.3.rs-268410/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 19 Sep, 2021

Read the published version in Scientific Reports →

You are reading this latest preprint version

Objectives: Risk prediction scores are important tools to support clinical decision-making for patients with coronavirus disease (COVID-19). The objective of this paper was to validate the 4C mortality score, originally developed in the United Kingdom, for a Canadian population.

Methods: We conducted an external validation study within a registry of COVID-19 positive emergency department visits and hospital admissions in the Kitchener-Waterloo and Hamilton regions of southern Ontario between March 4 and January 9, 2020. We examined the validity of the 4C score to prognosticate in-hospital mortality using the area under the receiver operating characteristic curve (AUC) with 95% confidence intervals calculated via bootstrapping.

Results: The study included 560 individuals, of whom 115 (20.5%) died in-hospital. Median age was 69 years and 281 individuals (51%) were male. The AUC of the 4C score was 0.83, 95% confidence interval 0.79-0.87. Mortality rates across the pre-defined risk groups were 0% (Low), 3.2% (Intermediate), 25.9% (High), and 59.5% (Very High). The AUC was 0.80 (0.76-0.85) among hospital inpatients.

Interpretation: The 4C score is a valid tool to prognosticate mortality from COVID-19 in Canadian emergency departments and hospitals.

Health Economics & Outcomes Research

COVID-19

Clinical risk prediction models

External validation

Coronavirus disease (COVID-19) caused by the Severe Acute Respiratory Syndrome Virus 2 (SARS-CoV-2) can progress to acute respiratory distress syndrome, multiorgan failure, and death in some individuals ¹. The COVID-19 infection-fatality risk varies widely with age, ranging from < 0.1% for children and younger adults to greater than 10% for adults 75 and older². The clinical presentation and progression of COVID-19 in patients is highly variable³, which makes it difficult for clinicians to triage patients and determine their risk of poor outcomes. A simple, validated prognostic tool utilizing data that is available at presentation can help clinicians better prognosticate and make clinical decisions. Numerous tools to predict mortality in COVID-19 patients have been developed, but many are limited due to small derivation cohort sizes and/or inadequate validation⁴.

The 4C mortality score is an accessible risk stratification score developed by the International Severe Acute Respiratory and Emerging Infections Consortium (ISARIC) ⁵. It was derived and internally validated on a large, diverse cohort within the United Kingdom but needs external validity to confirm its generalizability. The objective of this study was to validate the ability of the 4C score to prognosticate mortality in COVID-19 patients presenting to the emergency department or admitted to hospital in Ontario, Canada.

Study design and setting

We conducted a validation study using records from the McMaster Multi-Regional Hospital Coronavirus Registry (COREG). COREG is a multi-center data registry collecting information on positive COVID-19 cases in the Kitchener-Waterloo and Hamilton regions of southern Ontario, Canada. The registry includes COVID-19-related emergency department (ED) visits and hospital admissions from six hospitals across the regions, including both academic and community centres.

Participants

We selected all patients admitted up to four weeks before time of analysis, which included admissions between March 4th, 2020 and January 7^th, 2020 (n=596). We excluded patients with no clinical data collected at presentation (n=36), for a final sample size of 560.

Measures

Data in COREG were abstracted from electronic medical records using a modified case report form published by ISARIC and the World Health Organization. We utilized demographic and clinical data at presentation, typically from the emergency department. For patients who were directly admitted, we utilized the first day of inpatient records.

4C mortality score

The 4C mortality score was derived and validated within the ISARIC World Health Organization Clinical Characterisation Protocol UK study⁶. The score was derived from a population of over 35,000 hospital inpatients validation on over 22,000 inpatient records indicated good discriminability (area under the receiver operating characteristic curve (AUC)=0.77)⁵.

The 4C score incorporates age, sex, comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale, blood urea nitrogen, and C-reactive protein. We adapted the score to match our available data as the Glasgow Coma Scale was not collected at presentation we replaced this risk factor with the documented presence of altered consciousness or confusion (Supplementary Table S1). The 4C score ranges from 0 to 21 with risk groups defined as Low (0-3), Intermediate (4-8), High (9-14), and Very high (≥15).

Outcome

The primary outcome was in-hospital mortality.

Statistical Analysis

We validated the 4C score using the AUC, with 95% confidence intervals calculated via bootstrapping with 2000 resamples. Missing data was treated using multiple imputation with chained questions.

We plotted the proportion of patients who died in hospital by 4C score and risk group and compared them to the initial derivation work in the UK. As a sensitivity analysis, we also examined the discriminability of the 4C for only hospital inpatients (n=484). All analysis was done in R 4.0.3.⁷

Ethics approval

Our study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board, who waived the requirement for informed consent as the data for this study was retrospectively collected from hospital medical records. All methods were performed in accordance with relevant guidelines and regulations.

Our study included 560 patients, of which 484 (86%) were admitted to hospital, and 115 (21%) died (Table 1). Median age was 69 with patients split equally between males (51%) and females. After imputation, the low risk group contained 13.5% of patients, the intermediate group contained 25.3%, the high risk group contained 49.4% and the very high risk group contained 11.6%.

Validation

The AUC of the 4C score was 0.83, 95% CI (0.79–0.87). Mortality rates within the risk groups were 0% (Low), 3.2% (Intermediate), 25.9% (High), and 59.4% (Very High) (Fig. 1). These mortality rates are slightly lower the rates reported in the original research (1.2%, 9.9%, 31.4%, and 61.5%). The performance of the model was slightly lower when only hospital inpatients were considered (AUC = 0.80, 95% CI: 0.76–0.85).

Table 1

Characteristics of COVID-positive hospital inpatients and emergency department patients in the Kitchener-Waterloo region of Canada, March 2020 – January 2021.
Characteristic	N (%)¹ n=560
Age, yrs, (median Q1, Q3)	69 (54, 82)
Sex, males	283 (51)
Region
Kitchener-Waterloo	405 (62)
Hamilton	109 (27)
Presenting Symptoms
Temperature (C), (median Q1, Q3)	37.1 (36.7–38.1)
Cough	352 (63)
Shortness of breath	319 (57)
Fatigue	269 (48)
Altered Consciousness / Confusion	106 (19)
Heart Rate, per minute, (median Q1, Q3)	91 (78, 104)
RR rate, per minute, (median Q1, Q3)	20 (18, 24)
Oxygen saturation, (median Q1, Q3)	95 (93, 97)
Comorbidities
Chronic lung disease²	82 (15)
Chronic cardiac disease	86 (15)
Diabetes	178 (32)
Chronic liver disease	14 (3)
Chronic kidney disease	61 (11)
Chronic neurological disease	65 (10)
Cancer	69 (12)
Obesity³	55 (10)
Rheumatologic disease	30 (5)
Dementia	81 (14)
HIV /AIDs	< 6
Number of comorbidities, (median Q1, Q3)	1 (0, 2)
Biomarkers
Blood Urea Nitrogen, mmol/L, (median Q1, Q3)	6.6 (4.6, 10.6)
C-reactive protein, mg/L, (median Q1, Q3)	66 (19,147)
White blood cells, x10⁹/L, (median Q1, Q3)	6.6 (5.1, 9.5)
Platelets, x10⁹/L, (median Q1, Q3) Q3)	211 (162, 282)
Neutrophils, x10⁹/L, (median Q1, Q3)	5.0 (3.4, 7.3)
Lymphocytes, x10⁹/L, (median Q1, Q3)	1.0 (0.7, 1.4)
1. Percentages are denominated by number of non-missing cases
2. Does not include asthma
3. As defined by staff

We found that the 4C mortality score is a valid tool to prognosticate mortality among COVID-19 patients presenting to the emergency department or upon admission to hospitals in a Canadian population. We observed an AUC of 0.83 in our data, which is higher than was reported in the original derivation research in the United Kingdom but similar to validation conducted in Europe ⁸.

Clinical risk prediction models are critical tools that can be used to identify and prioritize high-risk patients. The proliferation of COVID-19 risk models is evidence of the demand for an accurate, accessible, and generalizable tool ⁹. Our finding that the 4C mortality score is valid in a jurisdiction distinct from the location of the original derivation and subsequent validation provides evidence for the generalized prognostic ability of the score.

The AUC we observed was higher than in the original research, which is unusual for external validation, but this was also true in other validation studies done in the United Kingdom and the Netherlands ^8,10. The higher discriminability in our study may in part because we examined a more heterogeneous cohort that included patients who visited the emergency department but were not admitted to hospital. The 4C score was likely able to identify that these patients were at low risk and assign them lower scores. Overall, the AUC was only slightly lower in the sensitivity analysis that only examined hospital inpatients and inclusion of outpatients into the validation population adds further value to the study.

Limitations

A key limitation of our study is that we were only able to include data from two regions within southern Ontario. While similarities between health systems across Canada suggest our findings will have excellent generalizability to other Canadian provinces and territories, our results may not generalize to geographically remote settings or to jurisdictions with substantially different health systems.

The 4C mortality score is an adaptable, valid prognostic tool for use in Canadian emergency departments and hospitals. It can be used to identify and prioritize care for COVID-19 patients at high risk of death.

Funding Statement

This study leveraged data from the COREG registry, which is supported by a grant from the Canadian Institutes of Health Research (CIHR) (172754) and from the Hamilton Academic Health Sciences Organization (HAHSO) (HAH-21-04).

Data Sharing Statement

The data used in this study can be accessed for research purposes by submitting a request through the COREG data access portal. (https://www.coregontario.ca/info-data-access)

Code availability

Code is available upon request

Conflicts of interest

None

Ethics approval

This study received ethics approval from the Tri-hospital Research Ethics Board and the Hamilton Integrated Research Ethics Board.

Contributor's statement

Study conception and design: AJ, TP, MD, TH, MB, AC, RK
Acquisition of data: TP, MJ, JK, RP, EF, LD
Analysis and Interpretation: AJ, TP, MJ, JK, JT, MD, MB

AJ wrote the initial draft of the manuscript and all authors critically revised it for important intellectual content. All authors agree to act as guarantors of the work.

Acknowledgements

This study leveraged the McMaster Coronavirus (COVID-19) Registry (COREG) led by Drs. Andrew Costa (NPI), Marla Beauchamp (co-PI), MyLinh Duong (co-PI), Rebecca Kruisselbrink (co-PI), Terence Ho (co-PI), and Jennifer LY Tsang (co-PI) and included the following institutions: Grand River Hospital, St. Mary’s General Hospital, Hamilton Health Sciences, St. Joseph’s Healthcare Hamilton, and the Niagara Health System. The COREG acknowledges the following individuals for their contributions: Muneeb Ahmed, Darly Dash, Megan Donaghy-Hughes, Hannah Farnworth, Stefan Jevtic, Catherine Lee, Candice Luo, Sarah MacGregor, Michael Mallender, Adhora Mir, Vivek Patel, Nivedh Patro, Noam Raiter, Pranali Raval, Brittany Salter, Adib Shamsuddin, Laura Spatafora, Xinxin Tang, Cooper Webb, Kristin Wright, Grace Xu, and Zaka Ahmad Zia.

Wu, C. et al. Risk Factors Associated With Acute Respiratory Distress Syndrome and Death in Patients With Coronavirus Disease 2019 Pneumonia in Wuhan, China. JAMA Intern. Med. 180, 934–943 (2020).
Yang, W. et al. Estimating the infection-fatality risk of SARS-CoV-2 in New York City during the spring 2020 pandemic wave: a model-based analysis. Lancet Infect. Dis. 0, (2020).
Struyf, T. et al. Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID‐19 disease. Cochrane Database Syst. Rev. (2020) doi:10.1002/14651858.CD013665.
Wynants, L. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 369, (2020).
Knight, S. R. et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 370, (2020).
Docherty, A. B. et al. Features of 20 133 UK patients in hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: prospective observational cohort study. BMJ 369, (2020).
R Core Team. R: A language and environment for statistical computing. (2020).
Wellbelove, Z., Walsh, C., Perinpanathan, T., Lillie, P. & Barlow, G. Comparing the 4C mortality score for COVID-19 to established scores (CURB65, CRB65, qSOFA, NEWS) for respiratory infection patients. J. Infect. (2020) doi:10.1016/j.jinf.2020.10.015.
Gupta, R. K. et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study. Eur. Respir. J. (2020) doi:10.1183/13993003.03498-2020.
Dam, P. M. E. L. van et al. Performance of prediction models for short term outcome in COVID-19 patients in the emergency department: a retrospective study. medRxiv 2020.11.25.20238527 (2020) doi:10.1101/2020.11.25.20238527.

No competing interests reported.

SupplementaryTableS1.docx

Download PDF

Journal Publication

published 19 Sep, 2021

Read the published version in Scientific Reports →

Editorial decision: Major revision
09 Jul, 2021
Reviews received at journal
05 Jul, 2021
Reviewers agreed at journal
24 Jun, 2021
Reviews received at journal
13 Jun, 2021
Reviewers agreed at journal
27 May, 2021
Reviewers invited by journal
27 May, 2021
Editor assigned by journal
17 May, 2021
Editor invited by journal
11 Mar, 2021
Submission checks completed at journal
08 Mar, 2021
First submitted to journal
22 Feb, 2021

You are reading this latest preprint version

External Validation of the 4C Mortality Score among COVID-19 Patients Visiting the Emergency Department or admitted to Hospital in Ontario, Canada

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Study design and setting

Participants

Measures

4C mortality score

Outcome

Statistical Analysis

Ethics approval

Results

Validation

Interpretation

Limitations

Conclusion

Declarations

Funding Statement

Data Sharing Statement

Code availability

Conflicts of interest

Ethics approval

Contributor's statement

Acknowledgements

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1