Implementation and Evaluation of a Novel COVID-19 Prognostic Score for Patients with Suspected COVID-19

doi:10.21203/rs.3.rs-500548/v1

Download PDF

Research

Implementation and Evaluation of a Novel COVID-19 Prognostic Score for Patients with Suspected COVID-19

https://doi.org/10.21203/rs.3.rs-500548/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Patient safety and optimal resource allocation are essential during the COVID-19 pandemic. We present a COVID-19 prognostic algorithm implemented in real-time as a clinical decision support structure for symptomatic persons under investigation (PUI) for COVID-19 in the emergency department (ED).

Methods: The training data included 1,469 patients who tested positive for SARS-CoV-2 within 14 days of acute care. The validation was done using a retrospective set of 414 SARS-CoV-2 positive patients and a PUI set of 13,271 patients who had symptomatic SARS-CoV-2 test during acute care visit. We performed a real-time model assessment on 2,174 patients with an ED visit and symptomatic test or COVID-19 positive result. The logistic regression prognostic model used demographics, comorbidities, home medications, and vital signs factors. The COVID-19 severity outcome comprised a composite of intensive care unit (ICU) admission, invasive mechanical ventilation (ventilator) use, and mortality.

Results: The area under the receiver operating characteristic (AUROC) was 0.87 (95%-CI: 0.83, 0.91) in the retrospective validation and 0.82 (95%-CI: 0.81, 0.83) in the PUI population for predicting COVID-19 severity. The model had an AUROC of 0.85 (95%-CI, 0.83, 0.87) in the real-time set. The rates of ICU admission, ventilator use, and death in patients with the lowest 20% of the scores were zero, significantly lower compared to those rates (20.6%, 9.5%, and 18.1%, respectively) for patients with the highest 20% of the scores in the real-time set. The model performed equitably across racial/ethnic minorities.

Conclusion: Our prognostic model score developed on COVID-19 positive patients performed well on PUI patients and in real-time in ED to support clinical decision making.

Critical Care & Emergency Medicine

COVID-19 severity

prognostic model

real-time validation

patient safety

ICU.

The dynamic of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) infection raised concerns regarding resource availability throughout medical systems, including intensive care unit (ICU) healthcare providers, personal protective equipment, total hospital, and ICU beds, and mechanical ventilators. On March 11th, 2020, the World Health Organization declared the Coronavirus disease 2019 (COVID-19) a pandemic. The COVID-19 pandemic has caused over 153 million confirmed infections and over 3,209,000 confirmed deaths as of May 4th, 2021.¹ One of the initial large observational studies, published from China, revealed that approximately 15% of the confirmed cases required hospitalization, 5% needed ICU admission, and 2.3% died.² A large descriptive study from the United Kingdom found that 17% of patients hospitalized with COVID-19 were admitted to either the ICU or high dependency units, and 26% died.³ A multihospital cohort study from the United States (U.S.) identified that the 30-day mean risk standardized event rate of mortality and referral to hospice of hospitalized patients with COVID-19 varied from 9–16%, with better outcomes occurring when the community’s prevalence was lower.⁴

Since the beginning, global efforts by the scientific community to understand SARS-CoV-2 and the COVID-19 from the bench to the bedside have been remarkable.⁵ Stratifying disease severity is an essential aspect of patient care; however, during a pandemic, its role becomes paramount and expands to improving patient safety while also optimizing hospital resource utilization. Several studies have developed emergency department (ED) evaluation systems with variable goals and methods.^6–12 These models successfully evaluated the possibility of isolating COVID-19 patients in ED, the epidemiology and COVID-19 clinical data, the advantage of distinguishing life-threatening emergencies, and the likelihood of COVID-19 diagnosis.^6–12

Most predictive models for COVID-19 severity involved patients with a positive polymerase chain reaction (PCR) test, not in patients with suspected COVID-19. A systematic evaluation of the predictive models for COVID-19 clinical deterioration found most of the published studies included patients with confirmed infection,¹³ making them less useful in the clinic or emergency departments in the moment. The majority of predictive models for patients with suspected COVID-19 infection aimed to diagnose COVID-19, and very few predicted severity.¹⁴ One systematic review of the prognostic models emphasized the high risk of bias while not recommending their use in clinical practice yet.¹⁵ Limitations mark the systematic reviews of the prognostic models, and a COVID-19 precise living document exists.¹⁶ A group of researchers proposed an open platform for such reviews that will be continuously updated using artificial intelligence and numerous experts.¹⁷ The QCOVID is a published living risk prediction algorithm that performed well for predicting time to death in patients with confirmed or suspected COVID-19.¹⁸

Despite this, whether successful incorporation of a predictive model into the electronic health records for clinical decision support followed by real-time evaluation of model performance in suspected COVID-19 patient population remains unknown. This aspect is critically important as the volume of patients with COVID-19 like illness is likely an order of magnitude higher than patients who test positive. Moreover, many of these predictive models are derived and validated using primarily observational data. We aim to fill these knowledge gaps, testing the hypothesis that a prognostic tool for COVID-19 can be accurate in the patient population with suspected COVID-19 and real-time practice, paving the way for decision support in emergency departments. On November 23rd, 2020, we implemented a prognostic model to inform the risk for COVID-19 severity in the ED as a clinical decision support system to providers. This study intended to evaluate this prognostic model's accuracy in real-time for patients who present to the ED with suspicion for COVID-19.

Study Design and Setting

This is a retrospective and prospective study that developed, implemented, and evaluated a prognostic model in patients with PCR-confirmed COVID-19 diagnosis or suspected COVID-19 (person under investigation) in a twelve-hospital system United States Midwest healthcare system. This study was approved by the [blinded for review] Institutional Review Board (STUDY00011742).

Selection of Participants

Patients were included if they were PCR confirmed COVID-19 positive or symptomatic persons under investigation (PUIs) with a patient status of emergency, observation, or inpatient at a participating center. We only included patients who did not opt out of research on admission. Patients were excluded if they did not have at least one recorded ED vital sign (heart rate, respiratory rate, temperature, oxygen saturation, or systolic blood pressure) or missing comorbidity data. A complete set of vital signs was deemed necessary given our model was intended to be implemented and utilized across patients receiving a complete evaluation which would include at least one complete set of vital signs.

Training dataset

The training data set included 1,469 patients who were PCR-positive for SARS-CoV-2 within 14 days of an acute care, hospital-based visit including emergency department, observation, and inpatient encounters between March 4th to August 21st, 2020.

Validation datasets

We included three validation sets:

1) a retrospective COVID-19 PCR-positive set comprised of 414 random patients who tested positive for SARS-CoV-2 between August 22nd to October 11th, 2020.

2) a person under investigation (PUI) data set comprised of 13,271 patients who had a SARS-CoV-2 test with a “symptomatic” designation ordered and a result pending during the first 24 hours of an acute care, hospital-based visit irrespective of the final results between May 4th and October 11th, 2020. The symptomatic designation for patients with fever, cough, dyspnea, sore throat, muscle aches, vomiting, diarrhea was based on clinical judgment and prioritized testing for faster turnaround time beginning May 4th, 2020.

3) a real-time data set included 2,174 patients with an ED visit and symptomatic test or a positive SARS-CoV-2 PCR test following implementation of the prognostic model in Emergency Departments (EDs) from November 23rd, 2020 to January 21st, 2021.

Factors of interest and factors selected for the final model

To reduce the likelihood of over-fitting a LASSO-logit model was used to facilitate variable selection with the tuning parameter determined by the Bayesian information criterion (BIC). LASSO is a penalized regression method that can facilitate factor selection by excluding factors with a minor contribution to the model.

Supplemental Table 1 listed the factors of interest and the factors selected for the final model based on the variables availability. Factors of the logistical model included age (years), male,^{3, 19} race or ethnicity, non-English speaking,^{20, 21} overweight or obese (body mass index [BMI] > 25),^22–24 three month prior home medications²⁵ (defined as whether a patient was prescribed a medication within 3 months or before and after the index acute care visit) and chronic comorbidities^{3, 26} extracted from ICD10 codes (Supplemental Table 2) collected in the 5 years prior to the index visit: Finally, we included the following vital signs: maximum heart rate (HR), respiratory rate (RR), temperature within the first 24 hours, and minimal oxygen saturation (SpO₂) and systolic blood pressure (SBP) within the first 24 hours.

Outcomes

COVID-19 severity comprised a composite of ICU admission, invasive mechanical ventilation (ventilator) use, and death at any time.

Analysis

The patients’ characteristics between data sets were compared using ANOVA and chi-square respectively for continuous versus categorical variables. Odds ratios (OR) and 95% Confidence Intervals were also reported. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), likelihood ratios, false negative and false positive rate, and the area under the receiver operating characteristics (AUROC) were summarized for the model performance. Statistical significance was defined with the alpha set to 0.05, all tests were two-tailed. Statistical analyses were performed using Stata MP, version 16 (StataCorp, College Station, TX).

The real-time model was evaluated across gender and racial/ethnic groups to compare performance across different groups and ensure the model performed equitably and did not pose a risk to widening the disparate outcomes observed throughout the pandemic.

Model construction

The purpose of this model generation was to develop a prognostic model that could predict patients that developed a severe case of COVID-19 defined as mechanical ventilation or death. A multivariable logistic regression model to predict the odds of the need for mechanical ventilation, in-hospital or out-of-hospital death was fit to the data using the variables selected from LASSO. This model was developed using only data from the training database. From this, a risk score was calculated in the validation cohorts based off the sum of the beta coefficients. The AUROC was calculated for all validation cohorts to evaluate discrimination in the validation datasets.

Model implementation

Implementation into EPIC Electronic Health Record (EHR) occurred for ED patients on November 23rd, 2020. The logistic prognostic model was exported as a predictive model markup language (PMML) file. An EPIC reporting workbench was developed to facilitate inputs into the model. All the inputs were mapped using corresponding ICD-10 codes (Supplemental Table 2), pharmaceutical subclasses, RxNorm codes, and EPIC documentation flowsheets (for vitals). The output was delivered as a clinical decision support system to ED providers. For visualization purposes, the COVID-19 severity risk score was multiplied by 100 and cut points that identified patients with Low Risk (low probability of composite outcome) and High Risk (high probability of composite outcome). Visualization (Supplemental Fig. 1) was highlighted on the patient sidebar, available to all ED providers and nurses, as well as physicians and staff involved in triage, patient flow, and capacity management.

Development of a metric of clinical deterioration and predictive threshold

A multidisciplinary group of ED, inpatient, ICU, providers, and informaticists, and hospital leadership decided the following institutional metric to guide ED decision-making for patients at high risk of deterioration: death (in or out of hospital), need for mechanical ventilation, or ICU admission.

A total of 2,041 patients were included in the final model training and retrospective validation (Fig. 1). Table 1 listed patients’ characteristics in each cohort. Overall, significant difference in all variables in demographics, use of home medications, comorbidities, and 24-hour vitals existed across training and validation cohorts, except for loop diuretic, inflammatory bowel disease, and rheumatoid arthritis. Compared to COVID-19 PCR-positive patients in the training set, the patients in the retrospective validation set and PUI set were slightly younger (median age of 52.2 and 49.1 years vs. 53.6 years) and had lower rates of ICU admission (18.1% and 10.8% vs 23.4%), ventilation use (3.4% and 5.3% vs. 11.1%), and mortality (1.7% and 3.5 % vs. 8.5%). Compared to the training set, the real-time data set was older (median age of 56.9 years) and had lower rates of ICU admission, ventilation use and mortality (9.4%, 3.5%, and 6.8%), respectively.

Table 1. Characteristics of the patients included in a training set, retrospective validation set, PUI set, and real-time validation set

	Training set	Prospective validation set	PUI set	Real-time validation set	p-value
	N=1,469	N=414	N=13,271	N=2,174
Age − median (IQR)	53.6 (34.8-70.0)	52.2 (32.6-70.6)	49.1 (31.3-66.8)	56.9 (35.4-72.4)	<0.001
Male − N (%)	726 (49.4%)	197 (47.6%)	5,967 (45.0%)	981 (45.1%)	0.01
Race − N (%) †					<0.001
White	570 (38.8%)	213 (51.4%)	9,045 (68.2%)	1,509 (69.4%)
Black	454 (30.9%)	86 (20.8%)	2,115 (15.9%)	239 (11.0%)
Asian	138 (9.4%)	23 (5.6%)	1,003 (7.6%)	145 (6.7%)
Hispanic	164 (11.2%)	41 (9.9%)	191 (1.4%)	84 (3.9%)
Declined	121 (8.2%)	38 (9.2%)	422 (3.2%)	163 (7.5%)
Other	22 (1.5%)	13 (3.1%)	495 (3.7%)	34 (1.6%)
Non-English Speaking − N (%)	480 (32.7%)	107 (25.8%)	1,548 (11.7%)	411 (18.9%)	<0.001
Obesity − N (%)	1,127 (76.7%)	318 (76.8%)	9,936 (74.9%)	1,456 (67.0%)	<0.001
Home Medications within 3 months (N[%])
Calcium Channel	89 (6.1%)	20 (4.8%)	782 (5.9%)	208 (9.6%)	<0.001
HCTZ	41 (2.8%)	16 (3.9%)	483 (3.6%)	121 (5.6%)	<0.001
Beta-Blocker	172 (11.7%)	50 (12.1%)	1,786 (13.5%)	426 (19.6%)	<0.001
ACE Inhibitor	73 (5.0%)	34 (8.2%)	952 (7.2%)	218 (10.0%)	<0.001
ARB	91 (6.2%)	23 (5.6%)	830 (6.3%)	193 (8.9%)	<0.001
Metformin	69 (4.7%)	20 (4.8%)	591 (4.5%)	149 (6.9%)	<0.001
Warfarin	49 ( 3.3%)	10 (2.4%)	392 (3.0%)	93 (4.3%)	0.013
Rivaroxaban	14 (1.0%)	3 (0.7%)	209 (1.6%)	48 (2.2%)	0.014
Oral Steroids	86 (5.9%)	18 (4.3%)	1,087 (8.2%)	219 (10.1%)	<0.001
PPI	217 (14.8%)	61 (14.7%)	2,303 (17.4%)	487 (22.4%)	<0.001
Clopidogrel	20 (1.4%)	9 (2.2%)	170 (1.3%)	61 (2.8%)	<0.001
Corticosteroid Inhaler	83 (5.7%)	21 (5.1%)	992 (7.5%)	187 (8.6%)	0.002
Aspirin	193 (13.1%)	65 (15.7%)	1,444 (10.9%)	370 (17.0%)	<0.001
Loop Diuretic	104 (7.1%)	21 (5.1%)	963 (7.3%)	181 (8.3%)	0.17
Comorbidities
Hypertension − N (%)	632 (43.0%)	173 (41.8%)	5,162 (38.9%)	1,170 (53.8%)	<0.001
T1DM − N (%)	98 (6.7%)	22 (5.3%)	580 (4.4%)	161 (7.4%)	<0.001
T2DM − N (%)	373 (25.4%)	96 (23.2%)	2,329 (17.5%)	521 (24.0%)	<0.001
Coronary Artery Disease − N (%)	201 (13.7%)	59 (14.3%)	1,665 (12.5%)	410 (18.9%)	<0.001
VTE − N (%)	149 (10.1%)	18 (4.3%)	928 (7.0%)	225 (10.3%)	<0.001
Heart Failure − N (%)	176 (12.0%)	39 (9.4%)	1,413 (10.6%)	318 (14.6%)	<0.001
COPD − N (%)	121 (8.2%)	32 (7.7%)	1,331 (10.0%)	318 (14.6%)	<0.001
Asthma − N (%)	213 (14.5%)	57 (13.8%)	2,382 (17.9%)	418 (19.2%)	<0.001
Pacemaker/AICD − N (%)	42 (2.9%)	11 (2.7%)	429 (3.2%)	99 (4.6%)	0.017
Pulmonary HTN − N (%)	74 (5.0%)	12 (2.9%)	448 (3.4%)	128 (5.9%)	<0.001
CKD − N (%)	249 (17.0%)	48 (11.6%)	1,590 (12.0%)	434 (20.0%)	<0.001
Atrial Fib/Flutter − N (%)	173 (11.8%)	33 (8.0%)	1,196 (9.0%)	308 (14.2%)	<0.001
CVA − N (%)	149 (10.1%)	38 (9.2%)	1,164 (8.8%)	298 (13.7%)	<0.001
IBD − N (%)	17 (1.2%)	5 (1.2%)	217 (1.6%)	38 (1.7%)	0.49
Rheumatoid Arthritis − N (%)	35 (2.4%)	8 (1.9%)	317 (2.4%)	57 (2.6%)	0.89
Malignancy − N (%)	117 (8.0%)	33 (8.0%)	1,234 (9.3%)	273 (12.6%)	<0.001
Sleep Apnea − N (%)	163 (11.1%)	47 (11.4%)	1,629 (12.3%)	344 (15.8%)	<0.001
Vitals
Max HR in 24 hr − mean (SD)	98.9 (20.7)	95.8 (19.5)	95.4 (20.4)	98.5 (21.1)	<0.001
Max RR in 24 hr − mean (SD)	25.5 (13.3)	23.4 (9.1)	22.1 (8.4)	23.9 (10.0)	<0.001
Max Temp in 24 hr − mean (SD)	99.7 (1.6)	99.5 (1.6)	98.8 (1.4)	99.0 (1.4)	<0.001
Min SpO2 in 24 hr − mean (SD)	92.1 (8.2)	93.1 (6.1)	94.7 (5.4)	92.8 (6.7)	<0.001
Min SBP in 24 hr − mean (SD)	112.1 (22.7)	114.9 (21.2)	122.7 (20.6)	116.6 (21.3)	<0.001
Outcomes
ICU Admission − N (%)	346 (23.6%)	75 (18.1%)	1,428 (10.8%)	100 (9.4%)	<0.001
Mechanical Ventilation − N (%)	164 (11.2%)	14 (3.4%)	478 (5.3%)	37 (3.5%)	<0.001
Died − N (%)	125 ( 8.5%)	7 (1.7%)	460 (3.5%)	73 (6.8%)	<0.001
Composite Outcome* − N (%)	382 (26.0%)	76 (18.4%)	1,627 (12.3%)	247 (11.4%)	<0.001

Table 2 described the odds ratios used in the logistic model generation. Other as race and inflammatory bowel disease, are the two variables with the highest odds ratios (4.13 [95% confidence intervals, 1.39, 12.33], p value 0.011, and 3.95 [1.10, 14.23], p value 0.036, respectively). Warfarin is the variable with lowest odds ratios (0.25 [0.1, 0.68], p-value of 0.006). The model included factors that increase the odds of COVID-19 severity, such as male, Asian or Hispanic race, obesity, use of calcium channel blocker, rivaroxaban, oral steroids, clopidogrel, aspirin, and a loop diuretic, hypertension, type 2 diabetes mellitus, venous thromboembolism, pacemaker/ automatic implantable cardioverter-defibrillator, pulmonary hypertension, chronic kidney disease, inflammatory bowel disease, the maximum temperature in 24 hours, and factors that decrease the odds, such as the use of hydrochlorothiazide, angiotensin-converting enzyme inhibitor, warfarin, and rheumatoid arthritis.

Table 2

Odds ratios of variables in the model
Variables	Odds Ratio	95% CI	p-value
Age	1.03	1.02, 1.04	< 0.001
Male	1.95	1.33, 2.84	0.001
Race
Black	0.94	0.54, 1.62	0.817
Asian	1.59	0.77, 3.26	0.208
Hispanic	1.49	0.67, 3.30	0.329
Declined	1.36	0.62, 2.97	0.437
Other	4.13	1.39, 12.33	0.011
Non-English Speaking	1.17	0.69, 1.99	0.560
Obesity	1.41	0.89, 2.22	0.140
Calcium Channel Blocker (home med w/in 3 months)	1.37	0.67, 2.76	0.386
HCTZ (home med w/in 3 months)	0.60	0.21, 1.69	0.330
Beta-Blocker (home med w/in 3 months)	1.00	0.54, 1.84	0.991
Ace Inhibitor (home med w/in 3 months)	0.49	0.20, 1.18	0.112
ARB (home med w/in 3 months)	0.83	0.40, 1.71	0.606
Metformin (home med w/in 3 months)	0.86	0.38, 1.98	0.727
Warfarin (home med w/in 3 months)	0.25	0.10, 0.68	0.006
Rivaroxaban (home med w/in 3 months)	1.34	0.38, 4.69	0.644
Oral Steroids (home med w/in 3 months)	1.45	0.69, 3.03	0.328
PPI (home med w/in 3 months)	0.99	0.60, 1.63	0.970
Clopidogrel (home med w/in 3 months)	1.95	0.60, 6.32	0.266
Corticosteroid Inhaler (home med w/in 3 months)	1.12	0.53, 2.38	0.769
Aspirin (home med w/in 3 months)	1.39	0.82, 2.34	0.217
Loop Diuretic (home med w/in 3 months)	1.58	0.85, 2.93	0.150
Hypertension	1.44	0.91, 2.27	0.119
T1DM	0.83	0.40, 1.72	0.620
T2DM	1.44	0.93, 2.23	0.103
Coronary Artery Disease	0.66	0.39, 1.12	0.125
VTE	1.37	0.81, 2.34	0.244
Heart Failure	1.01	0.57, 1.79	0.965
COPD	0.96	0.53, 1.75	0.896
Asthma	1.00	0.58, 1.74	0.992
Pacemaker/AICD	2.07	0.90, 4.79	0.088
Pulmonary HTN	1.44	0.70, 2.96	0.323
CKD	1.62	0.99, 2.66	0.056
Atrial Fib/Flutter	1.04	0.60, 2.78	0.897
CVA	1.27	0.75, 2.16	0.370
IBD	3.95	1.10, 14.23	0.036
Rheumatoid Arthritis	0.62	0.23, 1.72	0.363
Malignancy	1.03	0.57, 1.87	0.928
Sleep Apnea	0.87	0.51, 1.49	0.616
Max HR in 24 hr	1.01	1.00, 1.02	0.275
Max RR in 24 hr	1.02	1.01, 1.03	0.001
Max Temp in 24 hr	1.34	1.20, 1.51	< 0.001
Min SpO2 in 24 hr	0.94	0.92, 0.96	< 0.001
Min SPB in 24 hr	0.98	0.97, 0.99	< 0.001
Abbreviations: w/in: within; HCTZ: hydrochlorothiazide; ACE: angiotensin-converting enzyme; ARB: angiotensin receptor blocker; PPI: proton pump inhibitor; T1DM: Type 1 diabetes mellitus; T2DM: Type 2 diabetes mellitus; VTE: venous thromboembolism; COPD: chronic obstructive pulmonary disease; AICD: automatic implantable cardioverter-defibrillator; HTN: hypertension; CKD: chronic kidney disease; CVA: cerebrovascular accident; Afib: atrial fibrillation, Aflutter: atrial flutter, IBD: inflammatory bowel disease; hr: hour; Max: maximum; Min: minimum; HR: heart rate; RR: respiratory rate; Temp: temperature; SpO₂: peripheral oxygen saturation; SBP: systolic blood pressure.

In the validation cohorts, the risk score was used to identify a clinically useful threshold to predict the institutional metric. Multiple thresholds were defined, and 2x2 contingency tables including sensitivity, specificity, PPV, and NPV were created for each threshold. The system leadership reviewed the various thresholds and based on clinical resources, defined an appropriate threshold. The multidisciplinary team reviewed the model performance, including sensitivity, specificity, PPV, NPV, likelihood ratios across multiple thresholds to facilitate rapid implementation. Cut-off points flagging high and low-risk patients were chosen in collaboration with both system leadership following engagement with front-line providers. The goal for low-risk cut-off was to have a high sensitivity at the expense of specificity to reduce potential errors associated with inappropriate discharge home. The goal for high-risk cut-off was a higher specificity to balance the need for close monitoring with resource scarcity, including ICU and step-down capacity.

Retrospective validation and in PUI

The model produced an AUROC of 0.87 (95% CI: 0.83, 0.91) for predicting composite outcomes (ICU admission, ventilator use, or death) using the prospective validation cohort (Supplemental Fig. 2). None of the patients with the lowest 20% of the scores (0-0.0104) had ICU admission, ventilator use, or died, compared to 62%, 15.9%, and 7.3%, respectively, for patients with the highest 20% of the scores (0.168-1.0) (Supplemental Table 3). At a cut point of > 0.1, the model had a sensitivity of 73.7% and specificity of 79.9% in predicting the composite outcomes (Supplemental Table 4).

This model was further tested in the PUI cohort that included 13,271 patients who had a SARS-CoV-2 test with a “symptomatic” designation ordered in ED. Of note, the accumulative COVID positive rate in the PUI data set was 25.8%. A total of 68% of patients were discharged before the test resulted in our medical system. The model produced an AUROC of 0.82 (95% CI: 0.81, 0.83) for predicting the composite outcomes in the PUI cohort (Supplemental Fig. 3). For patients with the lowest 20% of the scores (0.00062–0.0074), only 1.0% had ICU admission, 0.3% ventilator use, and 0.2% died, compared to 31.6%, 13.8%, and 11.9%, respectively, for patients with the highest 20% of the scores (0.168-1.0) (Supplemental Table 5). At the cut point of > 0.1, the model had a sensitivity of 52.2% and specificity of 88.1% in predicting composite outcomes (Supplemental Table 6).

Real-time validation

Critically, we implemented this model to predict the composite outcomes to evaluating the COVID-19 severity and assessed the model’s real-time performance. This real-time cohort had a median age of 56.9 years (IQR: 35.4–72.4), had an ICU admission rate of 9.4 %, ventilation rate of 3.5%, and mortality rate of 6.8%. The model had an AUROC of 0.85 (95% CI, 0.83, 0.87) to predict the composite outcome in the real-time data set. (Fig. 2). The rates of ICU admission, ventilator use, and death in patients with the lowest 20% of the scores (0.001–0.009) were zero, significantly lower compared to those rates (20.6%, 9.5%, and 18.1%, respectively) for patients with the highest 20% of the scores (0.20–0.99) (Table 3). At the cut point of > 0.1, the model had a sensitivity of 78% and a specificity of 71% in the real-time data set (Table 4).

Table 3

Distribution of outcomes by score ranges in quintile for the real-time validation data set (n = 2,174)
	Score Range	ICU, n(%)	Vent, n(%)	Death, n(%)	n
Lowest 20% scores	0.001–0.009	0 ( 0.0%)	0 ( 0.0%)	0 ( 0.0%)	435
20–40%	0.009–0.02	3 ( 1.5%)	2 ( 1.0%)	1 ( 0.5%)	435
40–60%	0.02–0.06	11 ( 5.5%)	2 ( 1.0%)	3 ( 1.5%)	435
60–80%	0.06–0.20	27 (11.0%)	6 ( 2.4%)	17 ( 6.9%)	435
Highest 20% of scores	0.20–0.99	59 (20.6%)	27 ( 9.4%)	52 (18.1%)	434
Abbreviations: ICU: Intense Care Unit; Vent: Ventilator.

Table 4

Clinical performance of the logistical model for predicting COVID-19 severity* in the real-time validation data set (n = 2174)
Cut point		False +	True -	False -	Sensitivity
> 0.03	238	1032	983	12	95%	49%	0.19	0.99	1.86	0.10
> 0.05	228	828	1187	22	91%	59%	0.22	0.98	2.22	0.15
> 0.07	214	699	1316	36	86%	65%	0.23	0.97	2.47	0.22
> 0.09	197	618	1397	53	79%	69%	0.24	0.96	2.57	0.31
> 0.1	194	587	1428	56	78%	71%	0.25	0.96	2.66	0.32
> 0.11	189	553	1462	61	76%	73%	0.25	0.96	2.75	0.34
> 0.13	178	495	1520	72	71%	75%	0.26	0.95	2.90	0.38
> 0.15	166	459	1556	84	66%	77%	0.27	0.95	2.91	0.44
> 0.17	158	415	1600	92	63%	79%	0.28	0.95	3.07	0.46
> 0.19	150	387	1628	100	60%	81%	0.28	0.94	3.12	0.50
* COVID-19 severity or high probability for Composite Outcome = ICU admission, need for mechanical ventilation, or death
Abbreviations: True +: True Positive; False +: False Positive; True -: True Negative; False -: False Negative; NPV: Negative
Predictive Value; PPV: Positive Predictive Value; LR +: Likelihood Ratio Positive; LR -: Likelihood Ratio Negative.

The AUROC of all cohorts predicting various outcomes combined and individual are listed in the Supplemental Table 7. The performance remained strong for individual or composite outcomes. Furthermore, the model performed equitably across racial/ethnic minorities.(Supplemental Table 8)

We developed and implemented a model to predict increased risk for COVID-19 severity to support the ED physicians’ clinical decision-making throughout our medical system. Despite the significant variabilities of the factors, our model performed well in a large PUI study population with an AUROC of 0.82 (95% CI: 0.81, 0.83). This is particularly helpful in the ED where the COVID-19 PCR test results are not always available when clinical decisions need to be made. Importantly, we evaluated our model real-time in PUI patients seeking acute care in ED after the score became available in EPIC, and the model performance remained strong with an AUROC of 0.85 (95% CI, 0.83, 0.87). Furthermore, the model performed equitably across racial/ethnic minorities and did not increase the risk of widening the disparate outcomes observed throughout the pandemic.^{27, 28} The difference in ICU admission rate, ventilator use, and mortality rate between the training set and prospective validation set can be explained by the temporal improvement in COVID-19 patients’ outcomes that was noted in other studies.^{4, 29, 30} In our study, for a cutoff of 0.1 for COVID-19 severity, our model had a sensitivity of 73.7% and specificity 79.9% in the prospective validation set, 52.2% and 88.1%, respectively in the PUI set, and 78% and 71%, respectively in the real-time validation set. These results show good discrimination for patients with scores associated with increased rates of the composite outcome.

The pandemic’s immense burden on medical systems prompted clinical researchers across the world to develop various models to predict clinical needs and resource allocation. One group of researchers published a review about triaging COVID-19 patients in ED and developed a theoretical algorithm.⁶ An emergency department from Israel screened over 7,900 patients and developed a model that consistently separated the COVID-19 from non-COVID-19 patients seeking acute care.⁷ A retrospective analysis of patients presenting with suspicion of COVID-19 in six EDs in Korea, found that triage screening is available for early isolation of the infection without the need for ED closure.⁸ The COVED was a quality improvement project developed in Australia to support ED clinicians and generate clinical prediction tools.⁹ In France, the emergency system calls were screened by a two-level filtering emergency medical communication; the new approach helped filter the COVID-19 suspicion acute care needs and discerned the real life-threatening clinical situations.¹¹ Predictive models are designed to support medical decision making in the context of clinical judgment. Notably, our ED physicians are not instructed to use the score to make specific medical decisions in isolation, but to better inform their evaluation and assist in making informed decisions in a complex clinical environment.

We designed our model to be feasible for use in the ED with the intention of estimating the risk of severe disease across patients with (or suspicion of) COVID-19 when being evaluated in this specific setting. Our model variables included demographics, comorbidities, home medication, vital signs, which are readily available in the ED, and did not include laboratory values, which do not always result in ED. The variables associated with a significantly higher risk for COVID-19 severity in our model were male gender, older age, other as race, increased temperature, increased respiratory rate, decreased oxygen saturation, inflammatory bowel disease. Comparable to our model, one study found vital signs, age, BMI, and comorbidities were the most important predictors.³¹ Oxygen saturation and patient’s age were strong risk factors for deterioration and mortality in COVID-19 in a systematic evaluation of predictive models.¹³ The use of warfarin appeared to be protective for our study's composite outcome, similar to another report.³² Hypercoagulability and need for anticoagulation were well recognized in COVID-19.³³ We included variables that were not significant on univariate analysis as well as variables that were protective. These variables made our model valuable in real life when many covariates and confounding factors exist and increased the model calibration. We included race and ethnicity in our model, which are independent predictors of worse outcomes per se.³⁴ We confirmed that the model performed equitably across racial/ethnic minorities. By increasing treatment and resource allocation to non-whites, we hypothesize that this will increase equitable treatment allocation.

The majority of prognostic models were developed and tested in COVID-19 positive patients, not in patients with suspected disease.^{31, 35–37} Aside from model development, our study's purpose was to implement and assess the predictive model in patients with suspected COVID-19 disease, or PUI. It is worth noting that 68% of our patients were discharged before the test resulted in our medical system. During this uncertainty period, many ED physicians are required to make clinical and triage decisions. Previous predictive models for patients with suspected COVID-19 infection have used imaging, demographics, signs and symptoms, vital signs to predict the likelihood of COVID-19 diagnosis, but they have not sought to predict the severity of the disease.^{12, 14, 38} The data from the registry of suspected COVID-19 in the emergency care (RECOVER) developed in 27 US states with the scope of characterizing the course, epidemiology, clinical features, and prognosis of the patients tested for COVID-19 in ED remains to be published.³⁹ A large PUI population study evaluated the risk factors associated with a composite of ICU admission, mechanical ventilation, and death; however, the study analyzed a retrospective cohort.⁴⁰ One study that implemented a predictive model in real-time focused on the positive outcome of the patients with COVID-19 diagnosis.⁴¹

We validated our model in a large PUI population and tested it in a good-sized real-time data set. Some prognostic models that showed good performance were small.^{42, 43} A large and qualitative clinical sample is paramount for implementation in the context of multiple predictive models for the COVID-19 disease severity available in the literature.⁴⁴ The largest study to date published in Great Britain used the 4 C Mortality Score to stratify the severity of the COVID-19.⁴⁵ In contrast to our model, the 4 C includes some laboratory values not always available in ED, and used data from COVID-19 positive patients admitted to the hospital. A systematic external validation of 22 prognostic models in a cohort of 411 patients with COVID-19 found that NEWS2 score that predicted ICU admission or death within 14 days for symptoms onset (0.78; 95% CI 0.73–0.83) achieved the highest AUROC.¹³ The model performance for predicting COVID-19 severity in our prospective validation, PUI, and real-time data sets is more robust than in the above mentioned external validation of the prognostic models.

These findings must be viewed within the context of the following limitations. First, this study was done within a single healthcare system in Midwest. Despite a large catchment area that includes surrounding states, these results are specific to the regional patient population in which the models were derived until they have been validated in other populations. Our model over-predicted the disease severity making it a valuable tool for patient safety and less for resource utilization.This study sought to develop, validate, and implement a prediction model to support clinical decision-making. Importantly, the model was never intended to replace clinical judgment, rather it was intended to complement and better inform providers, specifically when there is a large degree of clinical uncertainty. The effect to which clinical decisions were changed, and the degree for which this tool reduced cognitive burden have yet to be determined and were beyond the scope of this analysis.

COVID-19 has burdened healthcare systems from multiple different facets and findings ways to alleviate stress is crucial. Assisting in clinical decision making through predictive modeling may add to patient care, reduce undue variations in decision making, and optimize resource utilization, especially during a pandemic. We present the successful development and implementation of a COVID-19 prediction model for three different outcomes. The severity of illness composite outcome performed well in the PUI population despite being developed on a COVID-19 positive population. The effect on patient outcomes and resource use are needed to further assess the benefits of the model presented here.

Ethics approval and consent to participate:

This study was approved by the Institutional Review Board (IRB) at the University of Minnesota. We only included patients who did not opt out of research on admission.

Consent for publication:

Not applicable.

Availability of data and materials:

Data is available on a secure research website of our hospital system. Only researchers who have IRB approval for their studies have access to data.

Competing interests:

There were no competing interests for any of the authors.

Funding:

Not applicable.

Authors contributions:

MIL: this author made substantial contribution to the conception and design for the work, the interpretation of data, participated in drafting the work and revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

DL: this author made substantial contribution to the conception and design for the work, the interpretation of data, participated in drafting the work and revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

NEI:^* this author made substantial contribution to the conception and design for the work, the analysis and interpretation of data, participated in drafting the work and revising critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

KDB: this author made substantial contribution to the conception and design for the work, participated in revising critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

BB: this author made substantial contribution to the conception and design for the work, participated in revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

MP: this author made substantial contribution to the conception and design for the work, participated in revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

DM: this author made substantial contribution to the conception and design for the work, participated in revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

GBM: this author made substantial contribution to the conception and design for the work, participated in revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

DS: this author made substantial contribution to the acquisition of data for the work, participated in revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

MGU: this author made substantial contribution to the conception and design for the work, the analysis, acquisition and interpretation of data, participated in drafting the work and revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

CJT: this author made substantial contribution to the conception and design for the work, the analysis, acquisition and interpretation of data, participated in drafting the work and revising it critically for important intellectual content, gave final approval of the version to be published, agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Acknowledgments:

The authors wish to thank Sean Switzer, DO, Eric Murray, and Iva Ninkovic for valuable technical support, and Gyorgy Simon, PhD, for regression analysis and model building advice.

https://.
Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA Feb. 2020. doi:10.1001/jama.2020.2648.
Docherty AB, Harrison EM, Green CA, et al. Features of 20 133 UK patients in hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: prospective observational cohort study. BMJ May. 2020;369:m1985. doi:10.1136/bmj.m1985.
Asch DA, Sheils NE, Islam MN, et al. Variation in US Hospital Mortality Rates for Patients Admitted With COVID-19 During the First 6 Months of the Pandemic. JAMA Intern Med. Dec 2020. doi:10.1001/jamainternmed.2020.8193.
Ingraham NE, Tignanelli CJ. Fact Versus Science Fiction: Fighting Coronavirus Disease 2019 Requires the Wisdom to Know the Difference. Crit Care Explor Apr. 2020;2(4):e0108. doi:10.1097/CCE.0000000000000108.
Wallace DW, Burleson SL, Heimann MA, et al. An adapted emergency department triage algorithm for the COVID-19 pandemic. J Am Coll Emerg Physicians Open Aug. 2020. doi:10.1002/emp2.12210.
Levy Y, Frenkel Nir Y, Ironi A, et al. Emergency Department Triage in the Era of COVID-19: The Sheba Medical Center Experience. Isr Med Assoc J Aug. 2020;8(22):404–9.
Chung HS, Lee DE, Kim JK, et al. Revised Triage and Surveillance Protocols for Temporary Emergency Department Closures in Tertiary Hospitals as a Response to COVID-19 Crisis in Daegu Metropolitan City. J Korean Med Sci May. 2020;35(19):e189. doi:10.3346/jkms.2020.35.e189.
O'Reilly GM, Mitchell RD, Noonan MP, et al. Informing emergency care for COVID-19 patients: The COVID-19 Emergency Department Quality Improvement Project protocol. Emerg Med Australas. 2020;06(3):511–4. doi:10.1111/1742-6723.13513. 32 ) .
Jaffe E, Sonkin R, Alpert EA, Magid A, Knobler HY. Flattening the COVID-19 Curve: The Unique Role of Emergency Medical Services in Containing a Global Pandemic. Isr Med Assoc J Aug. 2020;8(22):410–6.
Penverne Y, Leclere B, Labady J, et al. Impact of two-level filtering on emergency medical communication center triage during the COVID-19 pandemic: an uncontrolled before-after study. Scand J Trauma Resusc Emerg Med Aug. 2020;28(1):80. doi:10.1186/s13049-020-00775-0.
Hüfner A, Kiefl D, Baacke M, et al. [Risk stratification through implementation and evaluation of a COVID-19 score: A retrospective diagnostic study]. Med Klin Intensivmed Notfmed Nov. 2020. doi:10.1007/s00063-020-00754-4.
Gupta RK, Marks M, Samuels THA, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study. Eur Respir J Sep. 2020. doi:10.1183/13993003.03498-2020.
Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;04:369:m1328. doi:10.1136/bmj.m1328.
Shamsoddin E. Can medical practitioners rely on prediction models for COVID-19? A systematic review. Evid Based Dent. 2020;09(3):84–6. doi:10.1038/s41432-020-0115-5. 21 ) .
COVID-19 PRECISE. https://www.covprecise.org/living-review/.
Rada G, Verdugo-Paiva F, Ávila C, et al. Evidence synthesis relevant to COVID-19: a protocol for multiple systematic reviews and overviews of systematic reviews. Medwave. 2020;04(3):e7868. doi:10.5867/medwave.2020.03.7867. 20 ) .
Clift AK, Coupland CAC, Keogh RH, et al. Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ. 2020;10:371:m3731. doi:10.1136/bmj.m3731.
Petrilli CM, Jones SA, Yang J, et al. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study. BMJ. May 22 2020;369:m1966. doi:10.1136/bmj.m1966.
Ingraham NE, Purcell LN, Karam BS, et al. Racial/Ethnic Disparities in Hospital Admissions from COVID-19 and Determining the Impact of Neighborhood Deprivation and Primary Language. medRxiv. Sep 3 2020;doi:10.1101/2020.09.02.20185983.
Lusczek ER, Ingraham NE, Karam BS, et al. Characterizing COVID-19 clinical phenotypes and associated comorbidities and complication profiles. PloS one. 2021;16(3):e0248956.
Bramante CT, Ingraham NE, Murray TA, et al. Metformin and risk of mortality in patients hospitalised with COVID-19: a retrospective cohort analysis. Lancet Healthy Longev Jan. 2021;2(1):e34–41. doi:10.1016/S2666-7568(20)30033-7.
Dutta N, Ingraham NE, Usher MG, Fox C, Tignanelli CJ, Bramante CT. We Should Do More to Offer Evidence-Based Treatment for an Important Modifiable Risk Factor for COVID-19: Obesity. J Prim Care Community Health Jan-Dec. 2021;12:2150132721996283. doi:10.1177/2150132721996283.
Bramante CT, Buse J, Tamaritz L, et al. Outpatient metformin use is associated with reduced severity of COVID-19 disease in adults with overweight or obesity. J Med Virol Feb 12 2021;doi:10.1002/jmv.26873.
Elliott J, Bodinier B, Whitaker M, et al. COVID-19 mortality in the UK Biobank cohort: revisiting and evaluating risk factors. Eur J Epidemiol Feb. 2021. doi:10.1007/s10654-021-00722-y.
Guan WJ, Liang WH, Zhao Y, et al. Comorbidity and its impact on 1590 patients with COVID-19 in China: a nationwide analysis. Eur Respir J 05 2020;55(5)doi:10.1183/13993003.00547-2020.
Mackey K, Ayers CK, Kondo KK, et al. Racial and Ethnic Disparities in COVID-19-Related Infections, Hospitalizations, and Deaths: A Systematic Review. Ann Intern Med. 2021;03(3):362–73. doi:10.7326/M20-6306. 174 ) .
Ogedegbe G, Ravenell J, Adhikari S, et al. Assessment of Racial/Ethnic Disparities in Hospitalization and Mortality in Patients With COVID-19 in New York City. JAMA Netw Open. 2020;12(12):e2026881. doi:10.1001/jamanetworkopen.2020.26881. 3 ) .
Auld SC, Caridi-Scheible M, Blum JM, et al. ICU and Ventilator Mortality Among Critically Ill Adults With Coronavirus Disease 2019. Crit Care Med. 2020;09(9):e799–804. doi:10.1097/CCM.0000000000004457. 48 ) .
Kristinsson B, Kristinsdottir LB, Blondal AT, et al. Nationwide Incidence and Outcomes of Patients With Coronavirus Disease 2019 Requiring Intensive Care in Iceland. Crit Care Med. 2020;11(11):e1102–5. doi:10.1097/CCM.0000000000004582. 48 ) .
Hao B, Sotudian S, Wang T, et al. Early prediction of level-of-care requirements in patients with COVID-19. Elife. 10 2020;9doi:10.7554/eLife.60519.
Harrison RF, Forte K, Buscher MG, et al. The Association of Preinfection Daily Oral Anticoagulation Use and All-Cause in Hospital Mortality From Novel Coronavirus 2019 at 21 Days: A Retrospective Cohort Study. Crit Care Explor Jan. 2021;3(1):e0324. doi:10.1097/CCE.0000000000000324.
Obi AT, Barnes GD, Napolitano LM, Henke PK, Wakefield TW. Venous thrombosis epidemiology, pathophysiology, and anticoagulant therapies and trials in severe acute respiratory syndrome coronavirus 2 infection. J Vasc Surg Venous Lymphat Disord. 2021;01(1):23–35. doi:10.1016/j.jvsv.2020.08.030. 9 ) .
Rentsch CT, Kidwai-Khan F, Tate JP, et al. Covid-19 by Race and Ethnicity: A National Cohort Study of 6 Million United States Veterans. medRxiv May. 2020. doi:10.1101/2020.05.12.20099135.
Haimovich AD, Ravindra NG, Stoytchev S, et al. Development and Validation of the Quick COVID-19 Severity Index: A Prognostic Tool for Early Clinical Decompensation. Ann Emerg Med. 2020;10(4):442–53. doi:10.1016/j.annemergmed.2020.07.022. 76 ) .
Castro VM, McCoy TH, Perlis RH. Laboratory Findings Associated With Severe Illness and Mortality Among Hospitalized Individuals With Coronavirus Disease 2019 in Eastern Massachusetts. JAMA Netw Open. 2020;10(10):e2023934. doi:10.1001/jamanetworkopen.2020.23934. 3 ) .
Ioannou GN, Locke E, Green P, et al. Risk Factors for Hospitalization, Mechanical Ventilation, or Death Among 10 131 US Veterans With SARS-CoV-2 Infection. JAMA Netw Open. 2020;09(9):e2022310. doi:10.1001/jamanetworkopen.2020.22310. 3 ) .
Sun Y, Koh V, Marimuthu K, et al. Epidemiological and Clinical Predictors of COVID-19. Clin Infect Dis. 2020;07(15):786–92. doi:10.1093/cid/ciaa322. 71 ) .
Kline JA, Pettit KL, Kabrhel C, Courtney DM, Nordenholz KE, Camargo CA. Multicenter registry of United States emergency department patients tested for SARS-CoV-2. J Am Coll Emerg Physicians Open Dec. 2020;1(6):1341–8. doi:10.1002/emp2.12313.
Singer AJ, Morley EJ, Meyers K, et al. Cohort of Four Thousand Four Hundred Four Persons Under Investigation for COVID-19 in a New York Hospital and Predictors of ICU Care and Ventilation. Ann Emerg Med. 2020;10(4):394–404. doi:10.1016/j.annemergmed.2020.05.011. 76 ) .
Razavian N, Major VJ, Sudarshan M, et al. A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients. NPJ Digit Med. 2020;3:130. doi:10.1038/s41746-020-00343-x.
Hu H, Yao N, Qiu Y. Comparing Rapid Scoring Systems in Mortality Prediction of Critically Ill Patients With Novel Coronavirus Disease. Acad Emerg Med. 2020;06(6):461–8. doi:10.1111/acem.13992. 27 ) .
Ji D, Zhang D, Xu J, et al. Prediction for Progression Risk in Patients With COVID-19 Pneumonia: The CALL Score. Clin Infect Dis. 2020;09(6):1393–9. doi:10.1093/cid/ciaa414. 71 ) .
Martin GP, Sperrin M, Sotgiu G. Performance of Prediction Models for Covid-19: The Caudine Forks of the External Validation. Eur Respir J Oct. 2020. doi:10.1183/13993003.03728-2020.
Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ. 09 2020;370:m3339. doi:10.1136/bmj.m3339.

Supplemental.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Implementation and Evaluation of a Novel COVID-19 Prognostic Score for Patients with Suspected COVID-19

Status:

Version 1

Abstract

Figures

Background

Methods

Study Design and Setting

Selection of Participants

Training dataset

Validation datasets

Factors of interest and factors selected for the final model

Outcomes

Analysis

Model construction

Model implementation

Development of a metric of clinical deterioration and predictive threshold

Results

Retrospective validation and in PUI

Real-time validation

Discussion

Conclusions

Declarations

References

Supplementary Files

Status:

Version 1