Study population
A total of 15830 patients visited the ED during August 2018-July 2019. Of these, 1927 (12.2%) patients had suspected sepsis, and 305 were excluded because they had been treated and transferred from other units. Consequently, 1622 patients with suspected sepsis were included in the final analysis. Of these, 574 (28.2%) met the primary outcome of all-cause in-hospital mortality, and 280 (17.3%) died within 7 days of admission. A total of 1382 (85.2%) were diagnosed with sepsis at hospital disposition according to Sepsis-3 definition. Patient characteristics are shown in Table 1. The study population’s mean age ± SD was 72.6 ± 15.4 years, and 51.1% were female. Patients who had all-cause in-hospital mortality were older, had a greater prevalence of neoplasia, and more history of recent hospital admission. They also had significantly more severe abnormal initial vital signs, higher serum white blood cells and band-form cell counts, as well as higher rate of positive hemoculture, inotropic drug prescription and ICU admission compared with patients discharged alive.
Table 1
Baseline characteristics of patients with suspected sepsis
Characteristic
|
All
(1622)
|
Dead
(457)
|
Alive
(1165)
|
p-value
|
Age
|
72.6 ± 15.4
|
74.4 ± 15.1
|
71.9 ± 15.5
|
0.004
|
Sex(female)
|
829 (51.1)
|
233 (51.0)
|
596 (51.2)
|
0.95
|
Underlying disease
|
|
|
|
|
Diabetes mellitus
|
513 (31.6)
|
144 (31.5)
|
369 (31.7)
|
0.50
|
Hypertension
|
888 (54.7)
|
236 (51.6)
|
652 (56.0)
|
0.07
|
Hyperlipidaemia
|
525 (32.4)
|
160 (35.0)
|
365 (31.3)
|
0.09
|
CKD or ESRD
|
294 (18.1)
|
82 (17.9)
|
212 (18.2)
|
0.48
|
Coronary artery disease
|
206 (12.7)
|
51 (11.2)
|
155 (13.3)
|
0.14
|
Neuro-debilitating diseases
|
401 (24.7)
|
111 (24.2)
|
290 (24.8)
|
0.40
|
Cancer
|
404 (24.9)
|
164 (35.9)
|
240 (20.6)
|
< 0.001
|
Recent admission < 3 months
|
729 (44.9)
|
245 (53.6)
|
484 (41.5)
|
< 0.001
|
Infection site
|
|
|
|
|
Respiratory tract
|
982 (60.5)
|
294 (64.3)
|
688 (59.1)
|
0.04
|
Urinary tract
|
198 (12.2)
|
44 (9.6)
|
154 (13.2)
|
|
Other known sites
|
139 (8.6)
|
26 (5.7)
|
113 (9.7)
|
|
Unknown
|
303 (18.7)
|
93 (20.4)
|
210 (18.0)
|
|
Type of infection
|
|
|
|
|
Community-acquired
|
830 (51.2)
|
202 (44.2)
|
628 (53.9)
|
< 0.001
|
Healthcare-associated
|
89 (5.5)
|
18 (3.9)
|
71 (6.1)
|
|
Hospital-associated
|
703 (43.3)
|
237 (51.9)
|
466 (40.0)
|
|
Vital signs and mental status at time of sepsis suspicion
|
|
|
|
Body temperature (oC)
|
37.8 ± 8.4
|
38.1 ± 3.7
|
37.6 ± 3.7
|
0.40
|
Respiratory rate (breaths/min)
|
31.1 ± 8.6
|
32 ± 8.1
|
30.8 ± 8.7
|
0.01
|
Pulse rate (beats/min)
|
102.6 ± 38.4
|
102.9 ± 27.4
|
102.5 ± 42
|
0.80
|
Systolic blood pressure (mmHg)
|
125.4 ± 37.3
|
121 ± 43.5
|
127.6 ± 34.3
|
0.01
|
Diastolic blood pressure (mmHg)
|
71.3 ± 26.7
|
69.1 ± 19.3
|
72.1 ± 19.4
|
0.016
|
Mean arterial pressure (mmHg)
|
89.3 ± 26.2
|
86.4 ± 23.8
|
90.4 ± 26.9
|
0.004
|
Oxygen saturation (%)
|
91.9 ± 8.7
|
89.9 ± 10.2
|
92.7 ± 7.8
|
< 0.001
|
Glasgow coma scale score
|
12.5 ± 2.5
|
11.2 ± 2.9
|
13.0 ± 2.0
|
< 0.001
|
Early warning scores at time of sepsis suspicion
|
|
|
|
SIRS
|
2.40 ± 0.97
|
2.47 ± 0.90
|
2.37 ± 0.99
|
0.08
|
qSOFA
|
1.38 ± 0.65
|
1.52 ± 0.70
|
1.32 ± 0.62
|
< 0.0001
|
NEWS
|
8.05 ± 3.30
|
8.90 ± 3.37
|
7.71 ± 3.21
|
< 0.0001
|
REMS
|
9.08 ± 3.12
|
10.06 ± 3.12
|
8.70 ± 3.00
|
< 0.0001
|
Laboratory results
|
|
|
|
|
White blood cells count (cells/mm3)
|
12822.5 ± 9514.8
|
13474.1 ± 9755.8
|
12566.9 ± 9410.5
|
0.08
|
Band form (%)
|
2.5 ± 1.0
|
3.9 ± 1.6
|
1.9 ± 0.6
|
< 0.001
|
Hemoculture positive
|
276 (17.0)
|
100 (21.9)
|
176 (15.1)
|
0.001
|
ED management
|
|
|
|
|
Inotropic drugs
|
340 (21.0)
|
153 (33.5)
|
187 (16.1)
|
< 0.001
|
Outcome
|
|
|
|
|
Length of hospital stay (days)
|
6 (3,12)
|
5 (2,12)
|
7 (3,12)
|
0.38
|
ED disposition
|
|
|
|
|
ICU admission
|
82 (51.0)
|
38 (8.3)
|
44 (3.8)
|
< 0.001
|
Note: data presented as n (%), mean ± SD or median (IQR) |
Abbreviations: CKD, chronic kidney disease; ESRD, end-stage renal disease; SIRS, systemic inflammatory response syndrome; qSOFA, quick Sequential Organ Failure Assessment; NEWS, National Early Warning Score; REMS, Rapid Emergency Medicine Score; ICU, intensive care unit. |
Score Performance
Overall performance assessed by scaled Brier score and Nagelkerke’s R square showed REMS had the best overall performance, followed by NEWS, qSOFA, and SIRS (Table 2).
Table 2
Early warning score performance and clinical utility for all-cause in-hospital mortality and mortality within 7 days of admission in patients with suspected sepsis
|
|
Discrimination
|
|
Calibration
|
|
Overall performance
|
|
Clinical utility
|
Score
|
|
AUROC
(95%CI)
|
|
Hosmer-Lemeshow Test
|
|
Scaled Brier Score (%)
|
Nagelkerke’s R-Square
(%)
|
|
Score category
|
Sensitivity
(95%CI)
|
Specificity
(95%CI)
|
PPV
(95%CI)
|
NPV
(95%CI)
|
LR+
(95%CI)
|
LR-
(95%CI)
|
In-hospital mortality
|
SIRS
|
|
0.524
(0.494, 0.553)
|
|
0.074
|
|
0.2
|
0.2
|
|
SIRS ≥ 2
|
85.6
(82-88.7)
|
19.8
(17.6–22.2)
|
29.5
(28.5–30.5)
|
77.8
(73.1–81.8)
|
1.1
(1.0-1.7)
|
0.7
(0.6–0.9)
|
qSOFA
|
|
0.577
(0.549, 0.604)
|
|
0.688
|
|
2.0
|
1.9
|
|
qSOFA ≥ 2
|
45.3
(40.7–50)
|
69.0
(66.3–71.7)
|
36.4
(33.4–39.6)
|
76.3
(74.6–77.9)
|
1.5
(1.3–1.7)
|
0.8
(0.7–0.9)
|
NEWS
|
|
0.606
(0.575, 0.636)
|
|
0.059
|
|
2.7
|
2.6
|
|
NEWS ≥ 8
|
68.1
(63.7–72.3)
|
47.8
(44.9–50.7)
|
33.8
(32.0-35.7)
|
79.2
(76.7–81.5)
|
1.3
(1.2–1.4)
|
0.7
(0.6–0.8)
|
REMS
|
|
0.623
(0.593, 0.653)
|
|
0.565
|
|
4.0
|
3.8
|
|
REMS ≥ 9
|
66.5
(62-70.8)
|
50.1
(47.2–53)
|
34.4
(32.4–36.3)
|
79.2
(76.8–81.5)
|
1.3
(1.2–1.5)
|
0.7
(0.6–0.8)
|
Mortality within 7 days of admission
|
SIRS
|
|
0.539
(0.504, 0.571)
|
|
0.025
|
|
0.3
|
0.3
|
|
SIRS ≥ 2
|
88.6
(84.3–92.1)
|
19.8
(17.7, 22.0)
|
18.7
(18.0-19.5)
|
89.2
(85.5–92.1)
|
1.1
(1.1–1.2)
|
0.6
(0.4–0.8)
|
qSOFA
|
|
0.589
(0.556, 0.622)
|
|
0.154
|
|
1.6
|
1.6
|
|
qSOFA ≥ 2
|
49.3
(43.3–55.3)
|
68.5
(65.9–71.0)
|
24.6
(22.0-27.3)
|
86.6
(85.2–88.0)
|
1.6
(1.4–1.8)
|
0.7
(0.7–0.8)
|
NEWS
|
|
0.625
(0.589, 0.660)
|
|
0.189
|
|
2.7
|
2.7
|
|
NEWS ≥ 8
|
70.0
(64.3–75.3)
|
46.1
(43.4–48.8)
|
21.3
(19.8–22.9)
|
88.0
(85.9–89.9)
|
1.3
(1.2–1.4)
|
0.7
(0.5–0.8)
|
REMS
|
|
0.645
(0.608, 0.679)
|
|
0.234
|
|
4.0
|
3.8
|
|
REMS ≥ 10
|
57.5
(51.5–63.4)
|
60.8
(58.1–63.4)
|
23.4
(21.3–25.7)
|
87.3
(85.6–88.8)
|
1.5
(1.3–1.7)
|
0.7
(0.6–0.8)
|
Notes: cut-off values for SIRS and qSOFA were chosen from the literature. Cut-off values for NEWS and REMS were chosen by optimal Youden Index. |
Abbreviations: AUROC, area under the receiver operator characteristics curve; CI, confidence interval; LR+, positive likelihood ratio; LR-, negative likelihood ratio; NEWS, National Early Warning Score; NPV, negative predictive value; PPV, positive predictive value; qSOFA, quick Sequential Organ Failure Assessment; REMS, Rapid Emergency Medicine Score; SIRS, systemic inflammatory response syndrome. |
The discrimination performance for all-cause in-hospital mortality was highest for REMS (AUROC 0.62; 95%CI 0.59, 0.65), followed by NEWS (AUROC 0.61; 95%CI 0.58–0.64), qSOFA (AUROC 0.58; 95%CI 0.55–0.61), and SIRS (AUROC 0.52; 95%CI 0.49–0.55) (Table 2 and Fig. 3). All EWSs had better discrimination by AUROCs for all-cause mortality within 7 days of admission compared to all-cause in-hospital mortality although the trend of results of AUROCs was similar (Table 2 and Fig. 3). In pairwise comparisons between EWSs, REMS had significantly better discrimination than all other EWSs except for NEWS for both outcomes (Table 3). In subgroup analyses, all EWSs show better discrimination for all-cause in-hospital mortality and mortality within 7 days in those aged greater or equal to 70 years than those aged less than 70 years (Table S2). All EWSs show better discrimination for all-cause mortality in those without chronic comorbidities compared with those with at least one chronic comorbidity, but an opposite trend was seen for all-cause mortality within 7 days of admission for SIRS and NEWS (Table S2). Calibration for SIRS showed underestimation of predicted mortality risk at lowest and highest SIRS scores (Fig. 4 and S2). The other EWSs tended to be well-calibrated except for at underestimation of all-cause in-hospital mortality risk at high predicted probabilities in NEWS and for all-cause mortality within 7 days of admission for both NEWS and REMS (Fig. 4 and S2). However, only a few patients had very high NEWS and REMS scores (Fig. 2).
Table 3
Pairwise comparisons of area under the receiver operator characteristic curve of early warning scores for in-hospital mortality and mortality within 7 days among patients with suspected sepsis
|
In-hospital mortality
|
SIRS
|
qSOFA
|
NEWS
|
REMS
|
Mortality within 7 days
|
SIRS
|
|
**0.007
|
***<0.001
|
***<0.001
|
qSOFA
|
*0.03
|
|
*0.05
|
**0.005
|
NEWS
|
***<0.001
|
*0.04
|
|
0.27
|
REMS
|
***<0.001
|
**0.004
|
0.26
|
|
Notes: comparison were performed by bootstrap test. *p < 0.05 **p < 0.01 ***p < 0.001 |
Abbreviations: NEWS, National Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; REMS, Rapid Emergency Medicine Score; SIRS, systemic inflammatory response syndrome. |
Additional Contribution Of Ewss To Baseline Mortality Risk Model
All baseline risk model plus an EWS had significantly better discrimination than the baseline risk model for all-cause in-hospital mortality (Table S3). For all-cause in-hospital mortality, the baseline risk model plus REMS showed the greatest improvement in discrimination over the baseline mortality risk models, followed by NEWS, qSOFA, and SIRS (Table S4 and Figure S3). NEWS and REMS had significantly better discrimination than SIRS and qSOFA, but REMS was not significantly superior to NEWS (Table S3). The trend of results was generally similar for all-cause mortality within 7 days of admission except that NEWS did not have significantly better discrimination than qSOFA for both mortality outcomes (Table S3 and S4 and Figure S3). Integrated discrimination improvement also showed REMS had the greatest improvement over the baseline risk models, followed by NEWS, qSOFA, and SIRS (Table S4). REMS showed the greatest percentage improvement in sensitivity for all-cause in-hospital mortality and all-cause mortality within 7 days of admission compared with the baseline risk model (9.98% and 18.3%, respectively) (Table S4). Calibration plots for baseline risk model plus EWSs for both mortality outcomes generally were well-calibrated up to a predicted probability of 0.5 except for SIRS for all-cause in-hospital mortality. Above a predicted probability of 0.5 some models showed some over- or under-estimation of mortality risk (Figures S4 and S5).
The clinical usefulness of the EWS scores was assessed by sensitivity, specificity, PPV, NPV, LR+, and LR- (Table 2). For all-cause in-hospital mortality, SIR ≥ 2 had the highest sensitivity (85.6%; 95%CI 82.0, 88.7) but the least specificity (19.8%; 95%CI 17.6, 22.2%). qSOFA ≥ 2 had the highest specificity (69.0%; 95%CI 66.3, 71.7%) but lowest sensitivity (45.3%; 95%CI 40.7, 50.0%). At optimal Youden Index cut points, NEWS ≥ 8 and REMS ≥ 9 had a balance of sensitivity (68.1; 95%CI 63.7, 72.3 and 66.5; 95%CI 62, 70.8, respectively) and specificity (47.8; 95%CI 44.9, 50.7 and 50.1; 95%CI 47.2, 53.0, respectively), which favored sensitivity. PPV and NPV were similar for all EWSs except for SIRS, which was much lower. qSOFA had the highest LR+ (1.5; 95%CI 1.3, 1.7) while LR- of all EWSs were similar. For all-cause mortality within 7 days of admission, results of sensitivity and specificity were similar except that the optimal cut-off point for REMS was ≥ 10, and results for PPV, NPV, LR+, and LR- were similar to the primary outcome.
In subgroup analysis for all-cause in-hospital mortality, results were generally similar to the full cohort except that REMS ≥ 9 had higher specificity in those aged less than 70 years and higher sensitivity in those aged greater or equal to 70 years (Table S5). Subgroup analysis of all-cause mortality within 7 days of admission showed similar changes to all-cause in-hospital mortality in sensitivity and specificity for REM ≥ 10 (Table S5).
The NB for all-cause in-hospital mortality showed SIRS and qSOFA did not have an advantage over a treat-all strategy for all plausible threshold probabilities. The range of threshold probabilities over which any NB advantage over a treat-all strategy was 18–20% for NEWS and 14–20% for REMS. The number of avoided interventions per 100 patients at a threshold probability of 20% (NWT 5) using NEWS or REMS would be 1.1 and 2.6, respectively (Fig. 5). For all-cause mortality within 7 days of admission, all EWSs showed advantageous NB over a treat-all strategy within the plausible threshold probability range. NEWS and REMS had the lowest threshold probabilities at which advantage over a treat-all strategy began at 10% (NWT 10) and 4% (NWT 25), respectively. The number of avoided interventions per 100 patients in a hypothetical population at threshold probabilities of 10% and 20% for NEWS would be 2.4 and 23, respectively, and for REMS would be 2.4 and 25, respectively (Fig. 5). Results of NB analysis for baseline risk model + EWS were similar for both mortality outcomes (Figure S6).
Eighty-two percent (n = 1325) of all patients with suspected sepsis met at least 2 SIRS criteria (SIR ≥ 2), and 70% (n = 934) did not meet the primary outcome (false positive). Only 35% (n = 568) met at least 2 qSOFA criteria (qSOFA ≥ 2), and about 64% (n = 361) were false positive. NEWS ≥ 8 and REMS ≥ 9 could detect similar proportions of patients (about 56%), but REMS ≥ 9 had the highest absolute risk difference compared to the other three EWSs (difference 14%; 95%CI 9.7, 18.3). Similarly, REMS ≥ 10 could also provide the highest absolute risk difference in predicting mortality within 7 days of admission (difference 12.7%; 95%CI 10.7, 15.0%) (Table 4).
Table 4
Classification according to sepsis criteria.
Outcomes
|
All patients, no (%)
|
SIRS, no (%)
|
Absolute difference, % (95%CI)
|
qSOFA, no (%)
|
Absolute difference, % (95%CI)
|
NEWS, no (%)
|
Absolute difference, % (95%CI)
|
REMS, no (%)
|
Absolute difference, % (95%CI)
|
< 2
(n = 297)
|
≥ 2 (n = 1325)
|
< 2 (n = 1054)
|
≥ 2
(n = 568)
|
< 8
(n = 702)
|
≥ 8
(n = 920)
|
< 9
(n = 713)
|
≥ 9
(n = 909)
|
In-hospital death
|
457 (28.2)
|
66 (22.2)
|
391 (29.5)
|
7.3 (2.0-12.6)
|
250 (23.7)
|
207 (36.4)
|
12.7 (8.0-26.3)
|
146 (20.8)
|
311 (33.8)
|
13.0 (8.7–17.3)
|
145 (20.3)
|
312 (34.3)
|
14.0 (9.7–18.3)
|
|
All patients, no (%)
|
< 2
(n = 297)
|
≥ 2 (n = 1325)
|
Absolute difference, % (95%CI)
|
< 2 (n = 1054)
|
≥ 2
(n = 568)
|
Absolute difference, % (95%CI)
|
< 8
(n = 702)
|
≥ 8
(n = 920)
|
Absolute difference, % (95%CI)
|
< 10
(n = 935)
|
≥ 10
(n = 687)
|
Absolute difference, % (95%CI)
|
Death within 7 days
|
280 (17.3)
|
32 (10.8)
|
248 (18.7)
|
8.0 (3.8–12.1)
|
142 (13.5)
|
138 (24.3)
|
10.8 (6.7–14.9)
|
84 (12.0)
|
196 (21.3)
|
9.3 (5.8–12.9)
|
119 (12.7)
|
161 (23.4)
|
12.7 (10.7–15.0)
|
Abbreviations: CI, confidence interval; SIRS, systemic inflammatory response score; qSOFA, quick sequential organ failure assessment score; NEWS, national early warning score; REMS, rapid emergency medicine scores. |