Development and Internal Validation of a Model Predicting Safe, Early Discharge Among Patients Presenting to the Emergency Department with an Infection.

Background: Prolonged hospitalization is associated with high costs and mortality, and increases the chance of adverse events. This study aimed to identify predictors of safe, early discharge in patients presenting to the Emergency Department (ED) with an infection. Methods: This prospective observational study was performed in the ED of a tertiary care teaching hospital. Adult non-trauma patients with suspected infection and at least two Systemic Inammatory Response Syndrome (SIRS) criteria were included. Exclusion criteria were intensive care unit admission and transfer to another hospital. Safe, early discharge was dened as hospital-discharge within 24 hours without disease-related death or readmission to our hospital during the rst 7 days. A prediction model for safe, early discharge was created using a multivariate logistic regression analysis and validated with k-fold cross-validation. Results: 1381 patients were included, of whom 354 (25.6%) met the safe, early discharge criteria. Parameters associated with safe, early discharge were younger age, absence of co-morbidities, living independently, yellow or green triage urgency, absence of ambulance transport, absence of general practitioner referral, normal clinical impression, (q)SOFA, PIRO, MEDS, NEWS and SIRS scores, absence of abnormal vital sign measurements and absence of kidney and respiratory failure. A prediction model for safe, early discharge was developed with an area under the curve (AUC) of 0.824. Internal validation generated a minimal drop in performance, indicating a good t. Conclusion: By identifying predictors of clinical improvement and combining several readily available parameters in the ED setting, a model for safe, early discharge with good prognostic performance was created. Percentage of missing values per variable are shown in parentheses. Safe, early discharge, hospital-discharge ≤ 24 hours, without disease-related readmission or death ≤ 7 days. GP general practitioner ED Emergency Department COPD Chronic Obstructive Pulmonary Disease CIS Clinical Impression Score NEWS National early warning score SIRS Systemic Inammatory Response Syndrome SOFA Sequential Organ Failure Assessment qSOFA quick SOFA PIRO Predisposition Insult Response Organ dysfunction MEDS Mortality in Emergency Department Sepsis. 1 educated past the age of 14; 2 ≥ 1 cigarette per day; 3 ≥ 1 unit of alcohol per week; 4 working diagnosis upon leaving the ED; 5 dened as life-threatening organ failure caused by a dysregulated host response to infection; 6 at ED admission; 7 >4 mmol/L 8 the safe, early discharge group has fewer comorbidities. * p < 0.05.


Introduction
Sepsis is de ned as a dysregulated and disproportionate in ammatory response to an infection, causing life-threatening organ failure. [1] Sepsis is estimated to annually affect close to 49 million people worldwide. [2] Since the start of the Surviving Sepsis Campaign (SSC) in 2002, research has focused on aggressive recognition and treatment of sepsis. [3,4] Whether aggressive treatment is feasible for every potentially septic patient has been questioned in recent years. [5][6][7][8][9][10] Since not all patients with an infection will develop sepsis and only a minority will progress to severe sepsis requiring Intensive Care Unit (ICU) admission. [11][12][13][14][15] Further research should therefore focus on patient improvement, alongside deterioration, of patients presenting to the ED with an infection.
While clinical deterioration can, to a certain degree, be predicted with the use of clinical judgment, clinical scoring systems and repeated vital sign measurements, clinical improvement, on the other hand, cannot. [11,13,[16][17][18][19][20][21][22][23] Length of in-hospital stay of patients presenting to the ED with an infection varies, with an average of 1 to 7 days (range 0-38 days). [24][25][26] Which of these patients will develop sepsis and, in turn, need prolonged hospitalization, remains a complex and not well understood process. [27][28][29][30] entail: being 18 years or older, having a suspected or con rmed infection and having two or more SIRS criteria as de ned by the International Sepsis De nitions Conference. [41] Exclusion criteria were ICU admission and transfer to another hospital.

Data collection
A nurse measured patients vital signs and performed triage using the Emergency Severity Index (ESI) upon arrival to the ED. [42] Subsequently, patients vital signs were measured every 30 minutes by a trained member of our research staff, minimizing the amount of missing data. Vital sign measurements stopped if the patient was discharged or if the patient was relocated. Temperature was measured using an electronic tympanic ear thermometer (Genius 2; Mountainside Medical Equipment, Marcy, New York, USA). The other vital sign measurements were performed using a bedside patient monitor (IntelliVue MP30 System with Multi-Measurement Module; Philips, Eindhoven, the Netherlands). Vital signs measured by the referring general practitioner and emergency medical services (EMS) were imported from electronic patient records. After primary assessment of the patient, including vital signs measurements, the attending physician and nurse were asked for their clinical impression of the patient. To ensure su cient independence of answers, the physician and nurse were separately asked for their clinical impression. Their impression was recorded using the Clinical Impression Score (CIS), which ranges from 1 (not ill) to 10 (extreme illness). [43] All data was collected by trained research assistants of the 'Sepsis Research Team' of our hospital.
The qSOFA score and the NEWS were determined twice during ED stay. Once using the initial vital parameters measured upon admission to the ED and once more using the last known vital signs at the ED. The qSOFA score comprises of three parameters: altered mental status (GCS < 15), respiratory frequency > 22/min and systolic blood pressure < 100 mmHg. [1] The NEWS score is based on oxygen saturation, respiratory rate, use of supplemental oxygen, heart rate, systolic blood pressure and level of consciousness. This score ranges from 0 to 20 and a positive score indicates signi cant physiological derangement. [21] For each patient, the PIRO, MEDS and SOFA scores were calculated using initial vital parameters during admission, results from routine blood analysis, sociodemographic information gathered during admission and the information imported from electronic medical records. [15,44,45] These medical records were subsequently monitored to collect demographic data and patient history. They also allowed for patient follow-up.
To determine normalization of vital signs, patients rst vital sign measurement in the ED was compared to their last measured vital sign in the ED. Due to the difference between pre-hospital electronic records and ours, these data points could not be used due to a large number of missing variables. Normalization was hereby de ned as transition of the individual vital sign from abnormal to within normal range. Staying normal of vital signs was de ned as both rst and last vital sign measurement being normal. Using the aforementioned sepsis-related scoring systems, we a priori de ned which vital sign ranges would be regarded as being abnormal. These abnormal ranges are displayed in Table 1 of the appendix. Vital signs outside of these ranges were considered normal.

Missing data
Since arterial blood gas analyses (ABGs) were not performed in all patients meeting the inclusion criteria, the P/F ratio, partial pressure of oxygen (PaO 2 ) divided by fractional inspired oxygen (FiO 2 ), and lactate were the most frequent missing values. To calculate the SOFA scoring without a known P/F ratio, we estimated the PaO 2 using the peripheral oxygen saturation (SpO 2 ) that was known in all patients. [46,47] We chose the non-linear equation to calculate the P/F ratio in patients with absent PaO2 and a SpO2 ≤ 97%. If PaO2 was missing and the SpO2 was > 97%, we scored patients P/F ratio as normal. A missing value for blood lactate, as well as other missing values to calculate the clinical scoring systems used in this study, were deemed normal if missing. This method of data imputation provides similar outcomes for the qSOFA, PIRO and MEDS score compared to imputation with the multiple imputations function of SPSS. [48] The percentage of missing data points per variable will be shown in parentheses.

Endpoints and de nitions
The primary endpoint of the study is hospital length of stay (LOS). Secondary outcome measures were sociodemographic information, triage urgency, arrival mode, development of kidney, liver and respiratory failure, lactate, vital sign and score progression during ED stay and the NEWS, SIRS, (q)SOFA, PIRO and MEDS score at ED admission. Safe, early discharge was de ned as hospital-discharge within 24 hours, without disease-related readmission to our hospital or death during the rst 7 days. Development of kidney failure, or Acute Kidney Injury (AKI), was de ned using the Kidney Disease Improving Global Outcomes (KDIGO) criteria. These criteria are: an increase in serum creatinine 1.5 times the baseline (presumed or known to have occurred within the last 7 days) or an increase by 26.5 µmol/L (0.3 mg/dL) within 48 hours. [49] Liver failure was de ned as total bilirubin level > 34.2 µmol/L (2.0 mg/dL) and either a transaminase level or alkaline phosphatase above twice the normal limit. [11,50] Respiratory failure was de ned as the need for mechanical ventilation, either hypercapnia (PaCO2 > 6.5 kPa) or hypoxemia (PaO2 < 8.0 kPa) in the arterial blood gas analysis or either a SpO2 < 90% when breathing room air or < 95% with at least 2L/min of oxygen supplementation. [51] Statistics The Shapiro-Wilk test for normality was used to test for normality. For normally distributed data, the mean and SD were calculated. For non-normally distributed data, the median and interquartile ranges (IQR) were calculated. For categorical variables, frequency and percentage of cases were calculated.
Continuous data was analyzed using the Mann-Whitney U test and categorical data was analyzed with the Chi-square test.
To determine the relationship between clinical scoring systems and safe, early discharge, receiver operator characteristic (ROC) curves and the area under the ROC curve [area under the curve (AUC)] were calculated. The Wilcoxon signed rank test with continuity correction was used to test the AUCs against the null hypothesis (AUC = 0.5). Cut-off point, sensitivity, speci city and positive/negative predictive values were calculated for each combination of clinical score and outcome parameter with a signi cant AUC. Cut-off points were selected based on a Youden index with maximum sensitivity and speci city, closest to the upper-left corner of the ROC curve. The clinical scores below these cut-off points were considered low.
To identify parameters associated with clinical improvement, the aforementioned secondary outcome measures were analyzed using multivariate logistic regression analysis. Since this is the rst study analyzing parameters associated with safe, early discharge in this population, a Forward:LR method was selected. Strati ed K-fold cross-validation (k = 6) was utilized for model selection and internal validation. [52] For each patient the predicted probability of clinical improvement was calculated. The goodness-oft of the created model was analyzed using a ROC curve.
MedCalc Version 19.1.3 was used to compare AUC's. All other statistical analyses were performed using IBM SPSS Statistics for Windows, version 23.0. A P-value of < 0.05 was considered signi cant; all tests were two tailed.

Patient characteristics
During the study period 1381 patients were included. Of these patients, 423 were discharged within 24 hours of ED presentation. However, within 7 days, respectively 28 and 41 patients died or revisited our hospital due to the same diagnosis, resulting in 69 (5%) 'unsafe' discharges. Hence, 354 patients (25.6%) met the safe, early discharge criteria. (Fig. 1). First, we compared patients who needed prolonged hospitalization to patients who were early discharged. Patients who were early discharged were younger (P < .001), lived in their own home (P .005), were not referred by their General Practitioner (GP) (P .041) and were not transported by ambulance to the ED (P < .001). Moreover, less urgent triage categories, a history absent of cardiac disease, diabetes, COPD, chronic kidney disease and organ transplant, absence of sepsis, a low CIS, NEWS, SIRS, (q)SOFA, PIRO and MEDS score and the absence of respiratory and kidney failure at admission were signi cantly associated with safe, early discharge. Interestingly, a history of a malignancy or chronic liver disease, the presence of liver failure at admission and an abnormal lactate did not discriminate between our two groups. Malignancy was the most represented comorbidity (34.5%), with similar prevalence in both groups (P .708). The median length of hospital-stay of our entire group was 4 days, that of our safe, early discharge group 3 hours (P < .001). (Table 1) Clinical scoring systems Next, we assessed the accuracy of the aforementioned clinical scoring systems in predicting safe, early discharge. All scores were signi cantly associated with safe, early discharge ( Table 2). The discriminative performance of the attending physician's CIS was highest (AUC 0.739), that of the qSOFA and SIRS criteria lowest (AUC 0.576 and 0.593 respectively). There was no difference in performance between the NEWS, SOFA, PIRO and MEDS scores (AUC's ranging from 0.631 to 0.651). For all clinical scoring systems, optimal cut-off points for safe, early discharge were selected based on the Youden index of their respective sensitivity and speci city (Supplementary Table 2). The CIS, NEWS, SIRS, (q)SOFA, PIRO and MEDS score all showed a good negative predictive value (ranging from 78 to 85 percent) with a relatively low positive predictive value (ranging from 30 to 45 percent). (Table 2). Thus, a low score in and of itself, de ned as a score below the cut-off point, was not a great predictor of safe, early discharge. Vital sign and clinical score progression Third, we appraised the value of vital sign and score progression during ED stay. Patients with a normal systolic, diastolic and mean arterial blood pressure, respiratory rate, oxygen saturation, body temperature or mental status at ED admission and ED discharge, less frequently needed prolonged hospitalization (Pvalues < 0.05). As opposed to staying normal of vital signs, normalization of most vital signs did not discriminate between our two groups. Normalization of mental status was the only trend in vital sign normalization that occurred more frequently in the safe, early discharge group (P 0.033). Staying normal of the qSOFA and NEWS (P < 0.001 and 0.002 respectively), and normalization of the NEWS (P < 0.001) was signi cantly associated with safe, early discharge. (Table 3)

Prediction of safe, early discharge
To determine the effects of the aforementioned parameters on the likelihood of safe, early discharge, we performed a multivariate logistic regression analysis. Instead of including the sepsis related scores, their individual components were used. 55 parameters were selected for the analysis. The presence of sepsis-3 at ED discharge resulted in complete separation, meaning a patient in that category was always hospitalized. These patients were given a safe, early discharge probability of 0 and were excluded from the analysis. Using the Forward:LR method with strati ed k-fold cross-validation, 9 out of the 54 parameters were included in the nal model (Table 4 and supplemental table 3). Younger patients, without a history of an organ transplant, who did not arrive by ambulance while not being referred by their GP, with a low CIS, normal kidney and liver function, negative qSOFA, normal body temperature and without sepsis were more likely to be safely discharged within 24 hours. The logistic regression model was statistically signi cant (P < .001), correctly predicted 75.5% of the cases and explained 24.7% of the variance in safe, early discharge. (Table 4) ROC curve analysis yielded an AUC of 0.824, signi cantly better in predicting safe, early discharge compared to the aforementioned clinical scoring systems (P < .002 for all analyses). ( Scores predicting safe, early discharge are scarce To the best of our knowledge, this is the rst study that analyzes predictors of safe, early discharge among patients presenting to the ED with an infection. Several studies have focused on predicting early discharge among speci c patient populations [35,36,[53][54][55][56][57][58] and multiple have proposed prediction models. [36,[55][56][57][58] Only two, however, based their prediction model on ED patients. [57,58] The rst included 894 general ED patients and showed an AUC of 0.84 for the prediction of discharge within 48 hours. [57] The second included 297 general ED patients and demonstrated an AUC of 0.68 for the prediction of discharge within 72 hours. [58] Neither, however, analyzed the safety of these early discharges. One large prospective observational study did analyze safe, early discharges using a score called Halm's criteria. [35] This study, however, is based on patients admitted to the ward with pneumonia and the score they used has not been validated in the ED. Multiple studies analyzed ED discharge safety, using short-term outcomes, like readmission or death, following ED discharge. [59][60][61][62] Two of these studies proposed a model for the prediction of safe ED discharge, with AUCs ranging from 0.68 to 0.83. [60,61] Both the models predicting early discharge and the models predicting safe ED discharge include patient characteristics, like age and the presence of comorbidities, and most include arrival by ambulance, GP referral and vital signs measured closest to discharge. [36,57,58,60,61] However, only one incorporated repeated vital sign measurements [35], and none included biomarkers for organ dysfunction, components of various clinical scoring systems or the clinical impression score. The studies that did analyze the safety of ED discharges used endpoints that differ from study to study, with some utilizing ED readmission [61,62] and others ward, ICU admission or death [59,60]. In summary, available studies either analyzed predictors for early discharge or predictors for safe discharge. They utilized varying de nitions for unsafe discharge and limited predictor variables. Our study created a prediction model for safe, early discharge among patients presenting to the ED with an infection and included a wide variety of readily available predictor variables.

Repeated vital sign measurements should be used when evaluating ED patients
We found that normal vital signs at ED admission in combination with normal vital signs at ED discharge is associated with safe, early discharge. However, normalization of vital signs generally did not predict safe, early discharge. Multiple studies have analyzed the value of repeated vital sign measurements on patient outcome. Available studies either analyzed individual vital sign measurements [63] or combined these measurements in scores like the qSOFA or MEWS [11,35,[64][65][66]. Three studies compared single with repeated vital sign or score measurements and demonstrated that repeated measurements are superior in predicting clinical course of infectious or septic patients in the ED. [11,64,65] Several studies conclude that early changes in vital signs or scores are associated with patient outcomes like mortality or ICU admission. [11,35,[63][64][65][66] These studies show that, compared to patients with deteriorating vital signs, patients with vital sign normalization had a lower risk of mortality. Only one, however, analyzed repeated determination of vital signs in order to predict safe, early discharge. [35] This prospective observational study used Halm's criteria, combining normalization and staying normal of vital signs, and showed an AUC of 0.95 for 30-day mortality. Our study shows that patients with normal vital signs upon ED admission and discharge were more likely to be safely discharged compared to those who were admitted with abnormal vital signs which normalized during ED stay. Therefore, patients whose vital signs stayed normal during their ED visit have a better prognosis compared to patients with abnormal vital signs at either ED admission or discharge. Repeated vital sign measurements should therefore be included for risk assessment of ED patients for safe and early discharge.

Clinical impression scores have a wide array of use in the ED
Our study shows that both the clinical impression score of the attending physician and the nurse are strong independent predictors of safe, early discharge. This relationship may not be completely independent, since the attending physician eventually decides if the patient is discharged or admitted. This dependency could have introduced a selection bias, causing overestimation of the performance of the CIS. This bias, however, may be limited due to the fact that the CIS measurement is performed directly after primary assessment of the patient (generally within 30 minutes of the patient arriving), while the median length of stay of our safe, early discharge group was 3 hours. Moreover, the CIS of the nurse, which was independently measured, shows good predictive value as well. And, if we take a look at the arrival mode, since not arriving by ambulance while not being referred by a GP was the strongest independent predictor of safe, early discharge, we speculate that the clinical impression of the GP has also found its way into our prediction model. Several studies have demonstrated the value of the CIS in predicting 28-day mortality, hospital survival and patient disposition in ED and ICU patients. [17,20,67,68] When compared to the PIRO score, clinical impression scores show similar prognostic performance. [20] Clinical impression scores, as opposed to the PIRO, are easily determined, show good prognostic performance and could therefore be of great value for a wide variety of uses in the ED. Further research in our department will focus on identifying the various aspects of the CIS and its possible causes of bias.

Strengths and limitations
Our study is the rst study to analyze predictors of, and create a model for, safe, early discharge in patients presenting to the ED with an infection. This model was created using a 'simple' logistic regression analysis and internally validated with strati ed k-fold cross-validation. We did not only utilize patient demographics and comorbidities, but also included biochemical organ function, components of individual scoring systems, vital sign and score progression. The parameters used to evaluate biochemical organ function are routinely determined in almost every patient presenting to the ED with an infection. Vital sign measurements are part of the regular ED check-up and thus do not cause an extra burden for the patient. The created score can therefore effortlessly be determined for every patient presenting to the ED with an infection.
Our study has several limitations: rst, the generalizability of our study may be limited due to the fact that our study was single center and carried out in an academic tertiary care teaching hospital. Second, given the fact that the CIS, our strongest predictor, is not measured in other hospitals in the Netherlands, the created prediction model was not validated in external patient cohorts. We did, however, validate our model with strati ed k-fold cross-validation, by many viewed as the best method for internal model validation. [52] Third, to analyze the safety of the discharges, we arbitrarily chose a 7-day interval in which readmissions or death could occur. Because of this, and the fact that our patient records do not account for readmissions to other hospitals, the number of patients that were safely discharged may be overestimated. However, most of the studies that analyze the safety of ED discharges use this interval.
[ [59][60][61][62] Also, in the Netherlands, if a patient is known in a tertiary care center, like ours, readmission to another hospital is very unlikely due to the complexity of their condition.
It should be noted that our chosen cut-off only reached a PPV of 73.4%, resulting in 31 false positives.
However, as shown in Supplementary table 6, prolonged hospitalization accounted for 83.5% of the false positives and none of these patients died within 7 days of hospital discharge. Moreover, half of the patients with a false positive score were admitted on either a Thursday or a Friday. Given the fact that patients are less frequently discharged on weekend days [69], patients admitted on Thursdays and Fridays are more likely to be hospitalized for a prolonged timeframe. We therefore speculate that these false positives are in fact patients which would bene t from an early discharge.

Clinical relevance
By choosing a cut-off point with near maximum speci city, the number of false positives were minimized and the number of unsafe discharges resulting in death zero. Our created model is therefore ideal to identify low risk patients likely to bene t from safe, early discharge. In our study population, this resulted in 84 true positives, i.e. patients that could be safely discharged within 24 hours without disease related death or readmission within 7 days. (Supplementary table 5) Meaning that our model can be effectively used as a clinical decision tool to identify a relatively small number of patients that can bene t from safe, early discharge. With the use of our model, the effects of ED overcrowding can be minimized and the costs and adverse events associated with a general ward admission could be prevented. The choice of a cut-off for prospective research should be considered with care, weighing the chance and impact of a false positive score against the bene ts of early discharge. Our score, like every scoring system used in medicine, should be used as a tool to guide the treating physician. The created probability of safe, early discharge may augment the physicians' decision but should not replace the physicians' evaluation of the individual patient.

Conclusion
Our study shows that patient demographics and information about the arrival, rst impression, biochemical organ function and vital sign progression of a patient presenting to the ED with an infection is associated with safe, early discharge. By combining several readily available parameters in the ED setting, a prediction model for safe, early discharge with good prognostic performance was created.
Further research, preferably in a multi-center setting, is warranted to externally validate the created model and determine whether the use of such a discharge prediction tool can reduce the number of unnecessary hospitalizations. Availability of data and material

List Of Abbreviations
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.

Funding
This study is funded by the emergency department of the University Medical Center Groningen.
Authors' contributions MCFM participated in the study design, assisted with data acquisition, carried out data analysis and drafted the manuscript. JCtM participated in the study design, assisted with data interpretation and critically revised the manuscript. LB assisted with data acquisition and data analysis. TJO assisted with data acquisition and critically revised the manuscript. HRB drafted the study design, assisted with data interpretation, critically revised the manuscript and has given nal approval of the version to be published. and April 2019 were screened for inclusion. Safe, early discharge, hospital-discharge within 24 hours, without disease-related readmission to our hospital or death during the rst 7 days.