In this study, we identified significant factors associated with overcrowding and we built and validated predictive models for overcrowding days. The primary objective was achieved by describing the variables associated with Sustained Critical EDOC and by obtaining a predictive model with 7 predictive variables: Average amount of black NEDOCS hours in the prior day; percentage of black and red NEDOCS hours in the prior day; average prior day NEDOCS score; percentage of admitted patients who are in the 4th quartile of length of stay in days; relative ambient humidity; cold day and summer + winter holidays. Prediction was internally validated in a random set of internal validation days, using bootstrapping, with groups of temporal external validation days, and in another healthcare center within the hospital’s HMO.
In the bivariate analysis, indicators of increase in consults were positively associated with Sustained Critical EDOC as a percentage of Flu-like disease, blue triage category or cold days. Crowding indicators were associated with Sustained Critical EDOC in different measurements of the NEDOCS score from the previous day, hospital length of stay, percentage of occupation of closed areas and scheduled surgeries. The holidays protect from overcrowding just like non-working days and the years of experience of the residents working in the ED. This way, the baseline bivariate analysis is coherent with what is expected in all aspects that were evaluated. It is fundamental to remark also that the use of predictive models in this paper does not seek to explain the outcome, but to predict it. We believe that, considering both previous day measures in logistic and autoregressive models, the most valuable information lies in the NEDOCS behavior of the previous day.
The evaluation of calibration from the selected model was similar both in the generation as in the validation group. Both observed and predicted probabilities are equal to zero in all percentiles ranging from 1 to 8 in both groups. In percentile 9, the predicted and observed probabilities are very similar in both groups. In percentile 10, we observed 100% predicted probability and coinciding with observed probability in both groups, meaning the model showed near-perfect calibration. Also, the model predicts zero probability in the majority, one transition percentile and one last percentile of a 100% probability, a behavior similar to a dichotomous response variable. Discrimination for the selected model was optimal since the ROC curve has an estimated area under the curve of 99.7% both in the generation and validation groups, and in the discrimination adjusted through bootstrapping. In order to verify the consistency of these AUC values, we compared multiple cutpoints for the model’s output to simplify the results. With a cutoff point of 0.5 for the predictive model’s output, positive predictive values (94%) and negative predictive values (98%) are very high.
Since a Sustained Critical EDOC event starting is relevant for resource management, we considered important to evaluate variables predicting the occurrence of a day in which a Sustained Critical EDOC event that lasts for a number of days will commence, or in other words, to predict the start of Sustained Critical EDOC (or Sustained Critical EDOC start). Variables positively associated with this were the average NEDOCS score of the previous day and the percentage of hours in the black NEDOCS category from the previous day, while a negative association for this event was seen with the percentage of black + red triage category hours of the prior day NEDOCS score. In the same way, knowing when an overcrowded ED department will stop being overcrowded is also relevant to deactivate the alarm status in the hospital and return resources to a baseline status, ready for any potential event. The percentage of black and red NEDOCS hours in the previous day was positively associated with the end of Sustained Critical EDOC (Sustained Critical EDOC end). There was a negative association with previous day black NEDOCS hours, previous day general ward percentage of occupancy and days falling on a Saturday.
Even though the phenomenon reflected by these 3 models is the same, they do not represent the exact same concept, and were generated with different subsets of days and different number of observations. The Sustained Critical EDOC predictive model speaks of variables predicting initiation plus sustainment of OC. The Sustained Critical EDOC start model only predicts the day in which a consecutive number of days with OC will begin. Lastly, in the Sustained Critical EDOC end model, days without OC in them following a sequence of days with an OC event allow to establish the occurrence of the day in which this OC sequitur ends, with lesser power (the generation group for this model was done with 87 days) but using selected variables that are complementary with those from other models. The three models represent different aspects of EDOC that may be useful in different moments. The use of these 3 models simultaneously provides more adequate information to define appropriate actions.
Although autocorrelation ordered in time might generate some doubts regarding the use of logistic regression models, they’re very common in the literature such as in Hoot & Aronsky’s paper [52]. Without going any further, the original NEDOCS model ignores the time-dependent structure of errors [15]. The time correlation of errors is a component of total variability, therefore, ignoring correlation is a conservative scenario in the sense that standard errors estimated by our model are larger than they would be if we considered the correlation of errors through time.
Other predictive approaches previously published use many different outcomes that may not represent the complex relationship between different aspects of overcrowding: critical OC event defined as a period in which incoming ambulances need to be referred to different hospitals for lack of resources [53], waiting times in the next 2 hours in the ED [54], “peak day” or “non-peak day” of patients with chronic respiratory illnesses ED admissions [55], short-term ED bed occupancy [56, 57], number of patients in the ED exceeding the standard length of waiting time of 4 hours in the ED [58], the average daily NEDOCS and the percentage of LWOS patients [59], which has also been shown to positively correlate with the increase of the NEDOCS score [60]. Additionally, there is a large variability in predictors. Most predictive models and methods are highly variable, including logistic regression [61], linear regression [62], ARIMA and seasonal ARIMA (SARIMA) [57] and sinusoidal with an autoregressive error structure [56], artificial discrete event simulation models [59], recurrent neural networks [63], Long Short-Term Memory (LSTM) recurrent neural network [54], neural network for time series [58], among other machine learning approaches [55]. Some model generation and validation methods are incompletely described. All these characteristics impair comparison and fair evaluation of external validity.
Even though there are different alternatives to represent OC at a day level (e.g. average NEDOCS [59]), we decided to use a Sustained Critical EDOC event of 8 hour where every hour had a NEDOCS score in the black category (≥ 180 points). Even though it is arbitrary, it was elaborated taking into account the fact that a standard nursing shift lasts for that same amount of time and the potential easy interpretability. This adds simplicity for resource managing in our specific setting but it might differ in other places. Evaluation of NEDOCS score validity in different contexts may be variable [16, 23–25, 30, 32, 64–71], possibly indicating the need of local evaluation to represent OC in similar EDs. Day-level aggregation may be a possible simple solution for this variability. Different definitions of EDOC events might provide more insight on the degree to which it represents the OC phenomenon [18, 72–81]. Additionally, to add consistency to our purpose, we obtained evidence of the face validity of this outcome to represent an amount of OC that requires immediate action, in a convenient sample of ED experts from different contexts.
A possible issue we identified with NEDOCS is that it is susceptible to data input issues such as: failure to label an admission process as complete, patients self-discharging, and empty discharge dates, among others. Although these errors were infrequent, they can have a very large impact on the NEDOCS score in settings where these issues are prevalent.
One of the main strengths in our study was the possibility of measuring hourly NEDOCS scores through the entire study period. The informatical system implemented in the ED allowed for this measurement to be done and it might even be possible to implement it in real time. Former studies carried out by our research team have shown specific patterns in NEDOCS’s variability in different hours of the day and in different days of the week as seen in graph 1 presented in the results section [32, 34].
The 2-year input used for model generation is a lot longer than most studies published to the date. For example in the original paper where NEDOCS was proposed by Weiss & col. a 1 month time period was used [15]; in Anneveld & col. 4 weeks were used [82]; Bair & col. considered only 5 months [59]; Hoot & col. 6 months [52]; 1 year for Schweigler & col. [56] and Wang & col. [58]; 2 years in Peng & col. [55]; Abraham & col. almost 3 years [57]; and finally, Tideman & col. with a 7-year period [83].
Regarding study limitations, we did not reach the a-priori estimated sample size of 70 outcomes for 7 predictors included in the model. Nevertheless, all the included predictors were significant and demonstrated adequate performance. This may indicate appropriate power for the development of a parsimonious primary predictive model for Sustained Critical EDOC. The smaller sample size for the 2 secondary predictive models for Sustained Critical EDOC start and end, may indicate instability possibly affecting the development and validation of both models.
The inclusion of a time period larger than 1 year eliminates the possibility of seasonality which might compromise the external validity of the results. The inclusion of time periods that span across all 4 seasons during 1 year includes days representing all moments of the year. We selected this time period due to the absence of structural modifications in the architectural structure of the hospital’s emergency department. This is fundamental for the use of predictive models as changes at this level might affect almost every component of NEDOCS.
Overcrowding in the emergency setting remains a multi-factorial problem and the process of characterizing it will require multiple perspectives and approaches. The use of real-time predictive strategies to monitor the working status of the ED and signal a situation where the status quo is marching towards an impending overload is essential for a holistic hospital management and has the potential to be used to make quick decisions to prevent potential disasters. All variables considered for predicting overcrowding should be readily obtained, such as those considered in our paper, to construct a model that is viable and can be implemented in settings of different complexities and resources.
We believe the findings from our paper, combined with those of other authors, provide valuable information regarding a wide list of factors influencing ED overcrowding. Further research should be carried out to identify those that are specific to settings and/or regions.