Based on 1,023,613 visits, we built an MLS protocol with a satisfactory AUC to test for the odds of a patient with an initial ETS Level 3 or 4 encountering a ‘life-threatening mis-triage’ situation and requiring up-triage into the resuscitation room during their ED stay. At the same time, we used interpretability as an essential practical reference-point for triage officers, so that medical staff may intuitively understand the weight of each triage feature in the conclusions given by the MLS. By being more ‘understandable’ than a simple ‘black box’ system, we hoped this MLS would better aid in making triage decisions. Used in a prospective dataset, we found a significant decrease in the ‘life-threatening mis-triage’ rate, demonstrating its effectiveness and usefulness.
Current clinical triage systems, including the ESI in the United States and the ETS in China, have significant limitations. They rely to some degree on the subjective judgment of triage staff, which has led to significant variability in triage results[12], especially in pediatric patients[13]. About half of the ED patients in the United States are triaged as ESI Level 3[14], and many of these patients have conditions that potentially are quite emergent[15]. The mortality rate of ESI level 3 patients in most triage systems is not low, even reaching nearly half of that of level 2 patients[5]. In China, most patients are triaged into ETS Level 3 or 4[4]. In our retrospective study, the rate of ‘life-threatening mis-triage’ in ED patients with initial ETS Level 3 or 4 was 1.12%. Reducing this rate may save lives. For a typical ED with 100,000 patient visits a year, reducing the mis-triage rate to 0.9% with an MLS as we were able to do in this study, would translate into hundreds of fewer mis-triage cases.
Machine learning can modify its response patterns by creating data systems, developing algorithms, and applying them to inference outcomes from new data, which is most applicable to a number of complex, nonlinear relationships. Machine learning models have been applied to predict outcomes in different medical fields, such as predicting hypoxemia during surgery[9], predicting mortality in patients with sepsis[16], acute cardiac complications in patients with acute chest pain[17], and the likelihood of severity and hospitalization in children[18] and adults[19] with asthma or COPD exacerbations, respectively. Previous studies have also looked at adult ED triage to predict patients' risk of critical illness or hospitalization[20, 21], but these studies did not provide any explanatory interpretation on the clinical side, making it difficult for MLS results to be accepted by medical staff and integrated into clinical practice. Although relatively straightforward, providing the MLS reasoning allowed us to prospectively validate this MLS protocol and directly test its utility on ‘real-life’ triage cases. Compared to previous studies, our approach provides a detailed indication of a patient’s risk of requiring critical care. In our study’s MLS protocol, each patient is given an individualized odds ratio with the specific weight of each triage feature used in artificial intelligence decision making listed and displayed. Such a descriptive read-out provides a triage nurse with additional material to confidently call a supervising ED physician to assess a patient's condition. In our prospective results, it increased physician consults to tirage by almost 25% and reduced the ‘life-threatening mis-triage’ rate to less than one percent. Traditionally, because of the high workload of ED nurses and physicians, unless a patient's condition differed significantly from the electronic grading system, triage nurses were reluctant to call a physician for consultation. Our MLS can give triage nurses more confidence to make such a call.
Machine learning has some unique advantages in triage. Factors can now be included that were traditionally left out of formal triage protocols. For example, a patient’s time of arrival are important[22], and a patient’s mode of arrival is also anecdotally acknowledged as being important by experienced triage nurses: patients that arrive on ambulances or arrive at night tend to be sicker. In fact, in our MLS results, the most important triage feature was arrival mode, which is similar to other machine learning triage studies[20]. Age is also an essential parameter in most critical care scoring systems, as older patients often have more underlying diseases and poorer prognoses than younger patients[23]. However, age cutoffs in traditional triage systems tend to be too blunt. For example, the risk profile of a 64-year-old compared to a 65-year-old is quite similar, despite a cutoff at 65[24]. An MLS protocol that uses age as a continuous variable rather than simple cutoffs may provide better predictive power for ‘edge’ cases. Similarly, in the traditional triage system, it is often exhausting to determine the normal value and critical value of heart rate for pediatric patients of different ages. On the other hand, machine learning triage systems can computationally combine a patient's age with heart rate and other indicators to quickly give an appropriate weighting. Likewise, sex is not used as a high-risk factor in traditional triage, but there are differences between males and females in the risk-profiles of many diseases, and a machine learning method can consider a patient's sex in formulating a risk assessment. In our study, sex was one of the top three most important triage characteristics.
We included two additional indicators in our study: pulse pressure and shock index, which are two circulation indicators that have been found to be highly correlated with acute illness severity in previous studies[25–28] but have not been widely used in emergency triage to date. Our results found these two indicators to be beneficial as emergency triage characteristics, even more important than systolic blood pressure. Interestingly, we found pulse pressure has a lower sensitivity in older patients, which has not been reported before, perhaps because older patients have decreased vascular compliance, making their pulse pressure higher, thus lowering its sensitivity. The situation with shock index was just the opposite, as it has a higher sensitivity in older patients. This may be due to older patients' relatively poor cardiac compensation ability, requiring a higher heart rate to compensate for any decrease in blood pressure.
We should note that some important traditional triage factors were not included in our MLS system, specifically respiratory rate, which was not included due to many missing data values in the dataset, suggesting poor nurse compliance in obtaining this feature. Respiratory rate measurement has been criticized as requiring time and skills for an accurate measurement[29]. Missing data values for temperature were also large, possibly because most patients had a normal temperature or did not require a routine temperature check, and possibly because more ‘traditional’ temperature measurements are more invasive and time-consuming for triage staff. Chief complaint is clearly important but was also not included because of the large variability in complaints and many missing values. Chief complaints are unfortunately quite subjective, and future MLS acquisition and standardization techniques using very large datasets will likely be needed to better include this as a viable triage factor in the future. All these missing data values suggest that the EETS system may need to change, and some threshold indicators (e.g., shortness of breath, ischemic chest pain, etc.) can possibly be used instead. Interestingly, the non-inclusion of these indicators did not affect the validity and usefulness of this study’s results. On the contrary, the results of this study are more generalizable because we only used easily obtainable, objective indicators.