Herein, we showed that ML in combination with clinical data could improve the effectiveness and appropriateness of empirical antibiotic therapy in the ED. ML is a growing field in medicine, including in infectious disease (ID). By July 2019, there have been 60 ML-clinical decision support system (ML-CDSS) that have been developed to assist ID clinicians.14 Empiric antimicrobial therapy could benefit the most from ML-CDSS because of the unavailability of culture results at the time of prescription. Moreover, it is unlikely for emergency physicians to receive any feedback regarding initial selection of antibiotics because patients do not wait in the ED until the result is available. Therefore, ML-CDSS could be promising in assisting decision-making by presenting assessments and recommendations individualised to the patient.
To ensure that these ML tools are useful and successfully integrated into clinical practice, a clear understanding of medical perspectives through the decision-making process is fundamental. For instance, ID clinicians begin to assess a febrile patient with the primary objective of finding the focus of the fever. The possible pathogens and antibiotics with the proper coverage are only determined after the clinical diagnosis is made. As such, focusing only on individual antibiotics or pathogens using ML algorithms without a provisional diagnosis would be of no use to support clinical decisions. Thus is why we clearly defined the clinical scenario as upper UTI before predicting antibiotic resistance in order for the model to be successfully integrated into clinical practice, which sets our study apart from previous studies.
Under the premise of clinical diagnosis, ML-CDSS is constrained by comprehensiveness and quality of the clinical data used for their development. In this context, we analysed all available data from different sources, such as a structural clinical data, vital signs, and laboratory data. However, we also included relevant unstructured data such as free clinical texts, nursing notes, and medical imaging to ensure integration of detailed medical history. Studies that analysed the performance of ML-CDSS when using a reduced set of variables found that the sensitivity and specificity of ML-CDSS were systematically better when they used a larger set of variables, especially when unstructured data are added.15–17 We also distinguished cases of bacterial colonisation or contamination in urine cultures from true pathogens and excluded them from the study to improve the quality of the data. These efforts to maximize the comprehensiveness and quality of our data contributed to the development of a satisfactory model in our study. An additional important strength of our work is that algorithm training and evaluation were performed on different data sets.
However, this study also has a number of limitations. Firstly, our model was built on data from a single healthcare institution within a confined geographic region; thus, further validation at other institutions is needed. As resistance patterns can change over time,8, 18 our model may also become less relevant as time passes and should thus also be periodically retrained Therefore, identifying the most appropriate temporal and spatial selection windows for training data would be essential for future research. Secondly, we assessed the performance of the model against real-life prescribers, which were emergency physicians. Ultimately, performance would need to be assessed prospectively for validation purpose with well-defined endpoints such as hospital days, mortality, medical costs, and impact on bacterial resistance in the long term.
Thirdly, our result shows that detailed information from unstructured data ranked high as feature importance on prediction. These variables with high feature importance were found to be consistent previously identified risk factors.19 However, given the labour-intensive and time-consuming process of data identification, data should be easily entered into the CDSS in the future system by automatic extraction from the EMR, and progress in natural language processing may help.20
Lastly, we did not fix the decision threshold of the GBDT model in our result considering each clinical setting might require different emphasis on using the model. For instance, in the setting of emergency, prompt clinical improvement or survival benefit maximizing effectiveness would be the main concern. Meanwhile, a reduction in the inappropriate use of broad-spectrum antibiotics could be a priority in a community with a high rate of antibiotic overuse because improving antibiotic stewardship may lead to reduced costs, complications, and improved clinical outcomes.21 The best approach should make it possible to substantially increase the proportion of patients who receive effective empiric antibiotics while minimizing the risk of developing resistance in a given circumstance, and our model has the possibility of being modified to individualise this risk-benefit ratio. Prospective studies are required to assess the performance of our ML model for achieving better patient outcomes and minimizing the risk of antibiotic resistance.