Predicting Sepsis Onset in ICU Using Machine Learning Models: A Systematic Review and Meta-Analysis

doi:10.21203/rs.3.rs-2961713/v1

Download PDF

Research Article

Predicting Sepsis Onset in ICU Using Machine Learning Models: A Systematic Review and Meta-Analysis

https://doi.org/10.21203/rs.3.rs-2961713/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 27 Sep, 2023

Read the published version in BMC Infectious Diseases →

You are reading this latest preprint version

Background: Sepsis is a life-threatening condition caused by the body's abnormal response to infection, which imposes a significant health and economic burden worldwide due to its high mortality rate. Early recognition of sepsis is crucial for effective treatment. This study aims to systematically evaluate the predictive performance of various machine learning models in predicting the onset of sepsis.

Methods: We conducted a comprehensive search of the Cochrane Library, PubMed, Embase, and Web of Science databases, covering studies from the inception of the databases up to November 14, 2022. We used the PROBAST tool to assess the risk of bias. We calculated the predictive performance of sepsis onset using C-index and accuracy. We followed the PRISMA guidelines for this study.

Results: We included 23 eligible studies with a total of 4,314,145 patients and 26 different machine learning models. The most frequently used models in the studies were Random Forest (n=9), Extreme Gradient Boost (n=7), and Logistic Regression (n=6). Random Forest (test set n=9, acc=0.911) and Extreme Gradient Boost (test set n=7, acc=0.957) were the most accurate models based on our analysis of the predictive performance. In terms of the C-index outcome, Random Forest (n=6, acc=0.79) and Extreme Gradient Boost (n=7, acc=0.83) showed the highest performance.

Conclusion: Machine learning has proven to be an effective tool for predicting sepsis at an early stage. However, to obtain more accurate results, additional machine learning methods are required. In our research, we discovered that XGBoost and Random Forest exhibited the best predictive performance and were most frequently utilized for predicting the onset of sepsis.

Systematic review registration:CRD42022384015

Machine learning

Sepsis

Intensive Care Units

Meta-analysis

Sepsis is a severe and potentially life-threatening condition resulting from a dysregulated immune response to infection^[1]. Early detection and prompt treatment are crucial for improving patient outcomes and reducing healthcare costs. In recent years, machine learning (ML) models have emerged as a promising tool for detecting and managing sepsis in the Intensive Care Unit (ICU)^[2]. These models use complex algorithms and statistical methods to learn from large volumes of patient data, including vital signs, laboratory results, and electronic health records, and to predict the onset of sepsis before its clinical manifestations become apparent^[3]. By enabling early detection, ML models hold tremendous potential for enhancing patient care and reducing the burden of sepsis on healthcare systems worldwide.

In this article, we aim to explore the use of ML models for predicting the onset of sepsis in the ICU. Specifically, we will review the existing literature on ML models for sepsis prediction, highlighting their strengths and limitations. Additionally, we will discuss the potential impact of these models on patient outcomes, clinical decision-making, and healthcare costs. Through this meta analysis, we hope to shed light on the promise of ML models as a tool for improving the management of sepsis in the ICU and beyond.

Cochrane Library, Embase, PubMed and Web of science were searched from inception to 14/11/2022. Search formulas were constructed based on the combination of Mesh headings and free words. We did not put any restriction on the language and region. The literature search was finished by Zhenyu Yang and Xiaoju Cui. (The search detail is shown in Supplementary File 2) All selected studies were imported to EndNote 2020. We filter studies according to the abstract. Duplicated articles were deleted. Literature screening were accomplished by two reviewers independently(Zhenyu Yang and Xiaoju Cui). Any disagreement was settled by third reviewer. The retrieval formular file was presented in supplementary material 2.

Inclusion and exclusion criteria

Inclusion criteria

Randomized controlled trial (RCT), prospective cohort study, nested case-control study.

Predicting model was completely established.

Exclusion criteria

Unrelated to sepsis

Incomplete data

Outcome measures related to the effectiveness of predicting measures depleted.

Data extraction

The data extraction form was detailed according to the Modified CHARMS checklist. The checklist included name of the first author, publication date, nationality, duration of data hiring, study design, type of validation (internal, external, random split and time split) and sample size (total number, development and testing cluster)

Risk of bias assessment

We used PROBAST and external prognostic validity model to assess the risk of bias of the selected studies^[4]. The PROBAST is a checklist designed for systemic reviews of diagnostic or prognostic prediction models. The risk of bias was performed by two reviewers independently.

Statistical Analysis

We performed descriptive statistics to summarize the characteristics of the models. For prediction models that were evaluated in more than two independent datasets, a random effect meta-analysis was conducted to estimate their performance and accuracy. If a measure of uncertainty such as standard error or 95% confidence interval was unavailable for mean C-index, we computed it based on the number of events and participants. All data analyses were carried out using R software version 4.1.1.

Study selection

A total of 422 articles were identified through various databases including Cochrane (n = 12), Embase (n = 150), Pubmed (n = 74), and Web of Science (n = 186). After eliminating 15 duplicate articles and excluding ineligible records using automation tools, we browsed through 387 articles. Ultimately, 23 articles met the inclusion criteria and were included in our study^{[2, 5–26]}. Figure 1 displays the PRISMA flow diagram illustrating our study selection process. The selection was conducted independently by two reviewers (Zhenyu Yang and Xiaoju Cui). Any discrepancies were resolved by a third reviewer.

Characteristics of included studies

A total of 1,287,160 individuals were included in this study, with 167,338 in the validation set. All articles analyzed were published within the past 5 years, indicating a growing interest in using machine learning for sepsis prediction. Our research identified 81 prognostic models, including 5 based on Deep Learning, 4 on InSight, 10 on Logistic Regression, 6 on Multilayer Perceptron, 8 on Neural Network, 8 on Support Vector Machine, 14 on XGBoost, 15 on Random Forest, 11 on SOFA, and others. Detailed characteristics of the included studies can be found in Table 1 and Table 2.

Quality assessment

The quality assessment was conducted independently by two reviewers (Zhenyu Yang and Xiaoju Cui), and any discrepancies were resolved by a third reviewer. The results of the quality assessment are presented in the risk of bias picture (Fig. 2). Two studies (8.6%) were deemed to have a high risk of bias in the domain of participants, 13 studies (58.3%) were deemed to have a high risk of bias in the domain of analysis, and two studies (8.6%) were deemed to have a high risk of bias in the domain of outcome. No studies were deemed to have a high risk of bias in the domain of predictors. The high risk of bias in the analysis domain may be attributed to inadequate sample size, insufficient Events per Variable (EPV), improper handling of missing data, or failure to report how missing data were handled. The PRISMA checklist can be found in Supplementary File 1.

Predictors

Age, Creatinine, and Sodium were the most frequently used predictors (n = 12), followed by blood pressure and platelets (n = 11), and heart rate (n = 9). The remaining predictors were ranked in descending order of frequency as lactate and temperature (n = 9), WBC count (n = 8), respiratory rate and SOFA score (n = 7), glucose hemoglobin, MCHC, and PaO2 (n = 6), GCS, ICU LOS, Lymphocyte, and PaCO2 (n = 5), BUN, cancer, and gender (n = 4). These results are presented in Fig. 3.

Training set and test set accuracy

In the training set, the Random Forest model was the most frequently applied machine learning model (n = 9), with an accuracy of 0.911 (0.485, 0.991). XGBoost showed the best predicting performance (n = 6), with an accuracy of 0.970 (0.487, 0.997). In the test set, the Random Forest model was also the most frequently applied machine learning model (n = 7), with an accuracy of 0.795 (0.638, 0.895). The Deep Learning model showed the best predicting performance (n = 3), with an accuracy of 0.830 (0.814, 0.845). These results are presented in Figs. 4–8.

Training set and test set c-index

Regarding the c-index results, in the training set, XGBoost was the most frequently utilized machine learning model with a c-index of 0.83 (0.83,0.84) in 7 studies. InSight exhibited the best performance with a c-index of 0.91 (0.90,0.93) in 2 studies. On the other hand, in the test set, the random forest model was the most frequently employed machine learning model with a c-index of 0.83 (0.82,0.83) in 5 studies. In terms of performance, both the random forest model (n = 5, c-index = 0.83 (0.82,0.83)) and XGBoost (n = 3, c-index = 0.83 (0.82,0.84)) exhibited similar performance. Detailed datasets can be found in Figs. 9–13, and the overall results are presented in Supplementary File 3.

The present study investigated 68 prognostic predicting models across 23 studies to assess the potential of machine learning models for predicting sepsis in the ICU. However, the risk of bias assessment revealed a high risk of bias in the analysis, which may be attributed to the small sample size, processing of missing data, and interpretation of complex data. Therefore, the research findings may have some deviation due to the insufficient sample size.

Sepsis is a severe medical condition that can cause widespread inflammation and damage to vital organs. Early detection and treatment of sepsis are critical for improving patient outcomes and reducing healthcare costs. ML models can analyze large amounts of patient data, including vital signs, laboratory results, and electronic health records, to detect early signs of sepsis. ML algorithms can provide physicians with real-time recommendations for patient treatment and management based on the latest medical knowledge and the patient's data. The use of ML models for predicting the onset of sepsis in the ICU has the potential to revolutionize the way sepsis is detected, treated, and managed, leading to better patient outcomes and reduced healthcare costs.

Several studies have explored the potential of machine learning algorithms for predicting sepsis. Heather M et al^[27]. developed a machine learning algorithm to predict severe sepsis and septic shock, which can predict with high specificity the impending occurrence of severe sepsis and septic shock. Lucas M Fleuren et al. designed a meta-analysis that found that individual machine learning models can accurately predict sepsis onset ahead of time, similar to the present study. Nianzong Hou et al^[28]. developed an XGBoost model to predict 30-days mortality, which can assist clinicians in tailoring precise management and therapy for patients with sepsis-3. Dong Wang et al^[29]. developed an artificial intelligence algorithm to predict sepsis early, which has shown good predictive ability in Chinese sepsis patients. However, external validation studies are necessary to confirm the universality of the method in the population and treatment practice.

XGBoost and Random Forest are two machine learning algorithms that showed significant advantages compared to other models in the present study. XGBoost is a popular open-source software library for machine learning that is optimized for speed and scalability, making it one of the most efficient gradient boosting algorithms available. It can handle missing data and noisy data, making it a robust solution for real-world data problems. Random Forest is a widely used ensemble machine learning algorithm that combines multiple trees to form a forest and produces a final prediction by aggregating the results from all the trees. These algorithms have been applied to various industries, including finance, healthcare, and marketing, and have won several machine learning competitions^[30]. In our research, Random Forest and XGboost show significant advantage comparing to other models.

Machine learning has proven to be an effective tool for predicting sepsis at an early stage. However, to obtain more accurate results, additional machine learning methods are required. In our research, we discovered that XGBoost and Random Forest exhibited the best predictive performance and were most frequently utilized for predicting the onset of sepsis.

Ethics approval and consent to participate

This study does not involve ethical issues and therefore does not require approval from the ethics committee.

Consent for publication

This study is based on published research and does not contain data from any individual person, therefore the consent for publication is not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests

Funding

This study was conducted by the author at their own expense without any other funding support.

Author Contributions

Zhenyu Yang formulated the research purpose, searched, and screened the literature, processed the data and wrote the article. Xiaoju Cui participated in the literature collection and screening work. Zhe Song participated in data analysis work. Xiaoju Cui supervised the study. Zhenyu Yang and Zhe Song participated in text correction and format proofreading work.

Acknowledgements

Not applicable

SINGER M, DEUTSCHMAN C S, SEYMOUR C W, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) [J]. JAMA. 2016;315(8):801–10.
NEMATI S, HOLDER A, RAZMI F, et al. An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU [J]. Crit Care Med. 2018;46(4):547–53.
GREENER J G, KANDATHIL S M MOFFATL, et al. A guide to machine learning for biologists [J]. Nat Rev Mol Cell Biol. 2022;23(1):40–55.
WOLFF R F, MOONS K G M, RILEY R D, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies [J]. Ann Intern Med. 2019;170(1):51–8.
WANG Z. Multi-Branching Temporal Convolutional Network for Sepsis Prediction [J]. IEEE J Biomedical Health Inf. 2022;26(2):876–87.
SRIMEDHA B C, RAJ R N MAYYAV. A Comprehensive Machine Learning Based Pipeline for an Accurate Early Prediction of Sepsis in ICU [J]. Ieee Access. 2022;10:105120–32.
SINGH YV, SINGH P et al. KHAN S,. A Machine Learning Model for Early Prediction and Detection of Sepsis in Intensive Care Unit Patients [J]. Journal of Healthcare Engineering, 2022, 2022.
SHI S, PAN X, ZHANG L et al. An application based on bioinformatics and machine learning for risk prediction of sepsis at first clinical presentation using transcriptomic data [J]. Front Genet, 2022, 13.
PENG PENGL, YANG C. Machine learning approach for the prediction of 30-day mortality in patients with sepsis-associated encephalopathy [J]. BMC Med Res Methodol. 2022;22(1):183.
ZENG Z, YAO S, ZHENG J et al. Development and validation of a novel blending machine learning model for hospital mortality prediction in ICU patients with Sepsis [J]. BioData Min, 2021, 14(1).
WERNLY B, MAMANDIPOOR B, BALDIA P, et al. Machine learning predicts mortality in septic patients using only routinely available ABG variables: a multi-centre evaluation [J]. Int J Med Inform. 2021;145:104312.
WANG D, LI J, SUN Y, et al. A Machine Learning Model for Accurate Prediction of Sepsis in ICU Patients [J]. Front public health. 2021;9:754348.
XU SUL, CHANG Z. F, Early Prediction of Mortality, Severity, and Length of Stay in the Intensive Care Unit of Sepsis Patients Based on Sepsis 3.0 by Machine Learning Models [J]. Front Med, 2021, 8.
SHASHIKUMAR S P, JOSEF C S, SHARMA A, et al. DeepAISE - An interpretable and recurrent neural survival model for early prediction of sepsis [J]. Artif Intell Med. 2021;113:102036.
RODRIGUEZ A, MENDOZA D, ASCUNTAR J, et al. Supervised classification techniques for prediction of mortality in adult patients with sepsis [J]. Am J Emerg Med. 2021;45:392–7.
RAYAN Z, ALFONSE M, SALEM A B. M. Predicting Sepsis in the Intensive Care Unit (ICU) through Vital Signs using Support Vector Machine (SVM) [J]. Open Bioinf J. 2021;14(1):108–13.
RAFIEI A, REZAEE A, HAJATI F, et al. SSP: Early prediction of sepsis using fully connected LSTM-CNN model [J]. Comput Biol Med. 2021;128:104110.
PERSSON I, OSTLING A, ARLBRANDT M, et al. A Machine Learning Sepsis Prediction Algorithm for Intended Intensive Care Unit Use (NAVOY Sepsis): Proof-of-Concept Study [J]. JMIR formative research. 2021;5(9):e28000–e.
LU Z, ZHANG J, HONG J et al. Development of a Nomogram to Predict 28-Day Mortality of Patients With Sepsis-Induced Coagulopathy: an Analysis of the MIMIC-III Database [J]. Front Med, 2021, 8.
LIU W, TAO G, ZHANG Y et al. A Simple Weaning Model Based on Interpretable Machine Learning Algorithm for Patients With Sepsis: A Research of MIMIC-IV and eICU Databases [J]. Front Med, 2021, 8.
LIN, P-C, CHEN K-T, CHEN, H-C et al. Machine Learning Model to Identify Sepsis Patients in the Emergency Department: Algorithm Development and Validation [J]. J Personalized Med, 2021, 11(11).
YUAN K C, TSAI L W, LEE K H et al. The development an artificial intelligence algorithm for early sepsis diagnosis in the intensive care unit [J]. Int J Med Informatics, 2020, 141.
ROIMI M, NEUBERGER A, SHROT A, et al. Early diagnosis of bloodstream infections in the intensive care unit using machine-learning algorithms [J]. Intensive Care Med. 2020;46(3):454–62.
PARRECO JP, HIDALGO A E, BADILLA A D, et al. Predicting central line-associated bloodstream infections and mortality using supervised machine learning [J]. J Crit Care. 2018;45:156–62.
MAO Q, JAY M. HOFFMAN J L, Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU [J]. BMJ Open, 2018, 8(1).
LICHTER Y, STAVI D, KELER U et al. A novel model for early detection of patient deterioration in ICU [J]. Crit Care, 2018, 22.
GIANNINI H M, GINESTRA J C, CHIVERS C, et al. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice [J]. Crit Care Med. 2019;47(11):1485–92.
HOU N, LI M, HE L, et al. Predicting 30-days mortality for MIMIC-III patients with sepsis-3: a machine learning approach using XGboost [J]. J Transl Med. 2020;18(1):462.
WANG D, LI J, SUN Y, et al. A Machine Learning Model for Accurate Prediction of Sepsis in ICU Patients [J]. Front Public Health. 2021;9:754348.
WANG R, ZHANG J, SHAN B, et al. XGBoost Machine Learning Algorithm for Prediction of Outcome in Aneurysmal Subarachnoid Hemorrhage [J]. Neuropsychiatr Dis Treat. 2022;18:659–67.

Tables 1 and 2 are available in the Supplementary Files section.

No competing interests reported.

Download PDF

Journal Publication

published 27 Sep, 2023

Read the published version in BMC Infectious Diseases →

Editorial decision: Major revision
04 Sep, 2023
Reviews received at journal
20 Jun, 2023
Reviews received at journal
17 Jun, 2023
Reviewers agreed at journal
13 Jun, 2023
Reviewers agreed at journal
13 Jun, 2023
Reviewers invited by journal
13 Jun, 2023
Editor assigned by journal
13 Jun, 2023
Editor invited by journal
13 Jun, 2023
Submission checks completed at journal
31 May, 2023
First submitted to journal
21 May, 2023

You are reading this latest preprint version

Predicting Sepsis Onset in ICU Using Machine Learning Models: A Systematic Review and Meta-Analysis

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Literature search

Inclusion and exclusion criteria

Data extraction

Risk of bias assessment

Statistical Analysis

Results

Study selection

Characteristics of included studies

Quality assessment

Predictors

Training set and test set accuracy

Training set and test set c-index

Discussion

Conclusion

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1