With the expansion of electronic health records(EHR)-linked genomic data comes the development of machine learning-enable models. There is a pressing need to develop robust pipelines to evaluate performance of integrated models and minimize systemic bias. We developed a prediction model of symptomatic Clostridioides difficile infection(CDI) by integrating common EHR-based risk factors and genetic risk factors(rs2227306/IL8). Our pipeline includes 1)leveraging phenotyping algorithm to minimize temporal bias, 2)performing simulation studies to determine the predictive power in samples without genetic information, 3)propensity score matching to control for the confoundings, 4)selecting machine learning algorithms to capture complex feature interactions, 5)performing oversampling to address data imbalance, and 6) optimizing models and ensuring proper bias-variance trade-off. We evaluate the performace of including common clinical risk factors in prediction of CDI and the benefit of including genetic feature(s) into the prediction models. We emphasize the importance of building a robust integrated pipeline to avoid systemic bias and the value of genetic feature should be thoroughly evaluated in general population and subgroups.
Research Article
An Integrated Pipeline For Prediction of Clostridioides Difficile Infection
https://doi.org/10.21203/rs.3.rs-1247028/v1
This work is licensed under a CC BY 4.0 License
You are reading this latest preprint version
Clostridioides difficile
machine learning
IL-8(CXCL8)
genetic polymorphism
genome-wide association study
electronic health records
gradient boosting machine
XGBoost
With biobank and genetic data integrated in electronic health records (EHR) comes the development of predictive models designed for healthcare applications. There is an urgent need to develop robust modeling pipelines using machine learning (ML) to determine whether EHR-derived common clinical risk factors can predict phenotype of interest, and if addition of genetic factors can improve model performance. Several factors have to be considered during the model development including: 1) selection bias for the biobank data; 2) case-control imbalance; 3) temporal bias in feature acquisition; 4) impact of confounding factors; 5) optimal model selection to capture multi-way interactions; and 6) predictive power and generalizability of the final models. Here we choose C.difficile infection (CDI) as a proof-of-concept given its complexity and clinical importance. The purpose is to develop an integrated pipeline for predicting symptomatic CDI using common EHR-derived clinical and genetic risk factors. Focusing on symptomatic CDI is driven by the fact that testing and treatment for CDI are not recommended in asymptomatic individuals1.
CDI is considered the most common cause of healthcare-associated diarrhea and is listed as one of the top five urgent antimicrobial resistance threats by the Centers for Disease Control and Prevention (https://www.cdc.gov/drugresistance/biggest-threats.html). Existing literature reporting CDI prediction focuses on three outcomes –symptomatic infection, the severity of the infection, and recurrence. The prediction models developed in this field vary by setting, patient recruitment, data source, study design, feature selection, and algorithms. Overall, the EHR-based studies have become popular due to their improved predictability, specificity, and generalizability. Host genetic susceptibility to CDI and epidemiology of C. difficile strains have been topics of investigations2,3. Intestinal inflammatory cytokines correlate more closely to disease severity than pathogen burden4. The same mechanism (inflammatory cytokines) applies to the inflammatory cytokine signature (plasma level of IL-6, IL-8, and TNF-α) for the prediction of COVID-19 severity and survival5. A previous candidate gene approach revealed that genetic polymorphisms, rs4073(–251T>A) or rs2227306(+781T/C) from a pro-inflammatory cytokine, IL-8, are associated with IL-8 production and predisposition to CDI6-8 with functional impact (eFigure1). Genetic markers are not yet included in any established disease scoring system or clinical decision tool for risk stratification of CDI due to unclear causality, small effect size, complex gene by environment (GxE) interaction, and data availability/accessibility.
The study was conducted and reported according to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline9. The Geisinger Institutional Review Board approved this study to meet “Non-human subject research” using de-identified information. All research was performed in accordance with relevant guidelines/regulations. Geisinger built and regularly updates the de-identified structured EHR database for research, linked to the MyCode Community Health Initiative biorepository 10-12. The structured EHR and matching genetics data allowed us to conduct a retrospective study on primary CDI 3. Informed consent was obtained from all subjects and/or their legal guardian(s) for the MyCode patients. The analysis pipeline, as well as CDI-specific study parameters, are illustrated in Figure 1.
Robust phenotyping algorithm: The phenotype algorithm used to identify CDI cases and controls from EHR data was developed by eMERGE entitled “Phenotype Algorithm Pseudo Code (August 16, 2012)” and collected at PhenoKB (https://phekb.org/phenotype/clostridium-difficile-colitis). This algorithm adopted the golden standard, which used laboratory test data to identify CDI cases. Based on clinical symptoms, we first identified adults (age ≥18) with CDI from the Geisinger EHR. Three or more consecutive liquid stools within a day may be tested for C. difficile based on recommended guidelines13,14. The polymerase chain reaction was used as the laboratory reference standard test3. Patients tested for C. difficile but with negative results or exposed to antibiotics with high or moderate risk(Appendix-1) served as controls. The observation window for each risk factor was empirically defined to avoid an uneven sampling of the disease trajectory, which might lead to temporal bias in feature selection15. Clinical risk factors and demographic information were extracted from the structured EHR based on the International Classification of Disease (ICD)-related codes and medication codes(Appendix-1). Using data from January 1, 2009, through December 31, 2017, we identified 6,035 cases and 72,241 controls. Of these, 5,911 cases and 69,086 controls had self-reported European ancestry. Overall, 22.4% (1156/14148 for case/control) of European (EUR) patients enrolled in the MyCode project with genetic data available.
Data Pre-Processing and Imputation: The entire cohort of participants with EUR ancestry (n=5,911/69,086) was first split based on the availability of genetic data. No missing data was observed in any of the included variables. The MyCode samples were genotyped as previously reported3. Both SNPs (rs2227306 and rs4076) from IL-8 passed the quality control without missingness.
Simulations to determine predictive power: A genotype simulation study was conducted to determine different modeling algorithms' predictive power and generalizability in the nonMyCode cohort. The genotypes of rs2227306(IL-8) in this cohort of 4,755 cases and 54,938 controls were created by a simulation strategy based on an assumption of a binomial probability distribution of each allele equaling to the prior parameter, MAF, estimated from the corresponding MyCode subgroups stratified by CDI, age, and sex (Table 2.
where R is a vector or a matrix of the summary statistics derived from the MyCode cohort; N and MAF represented the number of subjects and the corresponding MAF for each subcategory. This simulation strategy considers the confounding factors such as age and sex, which may impact the association between the genetic variant and the outcome variable.
Individual SNP Association Testing: Genotype and phenotype association was conducted using Logistic Regression after controlling covariates such as age or sex in subgroups stratified by sex or age (binary).
Controlling confounding: Age and sex were identified as confounding factors for the association between genetic/nongenetic risk factors and CDI (see Result). They were selected as covariates in a Logistic Regression model to create propensity scores(R MatchIt package). We chose “nearest neighbor matching without replacement” to create a more balanced case:control ratio at 1:5 or 1:10 as shown in eFigure2.
Addressing data imbalance: The oversampled data improve ML models for prediction of minority class when there is a significant unbalanced case-control cohort as shown in this study (~1:12). We performed oversampling (using Synthetic Minority Oversampling Technique, SMOTE and random oversampling, ROSE) of the minority class during the model training to address the class imbalance and used F1 score to assess the model performance(eFigure3).
Selecting machine learning algorithms: Model development and optimization were based on selecting the optimal sampling approach followed by a comparison of the eight classification algorithms, including Logistic Regression (glm), Gradient Boosted Classification (gbm), Extreme Gradient Boosting with the dropout regularization for regression trees (xgbDART), Bagging for tree (treebag), Neural Network [nnet, using 1-layer fully connected neural networks (shallow)], C5.0 (c5), LogitBoost (lb), and Support Vector Machine (SVM).
Model optimization: To train the model, we split the data into training(70%) and testing sets(30%). A hyperparameter tuning grid was used to train the model with five-fold repeated cross-validation(CV) and ten repeats(R caret package). Model tuning was performed by an automatic grid search for each algorithm parameter randomly. Finally, the testing set was used to calculate model AUROC.
We also compared optimal models with the benchmark algorithm glm16,17, and extracted the feature importance of the 12 included variables. The ranks of each variable in feature importance, particularly the genetic variable, were compared across the algorithms. The DeLong test was used to compare AUROCs from two modeling algorithms and compare AUROCs of the best model with or without the genetic feature included. The 95% confidence interval(95%CI) of AUROC was also computed18,19.
Data Availability
The patient-level EHR data analyzed in this study may be shared with a third party upon execution of data sharing agreement for reasonable requests. Such requests should be addressed to V. Abedi. All the codes can be found at: TheDecodeLab/Prediction_of_CDI_by_EHR_and_Genetics (github.com)
3.1 Patients Demographics
The entire dataset was split based on the availability of the genetic data. Demographic and clinical information for MyCode and nonMyCode cohorts are listed in Table 1. The case:control ratio in MyCode(n=1156/14148, 1:11.55) was comparable to that in the nonMyCode cohort(n=4755/54938, 1:12.24). This ratio(~1:12) indicated approximately a ten-fold enrichment for controls as shown in the previously reported EHR-based large populational studies(more than 1:100). Several demographic features (e.g., sex, age) and known clinical risk factors showed significant differences between case and control groups in MyCode and nonMyCode cohort. Their bi-variate association among all predictive variables is illustrated in Figure 2a-2b. Antibiotics were the most significant risk factor for CDI.
3.2 Genotype-phenotype Association
The Logistic Regression analysis showed a significant association between rs2227306 genotype and CDI only in young MyCode patients (β=0.138, p=0.048 vs. β=0.062, p=0.263) after controlling for sex. After controlling for age, this nominal association was only observed in females (β=0.119, p=0.034 vs β=0.053, p=0.427). The minor alleles from both SNPs with the higher expression level of CXCL8 and CXCL6 were associated with an increased risk for CDI.
3.3 Comparing Oversampling Methods to Manage the Case-control Imbalance
SMOTE outperformed ROSE in seven out of eight examined algorithms(eFigure3). When using xgbDART and gbm models, SMOTE in the training dataset led to better F1 (0.264 and 0.272, respectively) than the ROSE (0.253 and 0.261, respectively) in the testing dataset. The process without resampling provided the worst F1 (0.037 and 0.056, respectively). SMOTE was chosen for the following analyses.
3.4 Predicting CDI in MyCode patients with or without propensity score matching
Among eight algorithms examined, we found that using the 11 clinical risk factors with rs2227306, in conjunction with gbm and xgbDART led to superior results (with genetic feature, AUROCgbm=0.72[0.694-0.746] versus AUROCglm=0.684[0.655-0.713], p=1.92e-07) (Figure 2c). There was no significant difference between gbm and xgbDART (AUROCxgbDART=0.715[0.689-0.742], p=0.247). The genetic feature was always ranked higher than known risk factors in gbm and xgbDART compared to glm, suggesting non-parametric algorithms can better capture some nonlinear interaction(See the radar plots in Figure 3). Compared to the base model, the integrated model provided the best discriminative power in the testing dataset, particularly for gbm (AUROC =0.710[0.683-0.737] vs. 0.72[0.694-0.746], p = 0.006). xgbDART showed similar trends, better with genetic feature, but did not reach a statistical significance (AUROC=0.710[0.683-0.736], vs. 0.715[0.689-0.742], p=0.304). Both gbm and xgbDART models provided a more balanced sensitivity (0.508 and 0.518 versus 0.481) and specificity (0.792 and 0.796 versus 0.777) and better PPV (0.178 and 0.183 versus 0.161) and NPV (0.948 and 0.949 versus 0.944) when compared to glm. No significant difference between age and sex was observed after the PSM(Table 1, eFigure2). The algorithms with the best performance after matching (e.g. 1:10 ratio) remained gbm (AUROCgbm=0.711[0.681-0.740]) and xgbDART (AUROCxgbDART=0.701[0.671-0.731]) with rs2227306 (Figure 2e). The former showed no statistically significant improvement over the model without rs2227306 (AUROCglm=0.703[0.675-0.732]). A similar trend was observed in the matching cohort with 1:5 ratio (Figure 2e). Both gbm and xgbDART models ranked the genetic feature the highest in feature importance(Figure 3 the top two rows), while glm did not. glm also ranked index age and sex lower.
3.5 Predictive Power in NonMyCode Patients
The simulated genotype in nonMyCode cohort can recapitulate the nominal association between rs2227306 and CDI (Table2). Consistent with the result from MyCode patients, with the simulated genetic feature included, gbm (AUROCgbm=0.820[0.809-0.831]) and xgbDART (AUROCxgbDART=0.819[0.808-0.831]) outperformed other modeling algorithms (e.g., AUROCglm=0.751[0.737-0.765], p=1.68e-57) (Figure 2d). Again, rs2227306 was ranked higher in gbm and xgbDARTthan glm (Figure 3 the bottom two rows)
We developed a prediction model of symptomatic CDI by integrating common risk factors extracted from electronic health records and genetic risk factors (rs2227306/IL8). Our modeling pipeline included steps to minimize systemic bias in the final models while adhering to best practices to improve model transparency. These steps included 1) applying robust and validated phenotyping, 2) selecting and optimizing a range of ML algorithms with a focus on attributes such as generalizability, interpretability, potential interactions, and bias-variance trade-off, 3) addressing data imbalance and performing extensive simulation studies to determine the predictive power in samples without genetic information, and finally using PSM to control for confounding factors.
Overall, our results supported that decision tree-based models such as gbm and xgbDART demonstrated superior discriminative power than glm. The improvements gained by including common genetic variants in the optimal models were limited and age- or sex-dependent after PSM. This finding was consistent with the decreased genetic heritability observed in late-onset compared to early-onset in multiple complex disease traits20, and rs2227306 (IL-8) was related to early-onset of disease21.
4.1 In the context of other similar studies
As summarized in Table 3, the majority of the predictive models for CDI22-28 are not based on large EHR or claims databases until recently29-35, whereas studies performed on recurrence36-39 or severity40-43 include very small cohorts. Further, the existing studies do not compare algorithms; instead, they focus on the amount of information extracted necessary for improved prediction. In general, the amount of information, such as the number of variables, correlates with model performance; however, including hundreds of variables can lead to models with lower interpretability and reduced generalizability to other healthercare systems. For example, some institution-specific features can rank in the top tier in feature importance; therefore, a healthcare-based model may have limited discriminative power in prediction of individuals from others healthcare systems when geographic, social-economic, and clinical management environment differ significantly.
4.2 Strength and limitation
The strength of this study lies in the following, 1) development of a prediction model of symptomatic CDI by integration of genetic and common clinical risk factors; 2) evaluation of several advanced ML algorithms to compare their performance; 3) identification of association between the genetic variant and the outcome variable, which was confounded by age and sex; 4) determination of the value of genetic feature in its contribution to the model performance, in general and propensity score matched subgroups; and 5) identification of the selection bias in the cohort with genetic data available.
This study has some limitations, including 1) the accuracy of EHR data collection and recording processes that may vary by clinician, hospital, and over time to possible prevent the generalizability of developed models to other healthcare systems; 2) our data came from a single healthcare system with a patient population that was predominantly European ancestry. The features selected from this homogenous population may not best represent or cover the complexity of feature space derived from a heterogenous population; and 3) we only tested a common genetic variant with a high MAF. We expect the polygenic risk score developed from the consortium-based GWAS with individual effect size estimated from thousands of genetic variants would better represent the genetic liability to CDI and other complex diseases.
In conclusion, we showed that developing robust prediction models for CDI, and perhaps other complex conditions, requires a step-wise approach to ensure the highest level of transparency and lowest possible systemic bias. This study leveraged CDI as a disease model to demonstrate that although genetic information may improve predictions, the benefit of including genetic feature(s) into the prediction models should be thoroughly evaluated.
Area Under the Receiver Operating Characteristic (AUROC), Bagging for tree (treebag), Body Mass Index (BMI), chemokine (C-X-C motif) ligand (CXCL), Clostridioides difficile (C. difficile),Clostridioides difficile infection (CDI), cross-validation (CV), electronic health records (EHR), expression quantitative trait loci (eQTL), Extreme Gradient Boosting with the dropout regularization for regression trees (xgbDART), genome-wide association study (GWAS), gradient boosting machine (gbm), inflammatory bowel disease (IBD), interleukin - 8 (IL-8), International Classification of Disease (ICD), machine learning (ML), Major Histocompatibility Complex (MHC), minor allele frequency(MAF), Neural Network (nnet), negative predictive value (NPV), phenome-wide association study (PheWAS), positive predictive value (PPV), Support Vector Machine (SVM). Synthetic Minority Oversampling Technique (SMOTE), random oversampling (ROSE), type 2 diabetic mellitus (T2D)
Author contributions statement:
Study conception and design: Jiang Li and Vida Abedi. Data extraction and systematic literature search: Jiang Li, Durgesh Chaudhary, Vaibhav Sharma, Vishakha Sharma. Code development: Jiang Li. Analysis of results: Jiang Li. Data preparation and pre-processing: Jiang Li and Venkatesh Avula; Interpretation of the findings: Jiang Li and Vida Abedi. Preparation of the first draft of the manuscript: Jiang Li and Vida Abedi. Critically evaluated the manuscript: Durgesh Chaudhary, Paddy Ssentongo, Donna M. Wolk, Ramin Zand, and Vida Abedi. All authors reviewed the results and approved the final version of the manuscript.
Acknowledgments
We would like to extend our thanks to Drs. Brodginski, Alison M; Martinez, Raquel M; and Olson, Jordan and Kreis, Steve; and Heiter, Barb J. for clinical data validation.
Conflict of Interest: No conflict of Interest related to this manuscript for all authors.
Funding: This work had no specific funding.
- Khanna, S. & Pardi, D. S. Clostridium difficile infection: new insights into management. Mayo Clin Proc 87, 1106–1117, doi:10.1016/j.mayocp.2012.07.016 (2012).
- Berkell, M. et al. Microbiota-based markers predictive of development of Clostridioides difficile infection. Nat Commun 12, 2241, doi:10.1038/s41467-021-22302-0 (2021).
- Li, J. et al. Variants at the MHC Region Associate With Susceptibility to Clostridioides difficile Infection: A Genome-Wide Association Study Using Comprehensive Electronic Health Records. Front Immunol 12, 638913, doi:10.3389/fimmu.2021.638913 (2021).
- El Feghaly, R. E. et al. Markers of intestinal inflammation, not bacterial burden, correlate with clinical outcomes in Clostridium difficile infection. Clin Infect Dis 56, 1713–1721, doi:10.1093/cid/cit147 (2013).
- Del Valle, D. M. et al. An inflammatory cytokine signature predicts COVID-19 severity and survival. Nat Med 26, 1636–1643, doi:10.1038/s41591-020-1051-9 (2020).
- Jiang, Z. D. et al. A common polymorphism in the interleukin 8 gene promoter is associated with Clostridium difficile diarrhea. Am J Gastroenterol 101, 1112–1116, doi:10.1111/j.1572-0241.2006.00482.x (2006).
- Czepiel, J. et al. The presence of IL-8 +781 T/C polymorphism is associated with the parameters of severe Clostridium difficile infection. Microb Pathog 114, 281–285, doi:10.1016/j.micpath.2017.11.066 (2018).
- Garey, K. W. et al. A common polymorphism in the interleukin-8 gene promoter is associated with an increased risk for recurrent Clostridium difficile infection. Clin Infect Dis 51, 1406–1410, doi:10.1086/657398 (2010).
- Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13, 1, doi:10.1186/s12916-014-0241-z (2015).
- Carey, D. J. et al. The Geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research. Genet Med 18, 906–913, doi:10.1038/gim.2015.187 (2016).
- Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, doi:10.1126/science.aaf7000 (2016).
- Dewey, F. E. et al. Distribution and clinical impact of functional variants in 50,726 whole-exome sequences from the DiscovEHR study. Science 354, doi:10.1126/science.aaf6814 (2016).
- Burnham, C. A. & Carroll, K. C. Diagnosis of Clostridium difficile infection: an ongoing conundrum for clinicians and for clinical laboratories. Clin Microbiol Rev 26, 604–630, doi:10.1128/CMR.00016-13 (2013).
- McDonald, L. C. et al. Clinical Practice Guidelines for Clostridium difficile Infection in Adults and Children: 2017 Update by the Infectious Diseases Society of America (IDSA) and Society for Healthcare Epidemiology of America (SHEA). Clin Infect Dis 66, 987–994, doi:10.1093/cid/ciy149 (2018).
- Yuan, W. et al. Temporal bias in case-control design: preventing reliable predictions of the future. Nat Commun 12, 1107, doi:10.1038/s41467-021-21390-2 (2021).
- Abedi, V. et al. Prediction of Long-Term Stroke Recurrence Using Machine Learning Models. J Clin Med 10, doi:10.3390/jcm10061286 (2021).
- Abedi, V. et al. Predicting short and long-term mortality after acute ischemic stroke using EHR. J Neurol Sci 427, 117560, doi:10.1016/j.jns.2021.117560 (2021).
- DeLong, E. R., DeLong, D. M. & Clarke-Pearson, D. L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44, 837–845 (1988).
- Sun, X. & Xu, W. Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves. IEEE Signal Processing Letters 21, 1389–1393, doi:10.1109/lsp.2014.2337313 (2014).
- Mars, N. et al. Polygenic and clinical risk scores and their impact on age at onset and prediction of cardiometabolic diseases and common cancers. Nat Med 26, 549–557, doi:10.1038/s41591-020-0800-0 (2020).
- Emonts, M. et al. Polymorphisms in genes controlling inflammation and tissue repair in rheumatoid arthritis: a case control study. BMC Med Genet 12, 36, doi:10.1186/1471-2350-12-36 (2011).
- Garey, K. W. et al. A clinical risk index for Clostridium difficile infection in hospitalised patients receiving broad-spectrum antibiotics. J Hosp Infect 70, 142–147, doi:10.1016/j.jhin.2008.06.026 (2008).
- Tanner, J., Khan, D., Anthony, D. & Paton, J. Waterlow score to predict patients at risk of developing Clostridium difficile-associated disease. J Hosp Infect 71, 239–244, doi:10.1016/j.jhin.2008.11.017 (2009).
- Dubberke, E. R. et al. Development and validation of a Clostridium difficile infection risk prediction model. Infect Control Hosp Epidemiol 32, 360–366, doi:10.1086/658944 (2011).
- Chandra, S., Thapa, R., Marur, S. & Jani, N. Validation of a clinical prediction scale for hospital-onset Clostridium difficile infection. J Clin Gastroenterol 48, 419–422, doi:10.1097/MCG.0000000000000012 (2014).
- Smith, L. A. et al. Development and validation of a Clostridium difficile risk assessment tool. AACN Adv Crit Care 25, 334–346, doi:10.1097/NCI.0000000000000046 (2014).
- van Werkhoven, C. H. et al. Identification of patients at high risk for Clostridium difficile infection: development and validation of a risk prediction model in hospitalized patients treated with antibiotics. Clin Microbiol Infect 21, 786 e781-788, doi:10.1016/j.cmi.2015.04.005 (2015).
- Tilton, C. S. & Johnson, S. W. Development of a risk prediction model for hospital-onset Clostridium difficile infection in patients receiving systemic antibiotics. Am J Infect Control 47, 280–284, doi:10.1016/j.ajic.2018.08.021 (2019).
- Wiens, J., Guttag, J. & Horvitz, E. A study in transfer learning: leveraging data from multiple hospitals to enhance hospital-specific predictions. J Am Med Inform Assoc 21, 699–706, doi:10.1136/amiajnl-2013-002162 (2014).
- Wiens, J., Campbell, W. N., Franklin, E. S., Guttag, J. V. & Horvitz, E. Learning Data-Driven Patient Risk Stratification Models for Clostridium difficile. Open Forum Infect Dis 1, ofu045, doi:10.1093/ofid/ofu045 (2014).
- Baggs, J. et al. Identification of population at risk for future Clostridium difficile infection following hospital discharge to be targeted for vaccine trials. Vaccine 33, 6241–6249, doi:10.1016/j.vaccine.2015.09.078 (2015).
- Press, A. et al. Developing a Clinical Prediction Rule for First Hospital-Onset Clostridium difficile Infections: A Retrospective Observational Study. Infect Control Hosp Epidemiol 37, 896–900, doi:10.1017/ice.2016.97 (2016).
- Zilberberg, M. D., Shorr, A. F., Wang, L., Baser, O. & Yu, H. Development and Validation of a Risk Score for Clostridium difficile Infection in Medicare Beneficiaries: A Population-Based Cohort Study. J Am Geriatr Soc 64, 1690–1695, doi:10.1111/jgs.14236 (2016).
- Oh, J. et al. A Generalizable, Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centers. Infect Control Hosp Epidemiol 39, 425–433, doi:10.1017/ice.2018.16 (2018).
- Aukes, L. et al. A Risk Score to Predict Clostridioides difficile Infection. Open Forum Infect Dis 8, ofab052, doi:10.1093/ofid/ofab052 (2021).
- Hebert, C., Du, H., Peterson, L. R. & Robicsek, A. Electronic health record-based detection of risk factors for Clostridium difficile infection relapse. Infect Control Hosp Epidemiol 34, 407–414, doi:10.1086/669864 (2013).
- LaBarbera, F. D., Nikiforov, I., Parvathenani, A., Pramil, V. & Gorrepati, S. A prediction model for Clostridium difficile recurrence. J Community Hosp Intern Med Perspect 5, 26033, doi:10.3402/jchimp.v5.26033 (2015).
- Escobar, G. J. et al. Prediction of Recurrent Clostridium Difficile Infection Using Comprehensive Electronic Medical Records in an Integrated Healthcare Delivery System. Infect Control Hosp Epidemiol 38, 1196–1203, doi:10.1017/ice.2017.176 (2017).
- Cobo, J. et al. Prediction of recurrent clostridium difficile infection at the bedside: the GEIH-CDI score. Int J Antimicrob Agents 51, 393–398, doi:10.1016/j.ijantimicag.2017.09.010 (2018).
- Drew, R. J. & Boyle, B. RUWA scoring system: a novel predictive tool for the identification of patients at high risk for complications from Clostridium difficile infection. J Hosp Infect 71, 93-94; author reply 94-95, doi:10.1016/j.jhin.2008.09.020 (2009).
- Lungulescu, O. A., Cao, W., Gatskevich, E., Tlhabano, L. & Stratidis, J. G. CSI: a severity index for Clostridium difficile infection at the time of admission. J Hosp Infect 79, 151–154, doi:10.1016/j.jhin.2011.04.017 (2011).
- Na, X. et al. A Multi-Center Prospective Derivation and Validation of a Clinical Prediction Tool for Severe Clostridium difficile Infection. PLoS One 10, e0123405, doi:10.1371/journal.pone.0123405 (2015).
- Li, B. Y., Oh, J., Young, V. B., Rao, K. & Wiens, J. Using Machine Learning and the Electronic Health Record to Predict Complicated Clostridium difficile Infection. Open Forum Infect Dis 6, ofz186, doi:10.1093/ofid/ofz186 (2019).
- Origuen, J. et al. Toxin B PCR Amplification Cycle Threshold Adds Little to Clinical Variables for Predicting Outcomes in Clostridium difficile Infection: a Retrospective Cohort Study. J Clin Microbiol 57, doi:10.1128/JCM.01125-18 (2019).
- Sun, B. B. et al. Genomic atlas of the human plasma proteome. Nature 558, 73–79, doi:10.1038/s41586-018-0175-2 (2018).
Table 1 is available in the Supplemental Files section.
Table2. Two simulation strategies to create the genetic data of rs2227306(IL8) in nonMyCode cohort.
|
CDI (outcome) |
AGE (zscore≥0) |
SEX (female) |
Number of Subject |
Minor Allele Frequency |
χ2/pvalue |
|||
|
MyCode |
NonMyCode |
MyCode |
NonMyCode |
MyCode |
NonMyCode |
|||
|
0 |
0 |
0 |
1646 |
12757 |
0.4067 |
0.4111 |
7.443/0.024 |
4.605/0.1 |
|
0 |
0 |
1 |
4928 |
16072 |
0.4074 |
0.4098 |
||
|
0 |
1 |
0 |
3553 |
12066 |
0.4191 |
0.4174 |
||
|
0 |
1 |
1 |
4021 |
14043 |
0.4193 |
0.4233 |
||
|
1 |
0 |
0 |
157 |
527 |
0.4522 |
0.4639 |
3.515/0.173 |
11.88/0.003 |
|
1 |
0 |
1 |
286 |
688 |
0.4353 |
0.4317 |
4.701/0.095 |
3.009/0.222 |
|
1 |
1 |
0 |
322 |
1464 |
0.4161 |
0.3924 |
0.026/0.987 |
7.23/0.027 |
|
1 |
1 |
1 |
391 |
2076 |
0.4501 |
0.4381 |
3.154/0.207 |
5.37/0.068 |
This table summarizes the sample size, minor allele frequency(MAF) of each subgroup stratified by the outcome variable (CDI), age, and sex in both MyCode and nonMyCode cohort. The genotype of rs2227306 was simulated based on an assumption of binomial distribution of each allele with frequency equaling to the prior parameter (MAF) estimated from the corresponding MyCode samples. The genotype for each subject would be a combination of sampling from two binomial distributions with the same MAF.
Table 3. Summary of the reported clinical decision tools to predict Clostridioides difficile infection (CDI), severity, or recurrence.
We conducted a comprehensive search on PubMed and Web of Science by combining two major themes of Clostridioides difficile and prediction. The search strings for prediction of Clostridioides difficile infection were: ("clostridioides difficile "[Mesh] OR "clostridium difficile"[Mesh]) AND ("prediction"[Mesh] OR "machine learning”). We focused on human subject studies. Any review articles and studies focusing on metagenomics or microbiome data were excluded. PubMed database was searched and studies available between January 1, 1990, and May 31, 2021, were included. We also checked the reference from each included studies and additional studies that were missed during the initial search were appended. A total of 23 original articles were included.
|
Study Design |
Study Characteristics (Timeline, Sample size, and Features) |
Outcome Measures |
Handing Missing Data |
Algorithms |
Performance |
Summary |
|
Retrospective cohort35.(2021)
|
EHR (May 2010 to July 2014) from a single healthcare; 9986 CDI cases and 2 230 354 members without CDI; 104, 518 hospital discharges for validation; case:control ≈ 1:23 ≈ 20 Risk factors |
CDI |
Not addressed |
Univariate logistic regression Logistic Regression to develop 2 risk scores. Model 1: hospital discharge IDRSA. Model 2: random IDRSA. |
Model 1: using hospital discharge as the IDRSA, C-statistic of 0.848 in subsequent 31-365 days; Model 2: using a random date as the IDRSA, C-statistic 0.722 |
Identification of high-risk populations for CDiff vaccine trials to determine the study feasibility (sample size and time to completion) |
|
Case-control study 28.(2019) |
Adult patients admitted to multicenter study July 1, 2015, to July 1, 2017, who received systemic antibiotics. 200 subjects (100 cases and 100 controls) Reported 2 features |
CDI(hospital-associated) |
Not addressed |
Univariate logistic regression Multivariate logistic regression model to formulate a point-base risk prediction model |
Sensitivity and specificity were 76% and 49% Highest accuracy (63%) AUROC = 0.7 |
A simple-to-implement hospital-onset CDI risk model; including only independent risks that can be obtained immediately on presentation to the healthcare facility |
|
Retrospective Cohort 34. (2018)
|
EHR based adult inpatients admitted to two healthcare systems one( January 1, 2010, and January 1, 2016) and the other ( June 1, 2012, and June 1, 2014) 191,014 (155,009/36005 for training/testing) and 65,718 (33,477/32,241 for training/testing) for two healthcare systems respectively Case:control ≈ 1:100 4,836 &1,837 features from two healthcare system respectively |
CDI |
Not addressed |
L2 regularized logistic regression Logistic regression to create a daily risk score for risk stratification |
AUROC = 0.82[0.80-0.84] and 0.75[0.73-0.78] for two cohorts respectively. |
Many of the top predictive factors differed between the cohorts from two healthcare systems. Institution-specific models instead of “one-size-fits-all” models |
|
Retrospective Cohort 33. (2016)
|
Population-based sample Medicare beneficiaries aged 65 and older on January 1, 2008, with continuous Medicare coverage from January 1, 2008, through December 31, 2009.Inpatient setting (58.5%) Of 1,165,165 Medicare beneficiaries meeting the enrollment criteria, 6,838 had an incident CDI episode; case:control = 1:170 22 features |
CDI |
Not addressed |
Logistic regression model for feature selection sequentially remove features with < 0.8 change in C-statistic; A weighted score was developed for each of the risk factors based on its odds ratio, with the sum of all of the risk values representing a participant’s individual risk score |
C-statistic = 0.858 NPV = 98.7% |
Developed a risk stratification scoring system Emphasized the age-dependent CDI |
|
Retrospective cohort 32. (2016)
|
admitted over a 1-year period (2013). Total of 61,482 subjects, Discovery dataset (40,990) and validation dataset (20,492) case:control ≈1:200 ~25 features |
CDI (hospital-associated) |
Not addressed |
Multivariable analysis to identify risk factors individually Multivariable model based on six risk factors to develop a risk score |
Sensitivity = 82.0%; Specificity = 75.7%; AUROC = 0.85 |
Developed a clinical prediction rule to identify patients at high risk for primary CDI. |
|
Retrospective cohort (longitudinal) 31. (2015)
|
Hospital discharge data and pharmacy data from two large academic centers linked to active population-based CDI surveillance data from the Emerging Infections Program (EIP) Of the 35,186 index hospitalizations, 288 (0.82%) had CDI ≥28 days post discharge 39 features to begin with, 4 features left |
CDI (Having CDI ≥28 days post discharge |
Not addressed |
Cox proportional hazards model (stepwise backward selection) for low and high risk groups |
C-statistics = 0.75 |
Develop a risk score applied at discharge to identify a risk of CDI≥28 days post discharge |
|
Case-control study 27. (2015) |
Patients admitted between January 2005 and December Discovery:180 cases and 330 controls; Validation: 97 cases and 417 controls; case:control ≈ 1:120 12 features |
CDI (hospital-associated) |
Not addressed |
Stepwise backward elimination to determine the best fit model. Logistic regression to develop a simplified risk score |
Corrected AUROC = 0.81[0.77-0.85]; calibration: Brier score = 0.004 |
Developed and validated a model to predict the incident CDI in hospitalized patients who receive systemic antibiotic treatment |
|
Retrospective cohort 30. (2014) |
All patients admitted on or after April 12, 2011 and discharged on or before April 12 Training: 34 846 admissions (372 cases of CDI).Validation: 34 722 admissions (355 cases of CDI) Case:Control ≈1:100 14 features (EHR Model) 1017 features (Curated Model) 10,859 features ( EHR ALL) |
CDI (hours from the time of admission) |
Not addressed |
L2-Reguralized Logistic Regression 3 Models Compared based on different number of features included in the final models to discriminate low-risk from high-risk patients |
AUROC Risk Period> 24h EHR = 0.81(.79–.83) Curated = 0.72 (.69–.75) EHR ALL = 0.8140 (.80–.83) Risk Period> 48h EHR = 0.7886 (.76–.82) |
Additional features from EHR data improved prediction and outperformed the model only considering a small set of known clinical risk factors. |
|
Retrospective cohort 29. (2014)
|
all inpatient visits for the 2 years between April 2011 and April 2013. 1348 test positive case of C difficile out of 132 853 admissions from three hospitals, varying in size and location; case:control = 1:100 578 binary features; Different feature space including common (256) and specific features |
CDI (hospital-associated) |
Missingness has been discussed; source feature space and target feature space |
L2-regularized logistic regression Multivariate Logistic regression |
AUROC ≈ 0.80 varied by the approach and target task |
The external data from other hospitals can be successfully and efficiently incorporated into hospital-specific models. |
|
Case-control study 26. (2014)
|
Not available (abstract only) 8 Known risk factors |
CDI |
Not addressed |
All feature included Multivariate regression model to create a weighted score tool |
Sensitivity = 92%; Specificity = 39% |
Developed a weighted scoring tool to predict incident CDI |
|
Retrospective cohort 25. (2014)
|
a consecutive cohort of patients admitted to the adult medical service over a period of 17 months (June 2011 to October 2012). 62 out of 7026 patients with over 48h hospital stay having hospital-onset CDI cases; case:control = 1:100 Reported 6 features |
CDI (hospital-onset) |
Addressed for missingness in serum albumin level |
Univariate analysis to determine the potential risk factors included in the model Multivariable logistic regression model using a forward stepwise selection for features |
AUROC = 0.94 [ 0.92-0.95]. Sensitivity = 98.3% [90.2-99.9]; Specificity = 85.2% [84.3-86.0] |
Developed a predictive scale for hospital-onset CDI which can be used for risk stratification |
|
Retrospective Cohort 24. (2011)
|
Patients admitted for ≥48 hours during the calendar year 2003 from a single healthcare system 35,350 total admissions & 329 CDI cases. Case:control ≈ 1:100 11 features |
CDI |
Not addressed |
Feature selection based on high dimensional data reduction techniques such as PCA, cluster analyses Logistic stepwise regression to determine the best fit model Logistic regression also test for the some feature interactions |
C index=0.88; Brier score 0.009) |
Developed and validated a CDI risk prediction model using EHR with strong discriminative capacity. |
|
Retrospective cohort 23. (2009)
|
Three phases design: discovery dataset (NA), testing dataset (n=1468), and external validation (n=29425) |
CDI Clostridium difficile-associated disease (CDAD) |
Not addressed |
Logistic regression model |
AUROC = 0.827; Sensitivity = 70% and specificity = 95% |
Developed a predictive score to predict patient’ risk of developing CDAD. |
|
Retrospective cohort 22 (2008)
|
Temporal split. development cohort (March 2005 to December 2006) and a validation cohort (January 2007 to October 2007). a cohort of hospital patients given broad-spectrum antibiotics 392 (288/104) out of 54226(41224/13002). Case:control ≈ 1:100 Reported 4 features |
CDI |
Not addressed |
Logistic regression model to identify significant predictor individually A scoring algorithm to create four categories of CDI risk |
AUROC = 0.712 |
Developed an easily implemented risk index for risk stratification of patients. |
|
Prospective Cohort 39. (2018)
|
Patients symptomatic of CDIFF 274 (Training dataset); 183 (Validation cohort). Reported 4 features |
Recurrence |
Not addressed |
Logistic regression Model with model calibration using Hosmer-Lemeshow test. Logistic regression to form a GEIH-CDI score |
AUROC = 0.72 (0.65-0.79). |
Develop a risk score for recurrent CDI prediction and stratification |
|
Retrospective cohort 44. (2019)
|
First-episode of adult CDI from January 1, to December 2015 For recurrence, 36 vs 191 For poor outcome, 70 vs157 (no testing dataset) ≈35 features |
Recurrence Severity |
addressed |
Univariate analyses; Backward stepwise multivariate logistic regression Multivariate regression
|
AUROC = 0.728/0.789 for clinical model; 0.775/0.801 for EIA-included model; 0.785/0.804 for PCR-including model for recurrence/severity |
|
|
Retrospective cohort (longitudinal)38 (2017)
|
EHR (2007–2013) from a single healthcare. Training: 9,386 incident CDI & 1,311 first CDI recurrence; testing: 1865 incident CDI &144 recurrent CDI 150 predictors |
Recurrence (inpatient or outpatient) |
Right-censored data |
Univariate and bivariate regression Competing risk discrete survival models and Cox competing risk survival regression |
Basic (C-statistic: 0.591, sensitivity: 75.69, specificity: 41.19). Enhanced (C-statistic: 0.587, sensitivity: 69.44, specificity: 43.64). Automated (C-statistic: 0.605, sensitivity: 79.17, specificity: 32.04). Zilberg (c-stat: 0.591, sensitivity: 74.31, specificity: 39.03). |
None of the models showed the well discriminative power. Suggest including environmental and ecological predictors |
|
Retrospective Cohort.37 (2015) |
Patients with lab test positive for CDIFF between January 2009 and June 2013 at single healthcare system 198 CDI with 30 having CDR and break into 70% & 30% for training and testing 25 features |
Recurrence |
Not addressed |
All features included Random Forest |
Sensitivity(83.3%), Specificity(63.1%), and AUROC (0.826) |
Expecting Random Forest model with a higher performance |
|
Retrospective Cohort 36. (2013)
|
Janurary 2006 – October 2010 from 4-hosptial Heath Care Organization 198 out of 829 with relapse for 56 days of follow-up Reported 6 features |
Relapse |
Not addressed |
Univariate logistic regression Multivariate logistic regression |
Predicted 14.6% of CDI Episodes |
Comprehensive EHR can be used to identify patients at high risk for CDI relapse. Major risk factors include antibiotic and PPI exposure |
|
Retrospective Cohort 43(2019)
|
adult inpatients diagnosed with CDI from October 2010 to January 2013 at a single healthcare. 89 out of 1144 cases of CDI having complicated CDI; 894 cases for training and 224 cases for tesing. 23 features for the curated model; 4271 features from EHR; final selected 900 features; 923 features for EHR + curated |
Severity (3Day Complications) |
No imputation or case-wise |
Compared EHR-based model to one based on a small set of manually curated features L2 regularization regression model Logistic regression |
AUROC = 0.69[0.55-0.83) on the day of CDI diagnosis; AUROC = 0.90[0.83-0.95] 2 days after CDI diagnosis; outperformed curated feature model with AUROC = 0.84[0.75-0.91] |
Develop a model based on EHR data to accurately stratify CDI cases according to their risk of developing complications. |
|
Prospective Cohort 42. (2015)
|
Discovery dataset: Boston site from December 2004 to January 2006, Validation dataset: Dublin site from November 2007 to June 2009 , & Houston site from January 2006 to August 2010 251 for Discovery and 345 for validation 3 features (Age, WBC, and Creatinine) |
Severity |
Not addressed |
Univariate logistic regression Multivariate logistic regression analysis to form a Clostridium difficile severity score (CDSS) |
AUROC = 0.725 [0.675-0.769] |
Developed a CDSS scoring system to predict severe CDI |
|
Retrospective cohort 41, et al. (2011)
|
January 2004 and December 2007 255 patients 4 risk factors (history of malignancy + 3 laboratory variables) |
Severity |
Not addressed |
Univariate analysis Composite scoring: CDI severity index score |
AUROC = 0.78; Sensitivity = 82%; Specificity = 65% |
Develop a composite score for risk stratification of severe CDI |
|
Prospective Cohort40. (2009)
|
A single healthcare 8 out of 58 for day1 and 75 for day 3 having severe complications 3 Laboratory variables |
Severity |
Not addressed |
No feature selection Composite scoring: RUWA scoring system |
Sensitivity: 80.0% [39.4-96.3] and 62.5% [32.3- 85.6]; Specificity: 77.4% [73.5-78.9] and 82.1% [78.5-84.8] on day 1 and day 3 respectively. |
the Ratio of white cell count on the day of the positive C. difficile toxin test to two days previously, as well as the Urea, White cell count and Albumin on the day of the positive C. difficile toxin test. |
No competing interests reported.
- Table1.docx
- 01252022supplementarymaterial.pdf
Appendix 1: Inclusion and exclusion criteria to identify cases and controls and extraction of clinical risk factors by an eMERGE phenotyping algorithm with some modifications.