Study population: development, internal and external validation cohorts
A total of 5,196 patients of MIMIC-IV（2008-2016: 3,986; 2017-2019: 1,210）and 1,494 patients of eICU were included according to the inclusion and exclusion criteria presented in Figure 1 and Supplemental Figure 1. Within the first week of ICU stay, 421 (10.6%), 154 (12.7%) and 119 (8.0%) MV patients developed AKI-23 in the development, internal and external validation cohorts, respectively. There were 2,069 (51.9%), 253 (20.9%) and 411 (27.5%) septic patients, and 785 (19.7%), 306 (25.3%) and 463 (31.0%) cardiac patients in the development, internal and external validation cohorts, respectively. Meanwhile, the median hour from MV to AKI-23 was 52.6 (37.3-122.8), 46.5 (31.5-107.0) and 44.0 (22.0-85.5) in the development, internal and external validation cohorts, respectively. The SCr baseline was 1.0 (0.8-1.4) mg/dL in all cohorts. The need for dialysis or RRT during ICU stay was 203 (5.1%) in the development cohort, 104 (8.6%) in the internal validation cohort, and 37 (2.5%) in the external validation cohort. Other detail information of each cohort was shown in Table 1 and Supplemental Table 1. The information about 7,458 patients excluded for insufficient data in the MIMIC-IV was provided in the Supplemental Table 2.
We ended up extracting a set of 57 candidate variables came from the time before, upon ICU admission, and after MV, including demographic information, primary diagnosis, comorbidities, vital signs, ventilator parameters, additional hemodynamic support, and laboratory values from the databases. Details of the candidate predictors were reported in Supplemental Table 3. The univariable association with AKI-23 and AKI-3 in the development cohort was listed in the Supplemental Table 4-5. The selected predictors based on further multivariable analysis for AKI-23 and AKI-3 model development were shown in Supplemental Table 6-7. In the final, there are 12 and 8 variables were selected as the predictors for AKI-23 and AKI-3 model development, respectively.
Development cohort model performance
In the development cohort, the performances of AKI-23 and AKI-3 models based on random forest were better than those based on logistic regression. The performance of AKI-23 model was with the AUCs of 0.77 (95% CI 0.69-0.84) for logistic regression and 0.82 (95% CI 0.76-0.88) for random forest. Meanwhile, the AUCs of AKI-3 model were 0.79 (95% CI 0.70-0.88) and 0.89 (95% CI 0.84-0.95) for logistic regression and random forest, respectively. Based on the maximum Youden index, the PPVs of AKI-23 model were 0.25 (95% CI 0.20-0.36) for logistic regression and 0.39 (95% CI 0.17-0.53) for random forest. The PPVs of AKI-3 model were 0.21 (95% CI 0.11-0.32) and 0.27 (95% CI 0.14-0.40) for logistic regression and random forest, respectively. The other discrimination results of the clinical prediction models for AKI-23 and AKI-3 in the development cohort based on two machine-learning algorithms were shown in Table 2. The random forest was selected as the final machine-learning algorithm for model development. To improve specificity or PPV, other classification thresholds within the range of clinical usefulness could be chosen (Supplemental Table 8).
Internal validation cohort model performance
The performances for AKI-23 and AKI-3 models developed by random forest in the internal validation cohort were reported in Table 3. The discrimination was shown with AUCs of 0.78 (95% CI 0.74-0.82) and 0.81 (95% CI 0.76-0.87) for AKI-23 and AKI-3, respectively. The classification thresholds identified in the development cohort were 23.5% and 11.9% for AKI-23 and AKI-3, respectively, remained robust and resulted in similar sensitivities, specificities, PPV, and NPV in the internal validation cohort.
The calibration of the models
AKI-23 and AKI-3 models were well calibrated with respective calibration slopes of 1.13 and 1.08 for development cohort, 1.04 and 0.90 for internal validation cohort. The calibration-in-the-large was close to 0 and calibration curve was close to the diagonal (Supplemental Figure 2).
External validation cohort model performance
Table 4 reported the performances of the clinical prediction models for AKI-23 and AKI-3 in the external validation cohort. The discrimination results, based on the classification thresholds identified in the development cohort, had the AUC of 0.80 (95% CI 0.76-0.84), sensitivity of 0.57 (95% CI 0.48-0.66), specificity of 0.83 (95% CI 0.81-0.85), PPV of 0.23 (95% CI 0.20-0.27) and NPV of 0.96 (95% CI 0.95-0.97) for AKI-23, and AUC of 0.80 (95% CI 0.73-0.86), sensitivity of 0.67 (95% CI 0.55-0.78), specificity of 0.85 (95% CI 0.83-0.87), PPV of 0.18 (95% CI 0.15-0.21), and NPV of 0.98 (95% CI 0.98-0.99) for AKI-3.
Model performance in the subpopulations
The AKI-23 and AKI-3 models performed well in all the subpopulations of cardiac patients, respiratory patients, septic patients and nervous patients from the internal and external validation cohorts. In the internal validation cohort, the AUCs were between 0.68 and 0.85 for AKI-23, and between 0.70 and 0.77 for AKI-3. Meanwhile, in the external validation cohort, the AUCs were between 0.73 and 0.85, and between 0.74 and 0.77 for AKI-23 and AKI-3, respectively. The other detailed results were showed in Supplemental Table 9-10.
Model performance of major clinical outcome
All AKI-23 and AKI-3 models performed well at predicting the requirement of dialysis or RRT with AUCs between 0.76 and 0.77 in the internal validation cohort, and 0.76 and 0.78 in the external validation cohort. The other detailed results were illustrated in Supplemental Table 11.
Online MV-associated severe AKI predicting models
We made the fine-tuned models MV-associated AKI-23 and AKI-3 publicly available through our online portal at https://apoet.shinyapps.io/mv_aki_2021_V2