Comparison of Machine Learning Methods and Conventional Regression for Predicting Renal Function Recovery and Recovery Time among AKI Patients in ICU

doi:10.21203/rs.3.rs-1344528/v1

Download PDF

Research Article

Comparison of Machine Learning Methods and Conventional Regression for Predicting Renal Function Recovery and Recovery Time among AKI Patients in ICU

https://doi.org/10.21203/rs.3.rs-1344528/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Acute kidney injury (AKI) is one of the most frequent complications of critical illness. Early predicted renal function recovery can improve AKI prognosis with appropriate interventions. We aimed to derivate and validate two prediction models with high prediction performance and clinical interpretability for renal function recovery and recovery time after AKI, respectively.

Methods: We reviewed patients diagnosed with acute kidney disease in intensive care (MIMIC-III) database between 2001 and 2012. The conventional logistic regression and machine algorithm were used to develop prediction models. The performance measures included the area under the receiver operating characteristic curve (AU-ROC), Precision-Recall (PR) curves, and the classification matrix.

Results: A total of 3,989 critically-ill AKI adult patients corresponding with Kidney Disease Improving Global Outcomes (KDIGO) with the length of stay in ICU more than 48 hours were included in our analysis cohort. Comparing the model performance, the gradient boosting machine model for predicting renal function recovery had a higher AU-ROC (0.92 [95%CI 0.89-0.94]) than the logistic regression model (0.88 [95%CI 0.84-0.92]). The ExtraTrees machine learning-based model (R2=0.2367) was developed to predict recovery time after AKI, which yielded a better result than the linear regression (R2=0.2006).

Conclusions: In this cohort study, machine learning algorithms showed superior ability to predict the prognosis of AKI patients in ICU compared with the traditional logistic and linear regression models. These models may prove clinically helpful and assist clinicians in providing intervention in time, potentially leading to improved prognosis.

acute kidney injury

renal function recovery

renal function recovery time

machine learning

Acute kidney injury (AKI) is one of the most common diseases with an incidence from 10–15% of in-hospital patients [1]; in contrast, it reaches up 50%-60% in critically ill patients [2]. Despite advances in healthcare, the development of AKI is still independently associated with an increased health care cost, length of hospital stay, in-hospital morbidity, and mortality [3–5]. However, prognostic uncertainty and optimism partialities may lead clinicians to delay essential conservations. Therefore, prognostic models may reverse bad outcomes by timely treatment.

The severity of AKI and its preventability and remedy make AKI a consummate candidate for predictive analytics. Some scholars point out that age, comorbidities, baseline renal function, and proteinuria have been shown to predict the probability of AKI recovery [6, 7]. In addition, Srisawat and colleagues constructed a prediction model, which found that the APACHE Ⅱ score and Charlson comorbidity index were predictors [8]. However, their study only included small group patients (n=76), which may reduce the accuracy of real-time implementation. Overall, current research showed the limitation for predicting whether the individual patient with AKI will recover and recovery time.

Currently, to the best of our knowledge, there are few clinical studies, including significant group patients compared machine learning models to conventional regression models to predict renal function recovery and recovery time after an episode of AKI. Therefore, it is hypothesized that our prognostic model may be accurate to earlier recognized renal function recovery among these vulnerable populations to improve prognosis by the increased opportunity to assist patients in reaching AKI recovery and the prevention of further renal insults in the setting of evolving injury, which may lead to chronic kidney disease (CKD).

Sources of Data

This retrospective study was conducted by collecting data from an extensive critical care database named Multiparameter Intelligent Monitoring in Intensive Care Database III (MIMIC III), which included all laboratory and medical test results, pharmacy, and diagnosis codes for more than 40,000 ICU patients treated at Beth Israel Deaconess Medical Center (Boston, MA, USA) from June 1, 2001, to October 31, 2012 [9]. In order to apply for access to the database, we completed the National Institutes of Health’s web-based course and successfully passed the Protecting Human Research Participants exam (No. 9936285). This study was approved by Peking University People’s Hospital's institutional review board and followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis and Diagnosis (TRIPOD) reporting guideline.

Selecting an AKI Cohort

According to Kidney Disease Improving Global Outcomes (KDIGO) clinical practice guideline, we initially screened all adult patients (≥18 years) meeting criteria for AKI [10]. Due to we do not have SCr values prior to the index hospitalization, and as such cannot truly determine baseline SCr for all patients, we estimated baseline kidney function by calculating the Modification of Diet in Renal Disease (MDRD) study equation assuming an eGFR of 75mlml/min/1.73m² for patients [11, 12]. Patients who were discharged or died within 48 hours after ICU admission and those who were stayed at ICU for more than 90 days were excluded. In addition, we excluded the cohort to participants who were died with renal function recovery. Additional exclusion criteria were history of receiving long-term renal replacement treatment (RRT), diagnosis of advanced CKD, absence of creatinine record after the AKI diagnosis, maximum SCr was smaller than baseline SCr calculated by equation.

Data Collection and Definition

Data were abstracted from MIMIC III using Structured Query Language (SQL) with Navicat Premium (version 12.0.28). We obtained demographic and clinical data within the first 48 h after ICU admission and diagnosis of AKI. Comorbidities and diagnoses were identified based on the ICD-9 code. Vital signs including systolic blood pressure (SBP), heart rate (HR), respiratory rate (RR), peripheral oxygen saturation (SpO₂), and temperature (T) were extracted. Furthermore, laboratory data including hemoglobin, platelet, total bilirubin, lactate, albumin, cardiac troponin T, bicarbonate, chloride, base excess, prothrombin time, urine ratio, proteinuria were also recorded. In addition, variables associated with AKI diagnosis were also abstracted, such as urine volume and SCr. Advanced life support therapy such as vasopressors, RRT, and mechanical ventilation was also collected. Because of the high sampling frequency, we used the minimize, maximum, and mean values when extracting the characteristics of vital signs and laboratory data. Meanwhile, we extracted the values related to the most significant disease severity for variables measured more than once during the first 48 h after ICU admission and diagnosis of AKI.

In the present study, we recognized that the renal function recovery was no longer fulfilled criteria for AKI stage 1, but SCr level may not yet be returned to baseline (defined as the return of SCr to <30% above baseline) before ICU discharge [13, 14]. An alternative definition of non-recovery was the presence of meeting AKI criteria or dying during the ICU stay. In addition, the AKI start-time was the first time when the patient fitted the KDIGO criteria. We defined AKI recovery-time as the first time SCr returned to 0.3 mg/dL above baseline. The AKI recovery time was measured as AKI recovery time minus start time.

Statistical Analysis

Patients were divided into two groups based on whether the renal function recovery and variables were displayed and compared. Demographics and other characteristics were summarized using means and standard deviations, medians, interquartile ranges, or frequency counts and percentages. The chi-Squared test was used for categorical variables, and the Mann-Whitney U test was used to compare discrete distribution, while continuous variables were tested by Independent t-test. We also conducted Pearson correlation analysis, which showed two variables were highly correlated if their correlation score was above 0.7. All data were analyzed using Python version 3.8 and R 4.0.5 (Vienna, Austria) statistical software, with statistical significance set at p-value <0.05.

Renal Function Recovery Model Development and Validation

We randomly separated the model development data into two parts: 90% for model derivation and 10% for internal validation. We developed conventional logistic regression and XGBoost algorithm model on the training dataset, including all variables, and verified these models in the validation dataset for renal function recovery prediction. In the conventional method, each risk factor was used in the univariate analysis, and then the backward stepwise multivariate analysis was conducted to build the best fit logistic regression model. XGBoost is based on the sparsity-aware algorithm and weighted quantile sketch, in which the weak learners can be converged sequentially into the ensemble to achieve a strong learner [15].

Recovery Time Model Derivation and Validation

The recovery population was randomly split into a training cohort, in which the recovery time algorithms were derived, and a validation cohort, in which the algorithms were applied and tested. The training cohort consisted of 90% of the AKI-recovery cohort, and the validation cohort consisted of the remaining 10%. We developed a multivariate linear regression model and ExtraTrees machine learning-based model to identify better predictors. We built an overall ExtraTrees model, which contained 2,654 AKI recovered patients firstly, in which we found gender was a significant predictor for the overall renal function recovery duration. Therefore, we conducted two separate models for male and female patients.

Evaluating Performance of Models

To assess the model quality of AKI recovery, we chose the area under the receiver operating characteristic curve (AU-ROC) as the measurement to compare the performance of the logistic regression and XGBoost algorithm model. In addition, Precision-Recall (PR) curves have been then cited as an alternative to ROC curves for tasks with a significant skew in the class distribution [16]. To further assess model performance, a plot of the percentage of observations above a probability threshold versus the percentage of observations was constructed and followed by secondary metrics of clinical prediction models, including accuracy, sensitivity, specificity, precision, and recall [17].

To envisage the contribution of the predictor variables in the most reliable AKI recovery time model, we used R² to evaluate the performance of the models, with higher values indicating a more incredible prediction. Furthermore, the models' root mean square error (RMSE) was also calculated.

Demographics, clinical characteristics, and AKI metric measurements

In total, 33,352 patient encounters met KDIGO criteria within the first 48 hours after ICU admission. After excluding the patients according to the exclusion criteria, the final analysis cohort consisted of 3,989 eligible patients (Fig. 1). In the analytic cohort, the average age was 65.91±15.08 years old, male patients accounted for 53.9% (n=2150), white patients occupied 72.7% (2903), and the median SOFA score was 5.82±3.41. Of these, the cohort was divided into two groups: AKI recovery group (n=2,947, 73.9%) vs. AKI non-recovery (n=1,042, 26.1%). Of the recruited patients, 923 were stage 1 (817 AKI recovery vs. 106 AKI non-recovery), 1,978 were stage 2 (1,625 AKI recovery vs. 353 AKI non-recovery), 1,088 were stage 3 (505 AKI recovery vs. 583 AKI non-recovery). The average recovery time was 2.1 days for total recovery critically ill patients. Baseline demographics, clinical characteristics, interventions, and outcomes were outlined in Table 1.

Table 1

Distribution of baseline characteristics between AKI recovery and non-recovery group
Characteristic	Total (n=3,989)	AKI recovery (n=2,947)	AKI non-recovery (n=1,042)	p value
Age, y, mean ± SD	65.91±15.08	65.75±14.68	66.38±16.16	0.026*
Male Gender, n (%)	2150 (53.9)	1601 (54.3)	549 (52.7)	0.381
Ethnicity, n (%)				0.009**
White	2903 (72.7)	2177 (73.9)	726 (69.7)
Asian	78 (1.9)	59 (2)	19 (1.8)
Black	273 (6.8)	195 (6.6)	78 (7.5)
Hispanic	120 (3)	90 (3.1)	30 (2.9)
Unknown	514 (12.9)	351 (11.9)	163 (15.6)
Alcohol Abuse, n (%)	336 (0.08)	240 (0.08)	96 (0.09)	0.316
SOFA Score, mean ± SD	5.82±3.41	5.31±3.18	7.24±3.63	<0.001***
Comorbidities, n (%)
CVD	1367 (34.3)	1105 (37.49)	262 (25.14)	<0.001***
ARDS	968 (24.3)	606 (20.6)	362 (34.7)	<0.001***
Hypertension	2517 (63.1)	1913 (64.9)	604 (57.9)	<0.001***
Chronic Pulmonary Disease	851 (21.3)	606 (20.6)	362 (34.7)	0.0284*
Heart Failure	413 (10.4)	311 (10.6)	102 (9.8)	0.524
Organ Dysfunction, n (%)	2386 (59.8)	1562 (53)	824 (79)	<0.001***
Sepsis, n (%)	547 (13.7)	335 (11.4)	212 (20.3)	<0.001***
Vital Signs
HR min 24h (bpm), M (P₂₅-P₇₅)	71 (61-82)	71 (61-81)	72 (62-83)	0.112
HR 24h (bpm), M (P₂₅-P₇₅)	103 (91-118)	103 (91-118)	103 (90-119)	0.949
RR min 24h (bpm), M (P₂₅-P₇₅)	12 (10-14)	12 (10-14)	12 (10-15)	<0.001***
RR max 24h (bpm), M (P₂₅-P₇₅)	12 (10-15)	27 (24-31)	28 (24-32)	0.165
MAP min 24h (mmHg), M (P₂₅-P₇₅)	58.39 (52.72-65.06)	58.5 (53.09-65.22)	57.98 (51.31-64.63)	0.013*
MAP max 24h (mmHg), M (P₂₅-P₇₅)	102 (92-115)	102 (92-115)	102 (90-115.92)	<0.001***
SBP min <90 (mmHg), M (P₂₅-P₇₅)	2373 (59.5)	1785 (60.6)	588 (56.4)	0.021*
T max >37.3 (℃), n (%)	2661 (66.7)	2049 (69.5)	612 (58.7)	<0.001**
T max 24h, mean ± SD	37.66±0.807	37.7±0.768	37.57±0.899	<0.001**
SpO2 1st, mean ± SD	91.13±7.91	91.54±6.94	89.97±10.13	<0.001***
Urinary variables
Urine Ratio 6h, M (P₂₅-P₇₅)]	0.46 (0.35-0.59)	0.46 (0.37-0.59)	0.43 (0.25-0.65)	<0.001***
Proteinuria 24h, M (P₂₅-P₇₅)	146.9 (90-160)	123.5 (90-135)	213.2 (130-160)	<0.001***
Pre-AKI Urine Volume 24h, M (P₂₅-P₇₅)	1000 (632-1407)	1000 (619-1644)	1000.5 (725- 1000.5)	<0.001***
Post-AKI Urine Volume 24h, M (P₂₅-P₇₅)	1416 (870-2170)	1530 (1010-2290)	997.5 (489.2-1728.8)	<0.001***
Post-AKI Urine Volume min 24h, M (P₂₅-P₇₅)	20 (10-30)	20 (10-30)	15 (5-30)	<0.001***
Serum laboratory variables, M(P₂₅-P₇₅)
SCr 1st, (µmol/l)	1.2 (0.9-1.7)	1.1 (0.9-1.4)	1.8 (1.3-3)	<0.001***
SCr max 24h, (µmol/l)	1.3 (1-2)	1.2 (0.9-1.5)	2.3 (1.6-3.5)	<0.001***
SCr min 36h, (µmol/l)	1 (0.8-1.4)	0.9 (0.8-1.1)	1.7 (1.1-2.8)	<0.001***
SCr max 36h, (µmol/l)	1.3 (1-1.9)	1.2 (1-1.4)	2.3 (1.5-3.6)	<0.001***
Post-AKI SCr max 48h, (µmol/l)	1.4 (1.1-2.2)	1.3 (1-1.6)	2.6 (1.8-4)	<0.001***
Hemoglobin min 24h, (g/dL)	9.5 (8-11)	9.5 (8-11.2)	9.4 (8-10.9)	0.006**
Hemoglobin max 24h, (g/dL)	12.1 (10.8-13.5)	12.4 (11.1-13.6)	11.4 (10.2-12.8)	<0.001***
Platelet min, (× 10⁹/L)	165 (116-225)	165 (119.5-227)	165 (107-220)	0.013*
Platelet max 24h, (× 10⁹/L)	198 (148-261)	200 (152-262)	196 (133-260)	0.008**
Bicarbonate min 24h, (mmol/L)	22 (18-24)	22 (19-24)	20 (16-23)	<0.001***
Bicarbonate max 24h, (mmol/L)	24 (22-27)	25 (22-27)	23 (20-26)	<0.001***
Cardiac Troponin T max, (µg/L)	0.11 (0.11-0.15)	0.11 (0.11-0.13)	0.11 (0.11-0.21)	0.195
Post-AKI Cardiac Troponin T 24h, (µg/L)	0.3 (0.3- 0.35)	0.3 (0.3- 0.32)	0.3 (0.3- 0.62)	<0.001***
Prothrombin Time min 24h, (sec)	14.1 (13.4-15.1)	14.1 (13.3-14.9)	14.1 (13.5-16)	<0.001***
Prothrombin Time max 24h, (sec)	15.2 (14-17)	15.1 (13.7-17.2)	15.6 (13.9-18.7)	<0.001***
Total Bilirubin min 24h, (mg/dL)	0.7 (0.6-0.7)	0.7 (0.65-0.7)	0.7 (0.5-1)	<0.001***
Total Bilirubin max 24h, (mg/dL)	0.8 (0.7-0.8)	0.8 (0.75-0.8)	0.8 (0.6-1.2)	<0.001***
Albumin max 24h, (g/dL)	3 (3-3.001)	3 (2.6-3.5)	3 (2.5-3.4)	<0.001***
Albumin min, (g/dL)	3.1 (3-3.1)	3.1 (3-3.1)	3.1 (2.9-3.1)	<0.001***
Lactate min 24h, (mmol/L)	1.5 (1.2-1.8)	1.5 (1.2-1.8)	1.5 (1.2-1.9)	<0.001***
Lactate max 24h, (mmol/L)	2.5 (2.1-3)	2.5 (1.7-3.8)	2.5 (1.6-4.6)	<0.001***
Chloride min 24h, (mmol/L)	103 (100-106)	103 (100-106)	103 (99-107)	0.022*
Base Excess min 24h, (mmol/L)	-4 (-6--2)	-4 (-5--1)	-4 (-8--3)	<0.001***
Interventions
RRT Duration, M (P₂₅-P₇₅)	0.712±3.3	0.264±2.1	1.976±5.22	<0.001***
RRT Times, M (P₂₅-P₇₅)	18.13±9.5	8.159±7.4	46.31±13.5	<0.001***
Mechanical Ventilation, n (%)	1654 (41.5)	1122 (38.1)	532 (51)	<0.001***
Vasopressor Duration, M(P₂₅-P₇₅)	29.66 (22.85-38.25)	26 (10.75-55.06)	42.83 (14.09-93.83)	0.3699
AKI Stage by SCr Criteria, n (%)				<0.001***
0	1183	1146 (38.9)	37 (3.6)
1	1580	1332 (45.2)	248 (23.8)
2	594	288 (9.8)	306 (29.4)
3	632	181 (6.1)	451 (43.3)
AKI Stage, n (%)				<0.001***
1	923 (23.1)	817 (27.7)	106 (11.5)
2	1978 (49.6)	1625 (55.1)	353 (17.8)
3	1088 (27.3)	505 (17.1)	583 (53.6)
Legend: AKI = acute kidney disease; ARDS = acute respiratory distress syndrome; CVD = cardiovascular disease; HR = heart rate; MAP = mean arterial pressure; RR = respiratory rate; RRT = renal replacement therapy; SBP = systolic pressure; SCr = serum creatinine; SOFA Score = sepsis-related organ failure assessment score; SpO2 = peripheral oxygen saturation; T = temperature. p: *< 0.001, <0.01, *: <0.05

Features Selected and model Comparison in Renal Function Recovery

Patients with AKI non-recovery were of more comorbidities, higher AKI stage, total bilirubin, prothrombin time, proteinuria, cardiac troponin T within 24 hours after AKI diagnosis, and higher maximum SCr within 48 hours from diagnosis AKI compared to those diagnosed with those AKI recovery patients (p<0.001). Patients belonging to the AKI non-recovery group had an increased proportion of needing mechanical ventilation and RRT (p<0.001). In addition, more patients with AKI non-recovery had a higher temperature but lower SpO₂ (p<0.001) (Table 1).

The results of logistic regression analysis were outlined in Table 2. Among 3,989 AKI recovery patients, we randomly selected 3,561 patients (90%) for model derivation. As expected, patients who were elderly (OR = 1.022, 95% CI 1.02-1.03), SBP minimum <90mmHg (OR = 0.665, 95% CI 0.55-0.81), lower post-AKI urine volume within 24 hours (OR = 0.999, 95% CI 0.99-0.99), higher SCr minimum within 36 hours (OR = 1.508, 95% CI 1.33-1.73), higher post-AKI troponin T within 24 hours (OR = 1.007, 95% CI 1-1.01), higher lactate maximum with 24 hours (OR = 1.049, 95% CI 1-1.09), or with higher AKI stage (OR = 3.097, 95% CI 2.73-3.52) demonstrated increased odds in failing to recover from AKI.

Table 2

Univariate and multivariate logistic regression analysis for AKI recovery
	Univariate analysis		Multivariate analysis
Variables	OR (95%CI)	P	OR (95%CI)	P
Age	1.004 (0.99-1.01)	0.089	1.022 (1.02-1.03)	<0.001***
Gender	0.939(0.81-1.09)	0.405
Ethnicity	1.051 (1-1.1)	0.037*
Alcohol Abuse	1.121(0.86-1.45)	0.395
SOFA Score	1.174 (1.15-1.2)	<0.001***
CVD	0.573 (0.48-0.68)	<0.001***
ARDS	2.024 (1.72-2.38)	<0.001***
Hypertension	0.761 (0.65-0.89)	0.0004***
Chronic Pulmonary Disease	0.784 (0.65-0.95)	0.0116*
Heart Failure	0.933 (0.73-1.19)	0.586
Organ Dysfunction	3.207 (2.7-3.82)	<0.001***
Sepsis	1.904 (1.56-2.31)	<0.001***
HR min 24h (bpm)	0.966 (0.93-0.99)	0.038*
HR max 24h (bpm)	0.814 (0.78-0.85)	<0.001***
MAP min 24h (mmHg)	0.988 (0.98-0.99)	0.002**
SBP min <90 (mmHg)	0.854 (0.74-0.99)	0.0409*	0.665 (0.55-0.81)	<0.001***
T max >37.3 (℃)	0.631 (0.54-0.74)	<0.001***
SpO2 1st	0.979 (0.96-0.99)	0.002**
Urine Ratio 6h	1.01 (0.98-1.04)	0.545
Proteinuria 24h	1 (1-1.01)	0.001**
Pre-AKI Urine volume 24h	0.99 (0.9-0.99)	<0.001***
Post-AKI Urine volume 24h	0.99 (0.9-0.99)	<0.001***	0.999 (0.99-0.99)	<0.001***
Post-AKI Urine Volume min 24h	1 (0.99-1.01)	0.759
SCr 1st	2.037 (1.88-2.21)	<0.001***
SCr max 24h, (µmol/l)	2.277 (2.1-2.47)	<0.001***
SCr min 36h, (µmol/l)	3.756 (3.33-4.26)	<0.001***	1.508 (1.33-1.73)	<0.001***
SCr max 36h, (µmol/l)	2.549 (2.34-2.78)	<0.001***
Post-AKI SCr max 48h, (µmol/l)	2.467 (2.28-2.67)	<0.001***
Hemoglobin min 24h, (g/dL)	0.966 (0.93-0.99)	0.038*
Hemoglobin max 24h, (g/dL)	0.814 (0.78-0.85)	<0.001***
Platelet min 24h, (× 109/L)	0.999 (0.998-0.999)	0.01**
Platelet max 24h, (× 109/L)	1.308 (1.12-1.54)	0.0009***
Bicarbonate min 24h, (mmol/L)	0.91 (0.9-0.93)	<0.001***
Bicarbonate max 24h, (mmol/L)	0.903 (0.89-0.92)	<0.001***
Cardiac Troponin T max 24h, (µg/L)	1.002 (0.99-1.01)	0.496
Post-AKI Cardiac Troponin T 24h, (µg/L)	1.01 (1-1.02)	0.001**	1.007 (1-1.01)	0.018*
Prothrombin Time min 24h	1.048 (1.03-1.07)	<0.001***
Prothrombin Time max 24h	1.023 (1.01-1.03)	<0.001***
Total Bilirubin min 24h	1.066 (1.04-1.09)	<0.001***
Total Bilirubin max 24h	1.059 (1.04-1.08)	<0.001***
Albumin min 24h	0.676 (0.58-0.79)	<0.001***
Albumin max 24h	0.809 (0.67-0.98)	0.031*
Lactate max 24h	1.085 (1.05-1.12)	<0.001***
RRT Duration	1.196 (1.16-1.24)	<0.001***
Mechanical ventilation	1.729 (1.49-2.01)	<0.001***
AKI Stage	4.004 (3.64-4.42)	<0.001***	3.097 (2.73-3.52)	<0.001***
Legend: AKI = acute kidney disease; ARDS = acute respiratory distress syndrome; CVD = cardiovascular disease; HR = heart rate; MAP = mean arterial pressure; RR = respiratory rate; RRT = renal replacement therapy; SBP = systolic pressure; SCr = serum creatinine; SOFA Score = sepsis-related organ failure assessment score; SpO2 = peripheral oxygen saturation; T = temperature. p: *< 0.001, <0.01, *: <0.05

Moreover, according to the analysis results of each variables’ contribution by XGBoost model, maximum SCr within 48 hours from diagnosis of AKI, AKI stage, Scr minimum within 36 hours, age, post-AKI urine volume within 24 hours, were the top 5 essential predictors for predicting renal function recovery (Fig. 2). Fig. 3 showed the training process of the XGBoost model.

A total of 398 (10%) patients were included in the model validation phase. The discrimination was appraised using AU-ROC and PR in the model development and validation phase. The XGBoost algorithm showed significantly greater discrimination than the traditional logistic regression model, with a higher and more narrowed 95% confidence interval (AU-ROC, 0.92; 95% CI 0.89-0.94 vs. 0.88; 95% CI 0.84-0.92, respectively) (Fig. 4). Table 3 describes the obvious model measures for the two models in identifying the AKI recovery and non-recovery status. When considering sensitivity and precision to predict an independent testing set, the XGBoost performed a more balanced result than the logistic regression.

Table 3

Model performance in development cohort for AKI recovery.
Model Performance	Logistic regression	XGBoost
Accuracy	0.8518	0.8492
Sensitivity	0.9628	0.9122
Specificity	0.5294	0.6667
Precision	0.8563	0.8882
Recall	0.9662	0.9122
AU-ROC (95%CI)	0.8828 (0.84-0.92)	0.9178 (0.89-0.94)

Model Establishment and Comparison in Renal Function Recovery Time

The 2,947 recovery population was randomly split into a training and validation cohort consisting of 2,654 (90%) and 293 (10%) recovery patients. We performed the Pearson correlation analysis, which showed that SCr minimum within 36 hours, AKI Stage, albumin minimum and other 32 variables were associated with recovery duration increases (r =0.32 p < 0.001, r = 0.26 p < 0.001, r = -0.1 p = 0.009, respectively). Therefore, we performed these factors in a multivariate linear regression model, which revealed SCr alone explained 11.09% of the variation, increasing AKI Stage, and minimum albumin explained 12.61% of the variation. The final linear regression model showed the best model performance (R² = 0.2006, RMSE = 4.177) (Table 4).

Table 4

Features selected in the multivariate linear regression model (n = 2,654)
Models	Predictors	Coefficients		R²	Increased R²	p-Value
		B	SE
1	SCr min 36h	1.948	0.107	0.1109		<.001***
2	SCr min 36h	1.549	0.125	0.1261	0.0152	<.001***
	AKI Stage by SCr Criteria	0.538	0.097			<.001***
	Albumin min	-0.48	0.148			<0.001***
3	SCr min 36h	1.374	0.123	0.2006	0.0745	<0.001***
	AKI Stage by SCr Criteria	0.504	0.099			<0.001***
	Albumin min	-0.705	0.220			0.001**
	Post-AKI Urine volume 24h	0.001	0.001			0.001**
	Hemoglobin min 24h	0.054	0.031			0.079
	ARDS	0.471	0.194			0.015*
	RRT Duration	0.277	0.033			<0.001***
	Mechanical Ventilation	1.047	0.165			<0.001***
	Prothrombin Time max 24h	0.049	0.014			0.001**
	Bicarbonate min	0.061	0.016			<0.001***
	Albumin max 24h	0.585	0.280			0.037*
	Prothrombin Time min	-0.073	0.025			0.004**
	Hypertension	-0.301	0.141			0.033*
	Total Bilirubin max 24h	0.015	0.024			0.524
	Ethnicity	0.057	0.044			0.192
Legend: AKI = acute kidney disease; ARDS = acute respiratory distress syndrome; RRT = renal replacement therapy; SCr = serum creatinine; SOFA Score = sepsis-related organ failure assessment score. p: *< 0.001, <0.01, *: <0.05

We also built an overall ExtraTrees model, which contained 2,654 AKI recovered patients who showed better predictive ability (R² = 0.3626, RMSE =2,932). Gender-specific ExtreTrees classifier revealed the higher predictive performance (R² = 0.3411, RMSE =2,978), which showed some similarly essential predictors with linear regression, such as SCr, urine volume, AKI Stage, RRT duration, and mechanical ventilation (Fig. 5). Table 5 compares model performance using R² and RMSE.

Table 5

Model performance in development cohort for AKI recovery time.
Model Performance	R²	RMSE
Linear regression	0.2006	4.177
Extratrees overall	0.3411	2.978
Extratrees gender-specific	0.3623	2.932

Preexisting studies pay more attention to the early predicted occurrence of AKI to reduce adverse patient outcomes, and it remains a complex condition to predict renal function recovery and AKI duration. The timing of renal function recovery is related to prognosis and ESRD [18]. A 2016 study of nearly 17,000 patients demonstrated that the persistence of AKI versus prompt recovery patterns is associated with higher morbidity and mortality [19]. Consequently, predicting the recovery time of AKI may assist in assessing the likelihood of needing RRT and, ultimately, could assist in determining the suitable timing of RRT initiation [20–22].

Few studies have predicted AKI prognosis with effective predictions to support decision-making. Several clinical tools, including prediction models [23, 24], urinary indices [25, 26], novel biomarkers [27, 28], and imaging techniques [29] were canvassed in previous studies to predict renal recovery, namely progression to severe AKI. For example, a recent study enrolling 8,320 critical patients with AKI found a poor performance of SOFA and △Cr for predicting persistent AKI, with the AUC of 0.69 (95%CI 0.66-0.71) and 0.74 (95%CI 0.71-0.77), respectively [23]. Furthermore, the renal angina index has been proposed to detect critically ill children at high risk of persistent AKI, validated as an effective screening tool. Its predictive ability for persistent AKI has been repetitiously identified [30–32]. Additionally, some studies reported that one of the possible methods to achieve reliable prediction is to make use of advanced machine learning technology, which have been tied up in the prevention of AKI [33], such as predicting the AKI development, volume response in patients with oliguria [34], and mortality in AKI patients [35–37].

In the current study, the machine learning algorithm achieved better predicting outcomes than the conventional logistic regression. XGBoost showed the best performance in predicting renal function recovery and ExtraTrees for predicting the time of renal function recovery. Our study demonstrated that the timing of SCr examination might provide a significant indication for the possibility of renal function recovery. Furthermore, the higher AKI stage was a strong predictor of AKI non-recovery, which further supported that patients who meet severe AKI stage are at high risk of RRT or death. Cardiac markers provide vital signals for heart disease, which play an essential role in renal function recovery. Similar to our finding, David Song showed that cardiac troponin T is common in acute coronary heart disease patients and can worsen renal function, resulting in ESRD [38]. Among patient-related variables, age, hypertension, sepsis, and higher SOFA scores were identified as persistent AKI predictors. They may be related to decreased glomerular reserve and delayed or incomplete renal function recovery [39]. A close relationship between ARDS and AKI development was reported in recent studies [40, 41]. Geri, G, and colleagues demonstrated that mechanical ventilation was associated with worsening renal function in a large cohort of ICU patients [42]. Our results further demonstrated that comorbidities, including hypertension, ARDS, and sepsis, could contribute to the prediction of longer renal function recovery time.

Limitations of this study should be noted. Most notably, the MIMIC Ⅲ database used in the present study only contains critically ill patients between 2001 and 2012. Thus, changes in treatment strategies for critically ill patients, including time of RRT, methods of mechanical ventilation, and vasopressor support strategies, may influence AKI prognosis. Secondly, we only validated models by internal dataset. Finally, we are limited in that we do not have baseline SCr value, which is an issue for both AKI research and bedside clinicians.

In this large-cohort retrospective study, we developed a novel machine learning assessment model to predict renal function recovery and a model for renal function recovery time with high practicability and interpretability. Our algorithm, which includes demographics, comorbidities, laboratories, vital signs, and clinical interventions, can assist in identifying high-risk patients, guiding treatment decisions, and improving prognosis. Implementing such predicted tools in real-time is increasingly common and requires no additional testing of novel AKI biomarkers. In the future, we will make full use of this algorithm with early renal function care to improve outcomes in AKI patients.

Acknowledgements

The authors thank all the participants for their contributions.

Authors’ contributions

Yunwei Lu and Xiujuan Zhao drafted the initial manuscript with oversight by Fengxue Zhu and all other authors critically reviewed and revised the manuscript. Wenfei Xie and Chong Zhang collected all data and completed figure making. Shu Li, Fuzheng Guo, Haiyan Xue, Lilei Jiang, and Zhenzhou Wang assisted with interpretation of the results. All authors reviewed and approved the final manuscript.

Funding

None.

Availability of data and materials

The clinical data used to support the findings of this study were supplied by the MIMIC-III database. After completing the National Institutes of Health’s web-based course known as Protecting Human Research, participants can apply for permission to access the database.

Ethics approval and consent to participate

This research was approved by Peking University People’s Hospital's institutional review board.

Consent for publication

Written informed consents were signed by all the patients.

Competing interests

The authors declare that they have no competing interests

Al-Jaghbeer M, Dealmeida D, Bilderback A, et al: clinical decision support for in-hospital AKI. J Am Soc Nephrol 2018; 29: 654–60.
Hoste EA, Bagshaw SM, Bellomo R, et al. Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study. Intensive Care Med 2015; 41: 1411–23.
Chertow GM, Levy EM, Hammermeister KE, et al: Independent association between acute renal failure and mortality following cardiac surgery. Am J Med 1998; 104:343–348
Chertow GM, Burdick E, Honour M, et al: Acute kidney injury, mortality, length of stay, and costs in hospitalized patients. J Am Soc Nephrol 2005; 16:3365–3370.
Hobson C, Ozrazgat-Baslanti T, Kuxhausen A, et al: Cost and mortality associated with postoperative acute kidney injury. Ann Surg 2015; 261:1207–1214.
Hickson LJ, Chaudhary S, Williams AW, et al: Predictors of outpatient kidney function recovery among patients who initiate hemodialysis in the hospital. Am J Kidney Dis. 2015;65:592–602.
Craven AM, Hawley CM, McDonald SP, et al: Predictors of renal recovery in Australian and New Zealand end-stage renal failure patients treated with peritoneal dialysis. Perit Dial Int. 2007;27:184–191.
Srisawat N, Wen X, Lee M, et al: Urinary biomarkers and renal recovery in critically ill patients with renal support. Clin J Am Soc Nephrol. 2011;6:1815–1823.
Johnson A E W, Pollard T J , Shen L, et al: MIMIC-III, a freely accessible critical care database. Sci Data. 2016 May 24;3:160035.
Levin A, Stevens PE, Bilous RW, et al: Kidney disease: improving global outcomes (KDIGO) CKD work group. KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2013;3:1–150.
Hoste EA, Bagshaw SM, Bellomo R, et al: Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study. Intensive Care Med2015;41:1411-1423.
Porter CJ, Juurlink I, Bisset LH, et al: A real-time electronic alert to improve detection of acute kidney injury in a large teaching hospital. Nephrol Dial Transplant 2014;29:1888-1893.
Heung M, Stefck DE, Zivin K, et al: Acute kidney injury recovery pattern and subsequent risk of CKD: an analysis of veterans health administration data. Am J Kidney Dis. 2016 May;67(5):742-52.
Pannu N, James M, Hemmelgarn B, et al: Association between AKI, recovery of renal function, and long-term outcomes after hospital discharge. Clin. J. Am. Soc. Nephrol. 8, 194–202 (2013).
Sheridan RP, Wang WM, Liaw A, et al: Extreme gradient boosting as a method for quantitative structure-activity relationships. J Chem Inf Model. 2016;56(12):2353–2360.
Qian M Zhou, Lu Zhe, Russell J Brooke, et al: A relationship between the incremental values of area under the ROC curve and of area under the precision-recall curve. Diagn Progn Res. 2021 Jul 14;5(1):13.
Smith GB, Prytherch DR, Meredith P, et al: The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013; 84:465–470
Perinel S, Vincent F, Lautrette A, et al: Transient and persistent acute kidney injury and the risk of hospital mortality in critically ill patients: results of a multicenter cohort study. Crit Care Med.2015;43:e269-75.
Kellum JA, Florentina E Sileanu, Azra Bihorac, et al: Recovery after acute kidney injury. Am J Respir Crit Care Med. 2017 Mar 15;195(6):784-791.
Gaudry S, Hajage D, Schortgen F, et al: Initiation strategies for renal-replacement therapy in the intensive care unit. N Engl J Med. 2016;375:122-33.
Barbar SD, Clere-Jehl R, Bourredjem A, et al: Timing of renal replacement therapy in patients with acute kidney injury and sepsis. N Engl J Med. 2018;379:1431-42.
Zarbock A, Kellum JA, Schmidt C, et al: Effect of early vs delayed initiation of renal replacement therapy on mortality in critically ill patients with acute kidney injury: the ELAIN randomized clinical trial. JAMA. 2016;315:2190-9.
Matsuura R, Iwagami M, Moriya H, et al: A simple scoring method for predicting the low risk of persistent acute kidney injury in critically ill adult patients. Sci. Rep. 2020 Mar 31;10(1):5726.
Bhatraju PK, Zelnick LR, Katz R, et al: A prediction model for severe AKI in critically ill adults that incorporates clinical and biomarker data. Clin. J. Am. Soc. Nephrol. 14, 506–514 (2019).
Vanmassenhove J, Glorieux G, Hoste E, et al: Urinary output and fractional excretion of sodium and urea as indicators of transient versus intrinsic acute kidney injury during early sepsis. Crit. Care.2013 Oct 13;17(5):R234.
Darmon M, Vincent F, Dellamonica J, et al: Diagnostic performance of fractional excretion of urea in the evaluation of critically ill patients with acute kidney injury: A multicenter cohort study. Crit. Care. 2011 Jul 27;15(4):R178.
Garnier F, Daubin D, Larcher R, et al: Reversibility of acute kidney injury in medical ICU patients: Predictability performance of urinary tissue inhibitor of metalloproteinase-2 x insulin-like growth factor-binding protein 7 and renal resistive index. Crit. Care Med. 48, e277–e284 (2020).
Dewitte A, Joannès-Boyau O, Sidobre C, et al: Kinetic eGFR and novel AKI biomarkers to predict renal recovery. Clin. J. Am. Soc. Nephrol. 10, 1900–1910 (2015).
Darmon M, Bourmaud A, Reynaud M, et al: Performance of doppler-based resistive index and semi-quantitative renal perfusion in predicting persistent AKI: Results of a prospective multicenter study. Intensive Care Med. 44, 1904–1913 (2018).
Basu R K, Yu Wang, Wong HR, et al: Incorporation of biomarkers with the renal angina index for prediction of severe AKI in critically ill children. Clin. J. Am. Soc. Nephrol. 9, 654–662 (2014).
Basu R K, Kaddourah A, Goldstein SL, et al: Assessment of a renal angina index for prediction of severe acute kidney injury in critically ill children: a multicentre, multinational, prospective observational study. Lancet. Child Adolesc. Heal. 2, 112–120 (2018).
Basu RK, Zappitelli M, Brunner L, et al: Derivation and validation of the renal angina index to improve the prediction of acute kidney injury in critically ill children. Kidney Int. 85, 659–67 (2014).
Koyner JL, Carey KA, Edelson DP, et al: The development of a machine learning inpatient acute kidney injury prediction model. Crit. Care Med. 46, 1070–1077 (2018).
Zhang Z, Ho KM, Hong Y, et al: Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit. Care 23, 112 (2019).
Lin K, Hu Y, Kong G, et al: Predicting in-hospital mortality of patients with acute kidney injury in the ICU using random forest model. Int. J. Med. Inform. 125, 55–61 (2019).
Huang H, Liu Y, Wu M, et al: Development and validation of a risk stratification model for predicting the mortality of acute kidney injury in critical care patients. Ann. Transl. Med. 2021 Feb;9(4):323.
Kang MW, Kim J, Kim DK, et al: Machine learning algorithm to predict mortality in patients undergoing continuous renal replacement therapy. Crit. Care. 2020 Feb 6;24(1):42.
David Song, Janak R de Zoysa, Alvin Ng, et al: Troponins in Acute Kidney Injury. Renal Failure. 2012;34(1):35-9.
Forni LG, M Darmon, M Ostermann, et al: Renal recovery after acute kidney injury.Intensive Care Med. 2017 Jun;43(6):855-866.
Darmon M, Clec'h C, Adrie C, et al: Acute respiratory distress syndrome and risk of AKI among critically ill patients. Clin. J. Am. Soc. Nephrol. 2014 Aug 7;9(8):1347-53.
Clemens MS, Stewart IJ, Sosnov JA, et al: Reciprocal risk of acute kidney injury and acute respiratory distress syndrome in critically ill burn patients. Crit. Care Med. 2016 Oct;44(10):e915-22.
Geri G, Ferrer L, Tran N, et al: Cardio-pulmonary-renal interactions in ICU patients. Role of mechanical ventilation, venous congestion and perfusion deficit on worsening of renal function: Insights from the MIMIC-III database. J. Crit. Care. 2021 Aug;64:100-107.

Download PDF

Version 1

posted

You are reading this latest preprint version

Comparison of Machine Learning Methods and Conventional Regression for Predicting Renal Function Recovery and Recovery Time among AKI Patients in ICU

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Sources of Data

Selecting an AKI Cohort

Data Collection and Definition

Statistical Analysis

Renal Function Recovery Model Development and Validation

Recovery Time Model Derivation and Validation

Evaluating Performance of Models

Results

Demographics, clinical characteristics, and AKI metric measurements

Features Selected and model Comparison in Renal Function Recovery

Model Establishment and Comparison in Renal Function Recovery Time

Discussion

Conclusion

Declarations

Acknowledgements

Authors’ contributions

Funding

Availability of data and materials

Ethics approval and consent to participate

Consent for publication

Competing interests

References

Status:

Version 1