A Validation Study Comparing Existing Prediction Models of Acute Kidney Injury in Patients with Acute Heart Failure: A Multi-Institution Database Study in Taiwan

doi:10.21203/rs.3.rs-97827/v1

Download PDF

Research

A Validation Study Comparing Existing Prediction Models of Acute Kidney Injury in Patients with Acute Heart Failure: A Multi-Institution Database Study in Taiwan

https://doi.org/10.21203/rs.3.rs-97827/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background:

Acute kidney injury (AKI) is a common complication in hospitalized acute heart failure (AHF) patients and is associated with prolonged hospitalization, increased readmission rates, and mortality. The aim of this study was to externally validate existing prediction models of AKI in patients with AHF.

Methods:

A total of 10,364 patients hospitalized for acute heart failure (AHF) between 2008 and 2018 were extracted from the Chang Gung Research Database and analyzed. The primary outcome of interest was AKI, defined according to the KDIGO definition. We also extended the existing prediction models to predict AKI stage 3 and dialysis. The area under the receiver operating characteristic (AUC) curve was used to assess the discrimination performance of each prediction model.

Results:

Five existing prediction models were externally validated, with the AUCs for AKI prediction ranging from 0.543 to 0.73. These prediction models also performed well in serious AKI event prediction, with AUCs of 0.565–0.858 for predicting AKI stage 3 and AUCs of 0.539–0.845 for predicting dialysis within 7 days. Among the five models, the Forman risk score and the prediction model reported by Wang et al. showed the most favorable discrimination and calibration performance. The Forman risk score had AUCs for discriminating AKI, AKI stage 3, and dialysis within 7 days of 0.696, 0.829, and 0.817, respectively. The Wang et al. model had AUCs for discriminating AKI, AKI stage 3, and dialysis within 7 days of 0.73, 0.858, and 0.845, respectively.

Conclusion:

The Forman risk score and the Wang et al. prediction model are simple and accurate tools for predicting AKI and serious AKI events in patients with AHF. They can aid clinicians in evaluating the risk of AKI in these patients and in planning and initiating adequate disease management in a timely manner.

Critical Care & Emergency Medicine

acute kidney injury

acute heart failure

predicting model

Acute heart failure (AHF) with abrupt onset dyspnea, sensation of suffocation, and sometimes with pink frothy expectoration is a leading cause of hospitalization. Acute kidney injury (AKI), previously termed worsening renal function (WRF), is a common complication among patients with AHF, with an incidence of 21–45%.[1–3] Previous studies have revealed that the development of AKI in patients with AHF results in longer hospital stays, higher readmission rates, and increased short- and long-term mortality.[4–8] Smith et al. further reported that even a slightly increased creatinine level (≥ 0.2 mg/dL) puts patients with AHF at a higher risk of mortality.[9] Therefore, the early recognition and identification of patients who are at high risk of developing AKI is essential for treating patients with AHF.

For these reasons, many studies have focused on identifying relevant risk factors, with some having derived AKI prediction models in patients with AHF.[1, 10–14] The Forman risk score, first reported in 2004, is the earliest and arguably best known prediction model.[10] Following the subsequent Basel risk score, prediction models were also proposed by Wang et al. and Zhou et al. between 2011 and 2016.[1, 12, 13] However, the definition of AKI or WRF varies in these studies due to the AKI classification changing from the RIFLE (Risk, Injury, Failure, Loss of kidney function, and End-stage kidney disease) classification and AKIN (Acute Kidney Injury Network) criteria to the KDIGO (Kidney Disease Improving Global Outcome) guidelines in past few years.[15–17] Considering the importance of early identification, prevention, and intervention of AKI in patients with AHF, revalidating the performance and discrimination of these prediction models according to the current AKI definition is necessary. Therefore, the aim of this study was to externally validate the existing prediction models for AKI in patients with AHF based on the KDIGO Clinical Practice Guidelines for Acute Kidney Injury.

Data source

This study was based on the electronic medical records of the Chang Gung Research Database (CGRD) from the Chang Gung Medical Foundation. The database comprises the data of the nationwide Chang Gung Memorial Hospital system, which is the largest health care system of its kind in Taiwan, comprising two medical centers, two regional hospitals, and three district hospitals. The CGRD contains clinical epidemiological data, laboratory data, inpatient and outpatient records, emergency medical records, pathology reports, and disease category data. The overall coverage rates of the CGRD are approximately 20% for outpatients and 12% for inpatients for the entire Taiwan population. More detailed information about the CGRD has been reported in previous studies.[18, 19] Its disease diagnoses are coded using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) for records before 2016 and International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) for those thereafter. This study was approved by the Institutional Review Board of Chang Gung Memorial Hospital (approval number: CGMHIRB No. 202000915B0).

Study population

We analyzed the records of patients who had emergency department visits and were subsequently admitted due to acute heart failure (ICD-9-CM diagnostic code: 428; ICD-10-CM diagnostic code: I50) between January 1, 2008, and December 31, 2018, in all 7 Chang Gung Memorial Hospital branches located in Linkou, Taipei, Taoyuan, Keelung, Yunlin, Chiayi, and Kaohsiung, which span northern to southern Taiwan. When a patient had multiple AHF episodes, the first episode of AHF hospitalization between 2008 and 2018 was selected as the index hospitalization. The admission date of the index AHF hospitalization was used as the index date.

Patients without sufficient data for AKI judgment were not included, such as those without baseline creatinine data or a second creatinine examination within 7 days of admission. The remaining patients were excluded if they met any of the following criteria: (1) were aged younger than 18 years, (2) had end-stage renal disease or were receiving maintenance dialysis, (3) had follow-up of less than 24 hours, (4) received extracorporeal membrane oxygenation during the index admission, (5) anticipated cardiac transplantation, (6) received a nephrotoxic agent (including contrast agents, nonsteroidal anti-inflammatory drugs, aminoglycoside, and vancomycin) within 4 weeks of admission or during the index admission, (7) had obstructive nephropathy, and (8) had acute coronary syndrome with inotropic agents used during the index admission. After exclusion, 10,364 patients remained eligible for the study, of whom 1,483 (14.3%) had AKI (Fig. 1).

Existing prediction models and covariates

A review of previous studies found five prediction models or risk factor studies for AKI prediction in patients with AHF. All five were externally validated in this study.

The prediction models were as follows: the Forman risk score, reported in 2004 and based on four prediction factors, including underlying diseases as well as clinical and laboratory parameters [10]; the Basel risk score, reported in 2011 and using chronic kidney disease, bicarbonate level, and outpatient diuretics treatment as indices [1]; a prediction model reported by Wang et al. in 2013 based on a data analysis of 1,709 patients and using 8 prediction factors [12]; the prediction model reported by Zhou et al. in 2016 that combines clinical parameters and novel urine biomarkers for AKI prediction in AHF [13], though we were unable to include the NT-proBNP and urine biomarkers used by Zhou et al. because the CGRD did not contain this data; and the predictors study by Verdiani et al. in 2010 for AKI in hospitalized AHF patients.[11]

The study populations, publication years, heart failure criteria, and full list of predictors of the prediction models are summarized in Table 1.

Table 1

Existing prediction models for acute kidney injury in patients with acute heart failure
Prediction model	Prediction factors	Population and patient number	Study years	Heart failure criteria	Specificity and sensitivity	AUC
2004 Forman risk score	CHF, DM, SBP level, creatinine	1004 patients, U.S.	1997–1998	ICD-9-CM†	Sen 81%; Spe 62%	AUC 0.65*
2010 Verdiani	age, CKD, heart rate, creatinine, CCB use, digoxin use	394 patients, Italy	2002–2008	Standard Framingham criteria	-	-
2011 Basel risk score	CKD, bicarbonate outpatient, diuretics	575 patients, Switzerland	2001–2002, 2006–2010	2008 ESC guidelines	-	AUC 0.71 (95%CI 0.63–0.79)
2013 Wang	age, heart functional class, admission times for acute heart failure, SBP level, creatinine, sodium, proteinuria, IV furosemide use	1709 patients, China	2004–2011	ICD-10-CM‡, 2012 ESC guidelines	Sen 70.0%; Spe 70.6%	AUC 0.76 (95% CI: 0.73–0.79)
2016 Zhou	age, sex, CKD, albumin, NT-proBNP, uNGAL, uAGT	507 patients, China	2011–2014	2005 ESC guidelines	-	AUC 0.765 for clinical model alone AUC 0.874 for prediction model
CCB: calcium channel blocker; CHF: congestive heart failure; CKD: chronic kidney disease; DM: diabetes mellitus; IV: intravenous therapy; SBP: systolic blood pressure.
*AUC is not calculated in original article. AUC 0.65 was documented by 2011 Basel risk score and 2013 Wang study.
†Heart failure was identified using ICD-9-CM codes 428.0, 428.1, 402.01, 402.11, 402.91, 404.01, 404.03, 404.11, 404.13, 404.91, and 404.93.
‡Heart failure was identified using ICD-10-CM codes I50.102, I50.106, I50.107, I50.902, I50.903, I50.904, I50.908, I50.910, and I50.911.

Outcome definition

The primary outcome was the development of AKI. The first record of creatinine level during emergency department admission was used as the baseline creatinine level, and AKI was defined as an increase in serum creatinine by 0.3 mg/dL or a 50% increase in serum creatinine within 7 days, in accordance with the KDIGO Clinical Practice Guidelines for Acute Kidney Injury.[17] This study also validated the performance of existing prediction models in predicting serious AKI events, including AKI stage 3 and dialysis. AKI stage 3 was defined as a ≥ 200% increase in serum creatinine, a serum creatinine of ≥ 4 mg/dl, or the initiation of dialysis within 7 days of study enrollment, according to the KDIGO guidelines. Urine output was not used to define AKI because this data was not complete in the CGRD.

The development of major adverse kidney events (MAKEs) on or after the 8th day following the index date was also examined. MAKEs were defined as the composite of chronic kidney disease (an estimated Glomerular filtration rate [eGFR] decline of > 25% from baseline), ESRD requiring chronic renal replacement therapy, and all-cause mortality. We assessed MAKEs within 1 year of AKI diagnosis and from the index date to the final visit date, the date of death, the date of event occurrence, or December 31, 2018, whichever came first. Only patients with a follow-up duration of > 7 days were included in the MAKEs analysis.

Statistical analysis

Due to the presence of a substantial amount of missing data, we imputed the data using the single expectation maximization (EM) method for the primary analysis in this study. To test the robustness of the results, only patients with complete data were retained and used in the sensitivity analysis. Because the missing rate of bicarbonate level data was substantially high (59%), we obtained the complete data with and without including bicarbonate level.

The patient characteristics of the AKI and non-AKI groups were compared using the independent sample t-test for continuous variables and the chi-square test for categorical variables. The discrimination ability of individual scores in predicting an outcome of interest (i.e., AKI, AKI stage 3, or dialysis) in patients with AHF was determined using the area under the receiver operating characteristic curve (AUC). Optimal cutoff points were determined using the Youden index, and the corresponding sensitivity and specificity were calculated. The AUCs among the existing prediction models were pairwise compared using the DeLong test. In addition, the calibration performance of each score was assessed using the Hosmer–Lemeshow (HL) goodness-of-fit test, with smaller statistics (chi-square) indicating a smaller discrepancy between the predicted probability and observed AKI event for the prediction models. The patients were divided into two subgroups according to the optimal cutoff of each score. The risk of MAKEs on or after the 8th day following the index date was compared between the higher and lower cutoffs of each score using the Cox proportional hazards model.

All tests were 2-tailed and P < 0.05 was considered statistically significant. No adjustment of multiple testing (multiplicity) was made in this study. Data analyses were conducted using SPSS 25 (IBM SPSS Inc., Chicago, Illinois).

Baseline characteristics

The patients’ characteristics at baseline are presented in Table 2. A total of 10,364 patients were included in the analysis, of whom 1,483 (14.3%) developed AKI. The mean age and sex distribution were similar in the AKI and non-AKI groups. Of the total patient population, 42.2% had been diagnosed with congestive heart failure, 36.8% with diabetes mellitus, 46.0% with chronic kidney disease, and 56.9% with hypertension. The AKI group showed a significantly higher prevalence of diabetes mellitus (43.2%), chronic kidney disease (56.1%), and hypertension (61.9%). A total of 1,581 patients of the total population exhibited severe heart failure symptoms and were categorized as New York Heart Association (NYHA) functional class IV. The AKI group also had a significantly higher percentage of patients categorized as NYHA functional class IV (22.3%) than the non-AKI group did (14.1%).

Table 2

Baseline characteristics of patients with and without AKI
Variable	Valid N	Total (n = 10,364)	AKI (n = 1,483)	Non-AKI (n = 8,881)	P
Baseline characteristics
Age, years	10,364	72.3 ± 14.5	71.6 ± 14.2	72.4 ± 14.5	0.073
Male	10,364	5,683 (54.8)	791 (53.3)	4,892 (55.1)	0.211
Previous diagnosis of CHF	10,364	4,378 (42.2)	571 (38.5)	3,807 (42.9)	0.002
Underlying diseases
Diabetes mellitus	10,364	3,819 (36.8)	641 (43.2)	3,178 (35.8)	< 0.001
Chronic kidney disease	10,364	4,772 (46.0)	832 (56.1)	3,940 (44.4)	< 0.001
Hypertension	10,364	5,898 (56.9)	918 (61.9)	4,980 (56.1)	< 0.001
Heart function
NYHA functional class IV	10,364	1,581 (15.3)	330 (22.3)	1,251 (14.1)	< 0.001
LVEF group	9,985				0.471
< 40% (Reduced)		3,617 (34.9)	507 (34.2)	3,110 (35.0)
40–54%		2,372 (22.9)	346 (23.3)	2,026 (22.8)
≥ 55% (Preserved)		3,996 (38.6)	566 (38.2)	3,430 (38.6)
Unknown		379 (3.7)	64 (4.3)	315 (3.5)
Vital signs
SBP, mmHg	10,150	140.1 ± 31.4	148.2 ± 34.8	138.7 ± 30.6	< 0.001
DBP, mmHg	10,151	81.4 ± 19.7	82.8 ± 20.8	81.1 ± 19.5	0.002
Heart rate, beat/min	10,023	91.0 ± 23.9	92.8 ± 23.9	90.7 ± 23.9	0.001
Baseline lab data
Hemoglobin, g/dL	10,364	11.8 ± 2.6	10.6 ± 2.6	12.0 ± 2.6	< 0.001
Platelets, 1000/uL	10,359	207.8 ± 87.9	205.6 ± 87.1	208.1 ± 88.0	0.297
Lymphocyte, %	10,345	18.3 ± 10.3	16.1 ± 9.8	18.7 ± 10.3	< 0.001
BUN, mg/dL	7,831	36.0 ± 27.1	56.5 ± 38.6	32.1 ± 22.4	< 0.001
Creatinine, mg/dL	10,364	2.0 ± 1.9	3.9 ± 3.4	1.7 ± 1.2	< 0.001
Bicarbonate, mmol/L	4,236	23.9 ± 6.7	21.0 ± 6.4	24.8 ± 6.5	< 0.001
Sodium, mg/dL	10,248	136.7 ± 5.6	136.4 ± 5.8	136.8 ± 5.5	0.011
Potassium, mg/dL	10,360	4.1 ± 0.7	4.3 ± 0.9	4.0 ± 0.7	< 0.001
Albumin, mg/dL	6,770	3.4 ± 0.5	3.3 ± 0.6	3.4 ± 0.5	< 0.001
Proteinuria (U/A dipstick), mg/dL	6,639				< 0.001
Negative (0–4)		2,384 (23.0)	188 (12.7)	2,196 (24.7)
Trace (5–29)		698 (6.7)	75 (5.1)	623 (7.0)
≥ 1+ (≥ 30)		3,557 (34.3)	833 (56.2)	2,724 (30.7)
Unknown		3,725 (35.9)	387 (26.1)	3,338 (37.6)
BNP, pg/mL	7,137	923 [447, 1815]	1226 [603, 2430]	870 [429, 1700]	< 0.001
Lactic acid, mg/dL	1,692	17.8 [11.9, 32.5]	20.9 [12.0, 46.0]	17.3 [11.9, 29.0]	< 0.001
pH	3,699	7.39 ± 0.11	7.36 ± 0.12	7.40 ± 0.10	< 0.001
Medication treatment
Digoxin	10,364	1,031 (9.9)	94 (6.3)	937 (10.6)	< 0.001
Calcium channel blocker	10,364	1,937 (18.7)	396 (26.7)	1,541 (17.4)	< 0.001
Beta-blocker	10,364	3,653 (35.2)	578 (39.0)	3,075 (34.6)	0.001
Loop-diuretics
Furosemide dosage, mg/ml	10,364	50 [10, 80]	60 [20, 100]	50 [10, 80]	< 0.001
Out-patient loop diuretics or spironolactone use	10,364	5,460 (52.7)	794 (53.5)	4,666 (52.5)	0.475
AKI, acute kidney injury; BNP, B-type natriuretic peptide; BUN, blood urea nitrogen; CHF: congestive heart failure; DBP, diastolic blood pressure; LVEF, left ventricular ejection fraction; NYHA, New York Heart Association; SBP, systolic blood pressure.
Data are given as frequency (percentage), mean ± standard deviation or median (25th, 75th percentiles).

Regarding the clinical parameters, the AKI group patients had significantly higher systemic blood pressure upon admission. The AKI group also exhibited significantly lower hemoglobin, blood urea nitrogen, serum albumin, and bicarbonate levels as well as higher creatinine, potassium, lactic acid, lymphocyte percentage, and BNP levels. A higher percentage of AKI group patients showed positive proteinuria results via dipsticks test. The AKI group received higher dosages of loop diuretics during their AHF admission period; however, there was no significant difference between the groups in outpatient diuretics treatment strategy. The AKI group was more likely to receive calcium channel blockers but less likely to use digoxin during admission (Table 2).

Validation of existing prediction models for AKI

The performance of predicting AKI events in the patients with AHF was externally validated for each existing prediction, as summarized in Table 3. The AUC discrimination ability was highest for the Wang et al. model (AUC = 73%), followed by the Forman risk score (69.6%), Basel risk score (59.7%), Verdiani et al. model (58.8%), and Zhou et al. model (54.3%) (Fig. 2A). Regarding calibration, the HL chi-square statistics were the smallest for the Wang et al. model, followed by the Forman risk score, Zhou et al. model, Basel risk score, and Verdiani et al. model (Table 3). The pairwise comparison results for the AUCs showed that all of the AUCs differed significantly between any two prediction models, except for the Basel risk score and Verdiani et al. model (Table 4).

Table 3

Prediction model performance in discrimination and calibration outcomes of interest
Outcome/risk score	AUC (95% CI)^a	Cutoff^b	Sensitivity (95% CI)	Specificity (95% CI)	χ² of HL test^c
AKI
2004 Forman risk score	69.6 (68.1–71.1)	3	68.31 (65.9–70.7)	65.38 (64.4–66.4)	36.2
2010 Verdiani	58.8 (57.3–60.4)	8	70.47 (68.1–72.8)	45.86 (44.8–46.9)	168.4
2011 Basel risk score	59.7 (58.1–61.2)	2	50.98 (48.4–53.6)	63.25 (62.2–64.3)	78.0
2013 Wang	73 (71.5–74.4)	12	59.88 (57.3–62.4)	77.22 (76.3–78.1)	35.4
2016 Zhou	54.3 (52.8–55.9)	10	56.10 (53.5–58.6)	55.64 (54.6–56.7)	68.1
AKI stage 3
2004 Forman risk score	82.9 (81.6–84.2)	3	91.82 (89.8–93.6)	65.34 (64.4–66.3)	81.9
2010 Verdiani	61.6 (59.8–63.4)	8	77.65 (74.7–80.4)	45.46 (44.5–46.5)	328.5
2011 Basel risk score	65.1 (63.2–67.0)	2	59.91 (56.6–63.2)	63.14 (62.2–64.1)	63.9
2013 Wang	85.8 (84.6–86.9)	12	83.41 (80.8–85.8)	76.97 (76.1–77.8)	46.6
2016 Zhou	56.5 (54.6–58.4)	10	64.17 (60.9–67.4)	55.61 (54.6–56.6)	139.3
Dialysis within 7 days
2004 Forman risk score	81.7 (79.9–83.5)	3	94.15 (91.2–96.3)	62.52 (61.6–63.5)	64.3
2010 Verdiani	58.2 (55.4–61.0)	7	81.89 (77.5–85.7)	38.67 (37.7–39.6)	192.3
2011 Basel risk score	62.3 (59.2–65.4)	2	57.66 (52.4–62.8)	61.89 (60.9–62.8)	41.1
2013 Wang	84.5 (82.9–86.0)	12	82.73 (78.4–86.5)	73.87 (73.0–74.7)	40.8
2016 Zhou	53.9 (51.0–56.8)	10	61.28 (56.0–66.3)	54.50 (53.5–55.5)	66.5
AKI, acute kidney injury; AUC, area under the receiver operating characteristic curve; CI, confidence interval; HL, Hosmer–Lemeshow.
a: Larger numbers indicate better performance;
b: Determined using the Youdex index;
c: Lower numbers indicate better performance.

Table 4

Pairwise comparisons of area under the receiver operating characteristic curve between the prediction models
	Difference in AUC (95% CI) (Column vs. Row) †
Outcome/score	2004 Forman risk score	2010 Verdiani	2011 Basel risk score	2013 Wang
AKI
2004 Forman risk score	-	-	-	-
2010 Verdiani	10.74 (9.06, 12.42)*	-	-	-
2011 Basel risk score	9.93 (8.29, 11.57)*	-0.81 (-2.24, 0.61)	-	-
2013 Wang	-3.37 (-4.49, -2.25)*	-14.11 (-15.79, -12.44)*	-13.30 (-15.09, -11.51)*	-
2016 Zhou	15.25 (13.52, 16.99)*	4.51 (3.46, 5.57)*	5.33 (4.14, 6.52)*	18.63 (16.88, 20.37)*
AKI stage 3
2004 Forman risk score	-	-	-	-
2010 Verdiani	21.28 (19.37, 23.19)*	-	-	-
2011 Basel risk score	17.82 (15.89, 19.75)*	-3.46 (-5.19, -1.74)*	-	-
2013 Wang	-2.86 (-3.96, -1.76)*	-24.14 (-26.03, -22.25)*	-20.68 (-22.73, -18.64)*	-
2016 Zhou	26.38 (24.45, 28.31)*	5.10 (3.84, 6.36)*	8.56 (7.11, 10.02)*	29.25 (27.31, 31.18)*
Dialysis within 7 days
2004 Forman risk score	-	-	-	-
2010 Verdiani	23.52 (20.67, 26.37)*	-	-	-
2011 Basel risk score	19.40 (16.42, 22.38)*	-4.12 (-6.69, -1.54)*	-	-
2013 Wang	-2.74 (-4.44, -1.03)*	-26.25 (-29.06, -23.45)*	-22.13 (-25.26, -19.01)*	-
2016 Zhou	27.83 (24.98, 30.67)*	4.31 (2.39, 6.22)*	8.43 (6.23, 10.62)*	30.56 (27.74, 33.38)*
* indicates P < 0.05;
† DeLong’s test.

Extension of models for predicting AKI stage 3 and dialysis

We extended the scores of the prediction models to examine their ability to predict AKI stage 3 and dialysis. The results showed that the Wang et al. study and Forman risk score demonstrated satisfactory discrimination performance (AUC = 85.8% and 82.9%, respectively) and relatively low HL chi-square statistics (Table 3 and Fig. 2B). Similar to the results for predicting AKI stage 3, the discrimination performance of the Wang et al. model (AUC = 84.5%) and Forman risk score (81.7%) was satisfactory and with relatively low HL chi-square statistics (Table 3 and Fig. 2C). In addition, all of the AUC pairwise comparisons differed significantly in predicting AKI stage 3 or dialysis.

Extension of models for predicting MAKEs

We next analyzed MAKEs within 1 year of AKI diagnosis and MAKEs from the index day to the end of follow-up. The patients were separated into two groups according to cutoff value determined by the Youden index in predicting AKI and AKI stage 3. The results showed that the higher group had a significantly greater risk of MAKEs than did the lower group in all five models’ risk scores, the hazard ratios of which ranged from 1.56 to 1.76 for 1-year follow-up and 1.52 to 1.70 for the index date to the end of follow-up (Figs. 3A and 3B).

In the present study, we externally validated five existing models for predicting the risk of AKI in patients with AHF. The Forman risk score and Wang et al. model showed superior discrimination and calibration performance compared with the three other models.

The development of AKI in patients with AHF leads to prolonged hospitalization, increased readmission rates, and increased short- and long-term all-cause mortality and cardiovascular mortality. Coexisting AKI and AHF also causes higher health care costs for patients with heart failure.[4–6] In the past two decades, many studies have focused on the early identification of patients with AHF who are at high risk of AKI development in order to initiate intervention earlier and improve their clinical outcomes. Some of these studies have used clinical parameters as risk predictors, and the others have introduced or added novel urine biomarker for AKI prediction.[1, 10–13] However, the widely varying definition and classification of AKI (or WRF in some studies) as well as differences in the observed time-at-risk and the heterogeneity of study populations have hindered the cross-comparison of published data. For this reason, AKI in the present study was defined according to the KDIGO Clinical Practice Guidelines for Acute Kidney Injury published in 2012 [17], which are currently the most widely accepted and used criteria. To our knowledge, this is the first multi-institution validation study to use the KDIGO guidelines to compare existing prediction models of AKI in patients with AHF.

Among the AKI prediction models for patients with AHF, the Forman risk score was the first to be published and utilizes 4 factors (i.e., congestive heart failure history, diabetes mellitus, systolic blood pressure over 160 mmHg during admission, and elevated creatinine). The study introducing the risk score showed predictive ability for AKI in AHF but it did not report any area under the ROC curve.[10] The AUC for AKI prediction was externally validated as being 0.65 by Breidthardt et al. in 2011 [1] and Wang et al. in 2013.[12] The subsequent Basel risk score sought to use fewer predictive factors to achieve better prediction ability. Chronic kidney disease, bicarbonate level, and outpatient diuretics treatment were used for AKI prediction and the AUC was reported as 0.71 in the original article. However, a few years later, Wang et al. found no difference in discrimination ability between the Basel and Forman risk scores, both of which had an AUC of 0.65 according to external validated results.[12] In 2013, Wang et al. reported a prediction score derived from a larger patient number and, for the first time in such research, included proteinuria as one of the risk factors of AKI prediction in the AHF papulation. Since then, proteinuria has been increasingly reported to be not only a predictive factor but also an aggravating factor in AKI.[20–22] The Wang et al. prediction model had a high sensitivity of 70.0%, specificity of 70.6%, and AUC of 0.76 in predicting AKI in AHF patients. Subsequently, Zhou et al. derived the first scoring system combining clinical risk factors and novel kidney injury biomarkers (uNGAL and uAGT).[13] The Zhou et al. study reported the AUC separately; the AUC for the clinical model alone was 0.765, close to that of the Wang et al. model, while the AUC for the prediction model was 0.874.[13]

Our current study not only externally validated these five prediction models in terms of AKI prediction but also estimated their performance in predicting serious AKI events including AKI stage 3 and dialysis. As Table 3 shows, the AUCs of these prediction models for AKI prediction ranged from 0.543 to 0.73. Better performance was noted in AKI stage 3 and dialysis prediction, with AUCs of 0.565–0.858 and 0.539–0.845, respectively. All five prediction models showed favorable ability in long-term outcome prediction, with significantly higher incidences of MAKEs in the high-score groups than in the low-score groups.

Of the five prediction models we validated, the Forman risk score and Wang et al. model showed superior discrimination and calibration. The AKI risk score for AHF derived using the Wang et al. model had the best performance; its AUC was 0.73 in AKI prediction and its AUCs for AKI stage 3 and dialysis were 0.858 and 0.845, respectively. This scoring system showed favorable calibration in predicting all three outcomes. The Forman risk score also showed good performance and calibration in AKI, AKI stage 3, and dialysis prediction, with AUCs of 0.696, 0.829, and 0.817, respectively.

Although the further pairwise comparison of AUCs revealed significant differences between the Wang et al. model and Forman risk score (Table 4), both had excellent discrimination (AUC of 0.8–0.9) by general definition.[23, 24] Considering this, the Forman risk score may be seen as the relatively easier and more convenient tool for predicting AKI in AHF patients clinically because it requires only 4 clinical factors.

Much current research is being conducted to identify serum or urine biomarkers for early AKI prediction. However, these biomarkers are more costly to utilize and are not yet widely examined in general laboratory settings. Some recent studies have reported that adding urine biomarker to clinical prediction models yielded no significant performance improvement [25–29], and Törnblom et al. even reported that new statistical methods no longer support using uNGAL to predict AKI in certain patient groups.[28] Taking this into consideration, prediction models based on clinical parameters seem to offer a faster, cheaper, and easier means of AKI prediction, thus increasing the likelihood of AKI prevention and early intervention. The current study demonstrated that a clinical prediction model alone can provide excellent discrimination ability for AKI in AHF patients. Clinical prediction models can achieve an AUC of 0.80, which is particularly high for serious AKI events prediction.

Strengths and limitations

Our study has several notable strengths. First, this is the first multi-institution validation study to compare existing prediction models of AKI in AHF based on the KDIGO Clinical Practice Guidelines. Second, our study further evaluated the performance of these prediction models in predicting serious AKI events, and revealed that these prediction models also offer high discriminative power for predicting AKI stage 3 and dialysis. Third, this study not only assessed the short-term renal outcomes of patients with AHF but also evaluated their long-term outcomes. We demonstrated that patients with scores above the cutoff value had poorer long-term outcomes (defined by MAKEs incidence) than did the lower score groups.

This study also has some limitations. First, this was a retrospective analysis, the inherent drawbacks of which cannot be avoided. Second, the first record of creatinine level upon emergency department admission was used as the baseline creatinine level, and AKI was defined by the creatinine change after that. Thus, our study could only examine predictive ability in terms of AKI developing during admission and not of AKI at admission. Third, data limitations prevented some prediction factors from being validated, including NT-proBNP, uNGAL, and uAGT.

We externally validated five existing prediction models for AKI in patients with AHF. The Forman risk score and Wang et al. model showed favorable discrimination and calibration in predicting AKI, AKI stage 3, and dialysis. The Forman risk score, as it comprises only 4 prediction factors, may offer the easiest and fastest means of individual risk prediction as well as risk stratification. By utilizing appropriate prediction models, clinicians can assess the risk of AKI in patients with AHF earlier and thus plan and initiate adequate disease management for these patients in a much timelier manner.

AKI= acute kidney injury

AHF= acute heart failure

AUC= area under the receiver operating characteristic

CGRD= Chang Gung Research Database

KDIGO= Kidney Disease Improving Global Outcome

MAKEs= major adverse kidney events

WRF= worsening renal function

Ethics approval and consent to participate:

Not applicable

Consent for publication:

Not applicable

Availability of data and materials:

Not applicable

Competing interests:

The authors declare that they have no competing interests

Funding: This research received no external funding

Tao Han Lee¹, Pei-Chun Fan^1,2, Jia-Jin Chen¹, Victor Chien‐Chia Wu³, Cheng-Chia Lee^1,2, Chieh-Li Yen¹, George Kuo¹, Hsiang-Hao Hsu¹, Ya-Chung Tian¹, Chih-Hsiang Chang^1,2*

Author Contributions:

Tao Han Lee, Chih-Hsiang Chang : methodology and writing; Tao Han Lee, Jia-Jin Chen, Victor Chien‐Chia Wu: formal analysis; Tao Han Lee, Chih-Hsiang Chang, Pei-Chun Fan, Chieh-Li Yen, Cheng-Chia Lee, George Kuo, Hsiang-Hao Hsu, Ya-Chung Tian: data extraction; Tao Han Lee, writing—original draft preparation; Chih-Hsiang Chang : writing—review and editing; Chih-Hsiang Chang : project administration. All authors read and approved the final manuscript.

Acknowledgments:

The authors would like to thank Alfred Hsing-Fen Lin, MS, and Zoe Ya-Jhu Syu, MPH, Raising Statistics Consultant Inc., for their statistical assistance.

Breidthardt T, Socrates T, Noveanu M, Klima T, Heinisch C, Reichlin T, Potocki M, Nowak A, Tschung C, Arenja N et al: Effect and clinical prediction of worsening renal function in acute decompensated heart failure. Am J Cardiol 2011, 107(5):730-735.
Bagshaw SM, Cruz DN, Aspromonte N, Daliento L, Ronco F, Sheinfeld G, Anker SD, Anand I, Bellomo R, Berl T et al: Epidemiology of cardio-renal syndromes: workgroup statements from the 7th ADQI Consensus Conference. Nephrol Dial Transplant 2010, 25(5):1406-1416.
Rastogi A, Fonarow GC: The cardiorenal connection in heart failure. Current Cardiology Reports 2008, 10(3):190-197.
Cowie MR, Komajda M, Murray-Thomas T, Underwood J, Ticho B, Investigators P: Prevalence and impact of worsening renal function in patients hospitalized with decompensated heart failure: results of the prospective outcomes study in heart failure (POSH). Eur Heart J 2006, 27(10):1216-1222.
Chen YT, Jenq CC, Hsu CK, Yu YC, Chang CH, Fan PC, Pan HC, Wu IW, Cherng WJ, Chen YC: Acute kidney disease and acute kidney injury biomarkers in coronary care unit patients. BMC Nephrol 2020, 21(1):207.
Logeart D, Tabet JY, Hittinger L, Thabut G, Jourdain P, Maison P, Tartiere JM, Solal AC: Transient worsening of renal function during hospitalization for acute heart failure alters outcome. Int J Cardiol 2008, 127(2):228-232.
Zhou Q, Zhao C, Xie D, Xu D, Bin J, Chen P, Liang M, Zhang X, Hou F: Acute and acute-on-chronic kidney injury of patients with decompensated heart failure: impact on outcomes. BMC Nephrol 2012, 13:51.
Chang CH, Fan PC, Chang MY, Tian YC, Hung CC, Fang JT, Yang CW, Chen YC: Acute kidney injury enhances outcome prediction ability of sequential organ failure assessment score in critically ill patients. PLoS One 2014, 9(10):e109649.
Smith GL, Vaccarino V, Kosiborod M, Lichtman JH, Cheng S, Watnick SG, Krumholz HM: Worsening renal function: what is a clinically meaningful change in creatinine during hospitalization with heart failure? J Card Fail 2003, 9(1):13-25.
Forman DE, Butler J, Wang Y, Abraham WT, O'Connor CM, Gottlieb SS, Loh E, Massie BM, Rich MW, Stevenson LW et al: Incidence, predictors at admission, and impact of worsening renal function among patients hospitalized with heart failure. J Am Coll Cardiol 2004, 43(1):61-67.
Verdiani V, Lastrucci V, Nozzoli C: Worsening renal function in patients hospitalized with acute heart failure: risk factors and prognostic significances. Int J Nephrol 2010, 2011:785974.
Wang YN, Cheng H, Yue T, Chen YP: Derivation and validation of a prediction score for acute kidney injury in patients hospitalized with acute heart failure in a Chinese cohort. Nephrology (Carlton) 2013, 18(7):489-496.
Zhou LZ, Yang XB, Guan Y, Xu X, Tan MT, Hou FF, Chen PY: Development and Validation of a Risk Score for Prediction of Acute Kidney Injury in Patients With Acute Decompensated Heart Failure: A Prospective Cohort Study in China. J Am Heart Assoc 2016, 5(11).
Alattar FT, Imran N, Debari VA, Mallah KN, Shamoon FE: Fractional excretion of sodium predicts worsening renal function in acute decompensated heart failure. Exp Clin Cardiol 2010, 15(3):e65-69.
Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, Acute Dialysis Quality Initiative w: Acute renal failure - definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care 2004, 8(4):R204-212.
Mehta RL, Kellum JA, Shah SV, Molitoris BA, Ronco C, Warnock DG, Levin A, Acute Kidney Injury N: Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Crit Care 2007, 11(2):R31.
Section 2: AKI Definition. Kidney Int Suppl (2011) 2012, 2(1):19-36.
Tsai MS, Lin MH, Lee CP, Yang YH, Chen WC, Chang GH, Tsai YT, Chen PC, Tsai YH: Chang Gung Research Database: A multi-institutional database consisting of original medical records. Biomed J 2017, 40(5):263-269.
Shao SC, Chan YY, Kao Yang YH, Lin SJ, Hung MJ, Chien RN, Lai CC, Lai EC: The Chang Gung Research Database-A multi-institutional electronic medical records database for real-world epidemiological studies in Taiwan. Pharmacoepidemiol Drug Saf 2019, 28(5):593-600.
Chen J-J, Chang C-H, Lee C-C, Yang C-H, Chen Y-T, Chang S-W, Fan P-C, Tian Y-C, Yang C-W, Chen Y-C: Proteinuria Enhances Prediction Ability of Sequential Organ Failure Assessment Score and Associated with Mortality in Coronary Care Units. Acta Nephrologica 2017, 31(4):188 - 196.
Hsu CY, Chinchilli VM, Coca S, Devarajan P, Ghahramani N, Go AS, Hsu RK, Ikizler TA, Kaufman J, Liu KD et al: Post-Acute Kidney Injury Proteinuria and Subsequent Kidney Disease Progression: The Assessment, Serial Evaluation, and Subsequent Sequelae in Acute Kidney Injury (ASSESS-AKI) Study. JAMA Intern Med 2020, 180(3):402-410.
Huang TM, Wu VC, Young GH, Lin YF, Shiao CC, Wu PC, Li WY, Yu HY, Hu FC, Lin JW et al: Preoperative proteinuria predicts adverse renal outcomes after coronary artery bypass grafting. J Am Soc Nephrol 2011, 22(1):156-163.
Jayawant N. Mandrekar: Receiver Operating Characteristic Curve in Diagnostic Test Assessment. BIOSTATISTICS FOR CLINICIANS 2010, 5(9):1315-1316.
Hosmer DW, Lemeshow S: Applied logistic regression, 2nd edn. New York: Wiley; 2000.
Alobaidi R, Basu RK, Goldstein SL, Bagshaw SM: Sepsis-associated acute kidney injury. Semin Nephrol 2015, 35(1):2-11.
Flechet M, Guiza F, Schetz M, Wouters P, Vanhorebeek I, Derese I, Gunst J, Spriet I, Casaer M, Van den Berghe G et al: AKIpredictor, an online prognostic calculator for acute kidney injury in adult critically ill patients: development, validation and comparison to serum neutrophil gelatinase-associated lipocalin. Intensive Care Med 2017, 43(6):764-773.
Ronco C, Legrand M, Goldstein SL, Hur M, Tran N, Howell EC, Cantaluppi V, Cruz DN, Damman K, Bagshaw SM et al: Neutrophil gelatinase-associated lipocalin: ready for routine clinical use? An international perspective. Blood Purif 2014, 37(4):271-285.
Törnblom S, Nisula S, Petäjä L, Vaara ST, Haapio M, Pesonen E, Pettilä V, Laru-Sompa R, Pulkkinen A, Saarelainen M et al: Urine NGAL as a biomarker for septic AKI: a critical appraisal of clinical utility—data from the observational FINNAKI study. Annals of Intensive Care 2020, 10(1):51.
Albert C, Albert A, Bellomo R, Kropf S, Devarajan P, Westphal S, Baraki H, Kutschka I, Butter C, Haase M et al: Urinary neutrophil gelatinase-associated lipocalin-guided risk assessment for major adverse kidney events after open-heart surgery. Biomark Med 2018, 12(9):975-985.

Download PDF

Version 1

posted

You are reading this latest preprint version

A Validation Study Comparing Existing Prediction Models of Acute Kidney Injury in Patients with Acute Heart Failure: A Multi-Institution Database Study in Taiwan

Status:

Version 1

Abstract

Figures

Introduction

Methods

Data source

Study population

Existing prediction models and covariates

Outcome definition

Statistical analysis

Results

Baseline characteristics

Validation of existing prediction models for AKI

Extension of models for predicting AKI stage 3 and dialysis

Extension of models for predicting MAKEs

Discussion

Strengths and limitations

Conclusion

Abbreviations And Acronyms

Declarations

References

Status:

Version 1