AKI Risk Score (AKI-RiSc): Developing an Interpretable Clinical Score for Early Identi cation of Acute Kidney Injury at the Emergency Department

Yukai Ang Duke-NUS Medical School Siqi Li Duke-NUS Medical School Marcus Eng Hock Ong Singapore General Hospital Feng Xie Duke-NUS Medical School Su Hooi Teo Singapore General Hospital Lina Choong Singapore General Hospital Riece Koniman Singapore General Hospital Bibhas Chakraborty Duke-NUS Medical School Andrew Fu Wah Ho Singapore General Hospital Nan Liu (  liu.nan@duke-nus.edu.sg ) Duke-NUS Medical School


Introduction
Acute kidney injury (AKI) is a common clinical syndrome affecting 10-20% of hospitalized patients worldwide. 1 It is independently associated with increased risk of inpatient mortality, 2 signi cant morbidity upon discharge, 3 and increased healthcare costs as well as length of hospital stay. 4 AKI is also often clinically silent and may not be promptly recognised by the attending physician. 5 Established kidney injury is often di cult to treat, 6 hence there is a need for early detection to initiate treatment promptly. AKI is typically diagnosed by the magnitude of serum creatinine (SCr) rise and is usually based on well accepted criterion such as the Kidney Disease Improving Global Outcome AKI (2012) guidelines. Early warning systems have been developed to ag patients with pathological rises in SCr, which can then be paired with intervention care bundles that can prevent the progression of AKI. 7 However, increase in SCr levels can lag kidney injury by up to 48 hours. 8 Physicians, therefore, are only reacting to damage that has already been done.
AKI prediction models are potential solutions to this problem. In recent years, there has been increasing interest in the development of machine learning (ML) models, which can accurately predict AKI development in patients before any rise in SCr. [9][10][11] Physicians will then be able to intervene earlier and halt the progression of the renal insult. 12 However, these ML models have yet to be widely utilized in clinical practice due to their complexity which may be di cult to implement in existing hospital IT systems. 11 Furthermore, many ML models derive their predictions via black box approaches, which are not always explainable to humans in a rational way. Hence, clinicians may not be as inclined to apply them in clinical practice. 13 Point based prediction scores offer a simple and interpretable solution to these ML models. However, most existing point based AKI risk scores were designed for speci c patient populations in the Intensive Care Unit, 14,15 post-operative period, 16 and undergoing procedures involving contrast, 16-18 when the majority of AKI occurs in the general ward setting. 19 The primary aim of this study was to create a simple point based clinical AKI Risk Score (AKI-RiSc) for the general patient population using a systematic, machine learning-based scoring framework -AutoScore. 20 The AKI-RiSc was designed to assess a patient's 7 day inpatient risk of AKI development in the setting of the emergency department (ED). It is envisioned that AKI-RiSc can function as a convenient and informative adjunct in allowing clinicians to assess a patient's risk of inpatient AKI and institute any necessary management steps more accurately.

Study design and setting
We conducted a retrospective, single-center study in Singapore General Hospital (SGH) to derive AKI-RiSc using EHR of the patients in the ED and inpatient wards. SGH is a tertiary hospital that receives over 120,000 ED visits and 36,000 admissions annually. A waiver of consent for EHR data collection was granted, and the study protocol was approved by Singapore Health Services' Centralised Institutional Review Board. All research has been performed in accordance with the Declaration of Helsinki.

Study sample
All adult patients (>18 years old) visiting the ED between 1January 2008 to 31 December 2016 and who were subsequently admitted to medical and surgical wards were studied. 21 Patients were excluded if they met any of the following criteria: 1) patients with no records of SCr, 2) patients with AKI on presentation at the ED, de ned based on KDIGO change of SCr from median annualised SCr baseline, 3) patients with pre-existing advanced chronic kidney disease (CKD) based on KDIGO guidelines (SCr ≥353.6 µmol/L on admission) and 4) patients with no information of comorbidities. Patients were then followed up to 7 days post-admission to determine if they developed AKI or not.

AKI de nition and outcomes
The primary outcome of this study was the development of inpatient AKI of any stage within 7 days of admission from the ED. We used the National Health Service (NHS) automated AKI algorithm based on KDIGO guidelines to de ne AKI. 22 The algorithm had been applied in clinical practice and was chosen because of its ability to account for patients with and without prior baseline SCr information. 23 A patient was determined to have AKI if any of the three criteria were met: 1) Increase in SCr to ≥1.5 times of the median of all SCr readings 8 to 365 days ago, 2) increase in SCr to ≥1.5 times of the lowest SCr reading in the past 7 days, 3) increase in SCr by ≥ 26.5µmol/L within 48 hours from the lowest SCr reading. Other outcomes that were examined in this study were patients who developed at least Stage 2 AKI and patients who developed Stage 3 AKI.

Data collection and candidate variables
We extracted the data from the hospital's EHR -SingHealth Electronic Health Intelligence System (eHints). The data was de-identi ed, and the death records were obtained from the national death registry and matched to speci c patients in the EHR. As the AKI-RiSc was designed to be applied in the ED setting, we only selected variables that were exclusively available and reliably obtained at the ED. The nal 33 candidate variables were selected based on literature review and expert opinion from clinicians, which consisted of patient demographics, vital signs, biochemical results, comorbidities, medical interventions, and visits to hospital -as seen in Table 1. Comorbidities were obtained from the hospital diagnosis and discharge records in the past ve years before each patient's current ED visits. They were recorded as the International Classi cation of Diseases (ICD) 9/10 codes. 24 Statistical analyses and predictive modeling Variables were included for the analysis only if more than 80% of them were available in the study cohort. Chi-square for categorical variables and t-tests for continuous variables were used where appropriate to compare between the patients who developed AKI and those who did not. Univariable analysis was also used to determine the odds ratios of risk factors between the two groups. Figure 1 depicts data preparation. Patients admitted between 2008 to 2015 were randomly divided into derivation and validation cohorts, at a 70% and 30% ratio, respectively. (Figure 1b) Patients admitted in 2016 were used as the test cohort to evaluate the nal score.
AKI-RiSc was developed using AutoScore, 20 a machine learning-based algorithm for interpretable clinical score generation. AutoScore facilitates the easy and transparent development of interpretable clinical score models for a pre-de ned clinical outcome. It uses a combination of machine learning, logistic regression, and user-de ned parameter ne-tuning. Figure S2 illustrates the process of score derivation and validation in the AutoScore framework.
The training cohort was rst used to generate the preliminary AKI-RiSc models using the AutoScore main ow. The validation cohort was then used to evaluate the performance of various candidate AKI-RiSc models and allow for parameter ne-tuning. After selecting the nal AKI-RiSc model, we evaluated its performance using the testing cohort, and bootstrapped samples were applied to calculate 95% con dence intervals. The primary outcome measure of inpatient AKI development was used for model derivation and model testing. The predictive power of AKI-RiSc was assessed using the area under the curve (AUC) in the receiver operating characteristic (ROC) analysis. Sensitivity, speci city, positive predictive value (PPV), and negative predictive value (NPV) of the various AKI-RiSc cut-offs were also calculated. We also compared AKI-RiSc with an existing AKI clinical score, the AKI Prediction Score (APS) 25 , by evaluating its performance using the same testing cohort with the same primary outcome measure.

Role of the funding source
The funder was not involved in the study design, collection, analysis, and interpretation of data, nor did they have a role in the writing of the paper and decision to submit the paper for publication.

Results
The initial cohort comprised 317,604 unique visits to the ED that were subsequently admitted. After excluding admissions with no available SCr (n = 174,098), and those with SCr >353.6 µmol/L or AKI on admission (n = 24,037), we arrived at a nal cohort of 119,468 unique entries. Among this cohort, 10,693 (9.0%) developed AKI of any stage within 7 days of admission, of which 8,491 were stage 1 (79.4%), 906 stage 2 (8.5%), and 1,296 stage 3 (12.1%) (Figure 1). The overall in-hospital mortality rate of patients admitted with AKI was 21.8%, as opposed to 3.6% in non-AKI counterparts. While most AKI patients were stage 1, the number of patients in stage 3 was higher than stage 2, which was also observed in another recent hospital-wide AKI study. 26 Patients who developed AKI were older, had higher baseline SCr, higher admission SCr, and had more comorbidities associated with them (Table 1). Univariable analysis (Table S1) identi ed tachycardia, tachypnoea, hypotension, and hypertension as the most signi cant risk factors among the vital signs. Higher SCr levels on admission at the ED also increased the risk of inpatient AKI. Among the other lab values, low serum bicarbonate was the most signi cant risk factor for developing AKI. Extremes of serum sodium and potassium values also conferred an increased risk of AKI. All comorbidities were signi cantly associated with the development of AKI, but patients with renal disease had the greatest risk (odds ratio [OR]: 4.07, 95% con dence interval [CI] 3.91 -4.24). Previous surgeries, admissions to the hospital, high dependency unit, and ICU were also signi cant risk factors.
AKI-RiSc was developed in a stepwise manner with the AutoScore framework. After ranking the essential AKI prediction variables, a parsimony plot (Figure 2) was created to visualise model selection. The variables shown in Figure 2 were ranked in order of importance of AKI prediction based on the initial random forest selection, with SCr being the most important and intubation being the least. We selected the six most important variables in the parsimony plot (SCr, serum bicarbonate, pulse, SBP, DBP, and age) to build the AKI-RiSc. Each variable was allocated an integer score based on its importance. The nal score was a summation of all the variable's assigned scores, ranging from 0 to 15 ( Table 2). Risk of inpatient AKI development was positively correlated with the higher AKI-RiSc scores. Higher SCr levels contributed most signi cantly with greater nal scores, with SCr values ≥250 μmol/L being allocated the highest score of 7. Tachycardia (≥120 beats per minute) was also given a higher score of 3. Other variables associated with higher scores were lower bicarbonate levels of <20 mmol/L, systolic blood pressure of ≥150 mmHg, diastolic blood pressure of ≥90 mmHg, and older age ≥50.
In terms of performance, AKI-RiSc performed reasonably well at an AUC of 0.730 (95% CI: 0.713 -0.747) when evaluated on the test cohort. At a cut-off of 4 points, it has a sensitivity of 82.5%, speci city of 46.7%, positive predictive value of 7.7%, and negative predictive value of 97.1%. When the APS, 25 a similar point based AKI risk score, was evaluated on the same test cohort as a basis of comparison, it scored an AUC of 0.665 (95% CI: 0.646 -0.679). The APS consisted of age, respiratory rate, mental status (AVPU), CKD, CHF, DM, Liver Disease. Table 3 presents the score thresholds and respective sensitivity, speci city, positive predictive value, and negative predictive value. Figure S1 shows the correlation between the calculated AKI-RiSc value and the proportion of patients who developed AKI within 7 days. Almost 30% of patients with an AKI-RiSc of 9 developed AKI.

Discussion
In this study, we used a large retrospective dataset and the AutoScore framework to construct a pointbased clinical score, AKI-RiSc, to assess the 7 day risk of AKI development in a general patient population after admission from the ED. The AKI-RiSc performed signi cantly better than the original APS score when evaluated on the same test cohort.
The development of AKI-RiSc using AutoScore and our dataset brings several advantages. First, our sample size of 119,468 is considerably large which improves the reliability of the results obtained from our study. Secondly, the AutoScore scoring framework is built upon machine learning, which achieves a better parsimonious solution for clinical prediction tasks with large data sets. 27 Thirdly, the selection of variables for AKI-RiSc was also performed in a transparent manner by identifying the most parsimonious solution from the parsimony plot (Figure 2), allowing clinicians to better understand the processes involved in generating the score. One key difference between the two scores was the use of biochemical variables in AKI-RiSc -SCr and serum bicarbonate. SCr is a well-accepted marker of kidney function and has also been the most crucial variable in other AKI prediction models. 10,29 Low bicarbonate, which typically indicates metabolic acidosis, was also found independently associated with increased AKI risk in other studies. 30 Including these biochemical results could have potentially improved the performance of AKI-RiSc. It is worth noting that the APS also achieved a similar AUC of 0.71 when evaluated on their own UK cohort. 31 One explanation of the poor performance of the APS in our Singapore study could be the inherent differences between the population demographics it was derived and tested on. 32 For example, the average population age of the UK cohort was overall older than that of the Singapore cohort.
Comparing to other ML models, however, the performance of AKI-RiSc lags. Tomasev's et al. developed a recurrent neural network AKI prediction model that calculated and updated the risk of AKI every 48 hours, performing well with an AUROC of 0.921. 29 Despite these remarkable results, the translation of ML models to practical clinical medicine has been limited. ML models often require signi cant amount of information to function optimally, which may be di cult to obtain consistently. 33 Furthermore, the complexity of the models' algorithms make it challenging for implementation in many hospital's IT infrastructure. 34 Many of these machine learning models are also developed using a black box approach as opposed to more traditional logistic regression models which clinicians are familiar with. 34 This can potentially limit data interpretation and the willingness of clinicians to adopt the model's predicted risk. 13 Therefore, while AKI-RiSc may not perform as well as pure ML models in AKI prediction, it has the bene t of being much more easily implemented in the clinical setting with a reasonable predictive performance.
Several actions can be taken after a high-risk patient has been identi ed using the AKI-RiSc in the ED. First, interventional bundle care plans can be instituted for patients identi ed with a high AKI-RiSc in the form of a checklist to guide clinicians in the treatment of AKI 7 and potentially improve patient outcomes. 35 Second, a high AKI-RiSc score can prompt the emergency physician to make earlier nephrology consults, which can improve the outcomes of patients with renal impairment. 36 Third, the utility of the AKI-RiSc can be further enhanced when paired with novel AKI biomarkers, in which high risk patients who may bene t from AKI biomarker test can be initially identi ed using AKI-RiSc.
There are limitations in this study. As with most other AKI studies, AKI was de ned only using serum SCr as urine output information was unavailable. Furthermore, more than half of the admissions were rsttime admissions, meaning that they did not have a median annual baseline SCr. A signi cant number of admissions relied on the ED SCr reading as the baseline. In reality, admission SCr could already have been pathologically raised due to prior kidney insults preceding the admission. Hence, using the ED SCr as a baseline reference meant that the actual incidence of AKI could be underestimated. 37 While we could accurately identify the diagnosis of AKI in patients by trending changes in SCr, we did not have any information on the aetiology of the AKI. However, AKI-RiSc was designed to be an alert system designed to recommend treatment, not as a diagnostic tool. 25 The next step would be to prospectively validate the AKI-RiSc in the various hospitals of Singapore. AKI-RiSc can then be sequentially applied to hospitals in regional South East Asian countries, and eventually to the international setting of the world abroad. We believe that the simplicity of AKI-RiSc and use of common clinical variables makes it a good candidate for validation in hospitals across different settings. Apart from assessing its predictive performance in a prospective validation cohort, it would also be essential to gain feedback from physicians on the score's perceived utility and ease of use in helping them better care for their patients. AKI bundles can also be developed to complement the AKI-RiSc to assess its effectiveness in improving patient outcomes.

Conclusions
In conclusion, we developed a simple and interpretable point-based AKI risk score which can be easily implemented on the ground for early identi cation of high-risk patients in the Emergency Department. It also has the potential be applied in healthcare settings internationally.

Declarations
Author contributions: NL conceived and designed the study. YA, SL, FX, and NL accessed and veri ed the underlying study data. YA, SL, FX, and NL analysed the data. All authors interpreted the data. YA, MEHO, FX, and NL drafted the manuscript. All authors critically revised the manuscript for intellectual content.
NL supervised the study. All authors had nal responsibility for the decision to submit for publication.  Tables   Table 1: Comparison of patient characteristics between those who developed AKI 7 days after admission and those who did not at the emergency department.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.