External validation of the Hospital Frailty Risk Score among older adults receiving mechanical ventilation

Abstract

Purpose:

To externally validate the Hospital Frailty Risk Score (HFRS) to predict clinical outcomes in critically ill patients.

Methods:

We selected index hospitalizations of older adults (≥75 years old) receiving mechanical ventilation, using the United States Nationwide Readmissions Database (NRD) from January 1, 2016, to November 30, 2018. Frailty risk was determined by the HFRS using International Classification of Diseases, Tenth Revision Clinical Modification (ICD-10-CM) codes, and further subcategorized into low-risk (score <5), intermediate-risk (score 5-15), and high-risk (score >15) for frailty. We evaluated the HFRS to predict prolonged hospitalization, in-hospital mortality, and 30-day emergency hospital readmissions, using multivariable logistic regression, after adjustment for patient and hospital characteristics. Model performance was assessed using the c-statistic, Brier score, and calibration plots.

Results:

Among the 649,330 weighted index hospitalizations in the cohort, 50.0% were female, the median (interquartile range [IQR]) age was 81 (78-86) years old, and the median (IQR) HFRS was 10.8 (7.7-14.5). Among the cohort, 9.5%, 68.3%, and 22.2% were subcategorized as low-, intermediate-, and high-risk for frailty, respectively. After adjustment, high-risk patient hospitalizations were associated with increased risks of prolonged hospitalization (adjusted odds ratio [aOR] 5.59 [95% confidence interval [CI] 5.24-5.97], c-statistic 0.694, Brier score 0.216) and 30-day emergency hospital readmissions (aOR 1.20 [95% CI 1.13-1.27], c-statistic 0.595, Brier score 0.162), compared to low-risk hospitalizations. Conversely, high-risk hospitalizations were inversely associated with in-hospital mortality (aOR 0.46 [95% CI 0.45-0.48], c-statistic 0.712, Brier score 0.214). Calibration plots demonstrated good calibration for the adjusted analyses.

Conclusions:

In this external validation study, the HFRS was not successfully validated in the NRD to predict in-hospital mortality in critically ill older adults receiving mechanical ventilation. While the HFRS may predict prolonged hospitalization and 30-day readmission, its use should be avoided in the critically ill.

Introduction

Frailty is increasingly being recognized as a risk factor for mortality, prolonged hospitalization, readmission, and poorer quality of life after discharge in critically ill older adults.^1–7 Up to 24% of critically ill patients may be frail at baseline prior to admission.⁸ Most studies have prospectively assessed frailty using the Canadian Study of Health and Aging Clinical Frailty Score (CFS).^4,8–10 However, most frailty scores (i.e., CFS, Fried’s frailty phenotype, Edmonton Frail Scale) have limited use in administrative databases (i.e., Canadian Institute for Health Information Discharge Abstract Database, United States [US] Centers for Medicare & Medicaid Services databases), as these databases do not contain the information necessary to calculate these scores.^10–13 Hence, our understanding of frailty in critical illness has been limited to prospective studies. 

Consequently, frailty scores for administrative databases have been developed. The electronic Frailty Index (eFI) was developed for use in primary care electronic records.¹⁴ The modified Frailty Index (mFI) has been studied in Brazilian intensive care units (ICUs); however, it requires the measurement of functional capacity.¹⁵ Recently, the Hospital Frailty Risk Score (HFRS) was derived using ridge regression and 109 International Classification of Diseases, Tenth Revision Clinical Modification (ICD-10-CM) codes, from a cohort of >20,000 hospitalized older adults.¹⁶ The HFRS has been validated to predict the risk of 30-day mortality, prolonged hospitalization, and 30-day emergency hospital readmissions in older hospitalized patients,¹⁶ and it has since been validated in other hospitalized population databases.^17–20 However, its validity in critically ill patients has been questioned in a single center study.²¹ Thus, there is a need to study the validity of the HFRS in large administrative databases of critically ill patients. The primary goal of this study was to externally validate the HFRS among a nationally representative US sample of older adults receiving mechanical ventilation.

Methods

This study was reported using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement.²² It was exempted by the Saskatchewan Health Authority Research Ethics Board (SHA-REB-20-77), as de-identified information was used.

Data source

We extracted information from the Nationwide Readmissions Database (NRD) from January 1, 2016, to November 30, 2018. The NRD is the largest all-payer US readmission database from the Healthcare Cost and Utilization Project (HCUP), and it includes hospitalizations of both insured and uninsured patients from 28 different state databases.²³ It samples >15,000,000 unique hospitalizations annually, representing >36,000,000 weighted hospitalizations, including general ward, intermediate care, and ICU patients.²³ The NRD accounts for 60% of the total US population and 59% of all hospitalizations, allowing for national estimates.

Study population

We included all hospitalizations of older adults (≥75 years old) receiving mechanical ventilation, using the ICD-10 procedure coding system (ICD-10-PCS) (Electronic Supplementary Material (ESM) eTable 1). An equivalent validated ICD-9 algorithm had a sensitivity of 47.0% and specificity of 99.6% for detecting mechanical ventilation in all types of patients, and a sensitivity of 94.0% and specificity of 99.3% in medical patients specifically.²⁴

We excluded patients who left against medical advice and hospitalizations with missing information for length of stay, time to next visit, and December admissions, as the NRD is unable to follow these patients beyond the calendar year. We also excluded hospitalizations of non-residents of the state, as the NRD does not have any linking state identifiers.

Measurements

The covariates in the NRD included age, biological sex, hospital characteristics (teaching status, size), income quartile, primary insurance status (Medicare, Medicaid, private insurance, self-pay, or other) and Elixhauser-van Walraven comorbidity index score.²⁵ The ICD-10-CM and ICD-10-PCS codes were used to classify comorbidities (ESM eTable 1). We determined the primary reason for admission of the index hospitalization and readmission, using the first listed diagnosis (DX1) and aggregate groups of the Clinical Classifications Software Refined (CCSR) developed by HCUP (ESM eTable 2).²⁶ Hospital costs were determined using total hospital charges multiplied by the all-payer cost-to-charge ratio, then inflation-adjusted to 2018 US dollars using the US Bureau of Labor Consumer Price Index for medical care.^27,28 Linked visits were identified through a linking variable.

Frailty risk

Frailty risk was assessed using the HFRS developed by Gilbert et al (ESM eTable 3).¹⁶ We classified patients as either low-risk (score <5), intermediate-risk (score 5-15), or high-risk (score >15) for frailty, based on the original HFRS study and subsequent validation studies.^16–18

Outcome(s)

We evaluated the performance of the HFRS to predict in-hospital mortality, as the primary outcome. The predictive performance of the HFRS for prolonged hospitalization and 30-day emergency hospital readmissions were evaluated as secondary outcomes. We only evaluated in-hospital all-cause mortality instead of 30-day mortality (inpatient or outpatient) because the NRD only records in-hospital deaths. We defined prolonged hospitalization as a length of stay >10 days in hospital and only evaluated 30-day emergency hospital readmissions, similar to Gilbert et al.^16,17

Statistical analysis

All statistical analyses were performed using Stata/MP 15.1 (College Station, Texas, US). A two-sided p-value <0.05 was considered statistically significant. We accounted for the complex sampling design of the NRD using sampling weights provided by HCUP.²³ Categorical variables were presented as unweighted numbers and weighted percentages. Continuous variables were presented as either means (standard deviation [SD]), or medians (interquartile range [IQR]), following testing for normality. Survey-specific Rao-Scott tests were used to compare nominal data. Survey-specific linear regression was used to compare continuous data, using the geometric means for non-normal data. Missing data were present in <5% of all patient visits. As a result, a complete case analysis was performed for all analyses given the complex sampling design.^29,30

We assessed the validity of the HFRS for predicting in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission, using unadjusted and adjusted logistic regression. For in-hospital mortality and prolonged hospitalization, we performed adjustment for age, biological sex, income quartile, insurance status, do-not-resuscitate status, admission diagnosis, hospital characteristics, and year. For 30-day emergency hospital readmissions, we performed adjustment for the same variables, also including hospital disposition. Model discrimination was assessed with the c-statistic and calibration with the Brier score.^31,32 Calibration plots additionally were constructed.

Sensitivity analyses

We performed several sensitivity analyses to assess the robustness of our findings. First, we re-evaluated our findings using the HFRS as a continuous variable and using restricted cubic splines with five knots.³³ Next, we performed survey-specific Cox proportional hazards regression for in-hospital mortality and 30-day emergency hospital readmissions.³⁴ Subsequently, we derived 30-day in-hospital mortality, using hospitalization data from the NRD, and re-performed our primary analysis. We performed additional post hoc analyses, restricting the population to those who only received mechanical ventilation for greater than 24 hours and restricting the population to only those who were admitted emergently. We then performed multiple imputation with chained equations for missing data using 10 imputations, and repeated the primary analysis with the imputed dataset.³⁵ Finally, as a post hoc analysis, we evaluated the total population of older adults in the NRD, independent of receiving mechanical ventilation, to determine whether our findings held for the entire older adult population.

Results

There were 371,410 hospitalizations of older adults receiving mechanical ventilation, representing 649,330 weighted hospitalizations (Figure 1). A summary of baseline characteristics is described in Table 1 and ESM eTable 4. Missing data are described in ESM eTable 5. Of the hospitalizations, 50.0% had female patients, the median (IQR) age was 81 (78-86) years old, and the median (IQR) Elixhauser-van Walraven comorbidity index score was 18 (12-25). Infection-related diagnoses (30.5%) were the most common primary diagnoses. Many patients had primary or secondary diagnoses of severe sepsis (32.8%), shock (40.2%), and acute kidney injury (51.5%). Referral to palliative care occurred in approximately 26.8% of hospitalizations, with the high-risk for frailty group receiving the most referrals (p <0.001).

The median (IQR) HFRS was 10.8 (7.7-14.5) (ESM eFigure 1). Of all hospitalizations, 9.5% were classified as low-risk, 68.3% as intermediate-risk and 22.2% as high-risk.

Prevalence of mortality, long hospital length of stay and 30-day hospital readmissions

In-hospital mortality occurred in 45.3% of all hospitalizations, and prolonged hospitalization occurred in 41.1% of all hospitalizations (Table 1). Of survivors, 20.3% were readmitted to hospital by 30 days. Among high-risk for frailty hospitalizations, they had an increased incidence of prolonged hospitalization and 30-day emergency hospital readmissions (all p <0.001) compared to the low-risk for frailty group. However, they had a reduced incidence of in-hospital mortality compared to other frailty groups (p <0.001). 

Assessment of model performance

Model performance was assessed for in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission (Table 2). In the unadjusted analysis, the intermediate- and high-risk groups were associated with reduced risk of in-hospital mortality, prolonged hospitalization, and increased risk of 30-day emergency hospital readmission. After adjustment, the intermediate- and high-risk for frailty groups were associated with reduced in-hospital mortality in this patient population (aOR 0.79 [95% CI 0.77-0.82] for intermediate-risk and aOR 0.46 [95% CI 0.45-0.48] for high-risk, c-statistic 0.712, Brier score 0.214), compared to the low risk for frailty group. Additionally, they were associated with prolonged hospitalization (aOR 2.61 [95% CI 2.46-2.78] for intermediate-risk and aOR 5.59 [95% CI 5.24-5.97] for high-risk, c-statistic 0.694, Brier score 0.216) and increased risk for 30-day emergency hospital readmission (aOR 1.18 [95% CI 1.12-1.24] for intermediate-risk and aOR 1.20 [95% CI 1.13-1.27] for high-risk, c-statistic 0.595, Brier score 0.162) after adjustment. Model calibration assessed using calibration plots (Figure 2) visually demonstrate good calibration of the adjusted models.

Sensitivity analyses

Detailed information on the sensitivity analyses are presented fully in the ESM eResults and in eTable 6-eTable 13. We performed several different analyses to evaluate the robustness of our analysis method, including re-analyzing our data using the HFRS as a continuous variable (ESM eTable 6) or using restricted cubic splines with five knots (ESM eTable 7, Figure 3), performing Cox proportional hazards regression (ESM eTable 8), evaluating in-hospital 30-day mortality (ESM eTable 9), and performing multiple imputation with chained equations (ESM eTable 12). These additional analyses did not yield any significantly different results in our overall findings.

Discussion

In this study, the primary objective was to externally validate the HFRS to accurately predict in-hospital mortality in a large nationally representative cohort of older adults receiving mechanical ventilation. In its current form, the HFRS could not be successfully validated for use in this population. As expected, we found that patient hospitalizations receiving mechanical ventilation with intermediate- and high-risk for frailty, as categorized by the HFRS, were associated with increased risks of prolonged hospitalization and 30-day emergency hospital readmissions, compared to low-risk hospitalizations. Counterintuitively, they were inversely associated with in-hospital mortality when compared to the low-risk hospitalizations, suggestive of a potential spurious relationship. Regardless, the HFRS had only moderate discrimination and accuracy in predicting any of these outcomes. Using the HFRS as a continuous variable or with splines did not provide additional value over using the HFRS subcategories of low-, intermediate-, and high-risk. Our findings would suggest that clinicians and researchers should avoid using the HFRS in administrative datasets of only critically ill patients.

Comparison with previous studies

Prior HFRS studies focused on validating it in general hospitalizations, including non-ICU and ICU patients.^{17–19,36–39} Recently, there has been interest in externally validating the HFRS in ICU administrative databases.^21,40,41 A German ICU study of 1,498 patients evaluated the HFRS to predict a combined endpoint of mortality and risk of readmission and found no association after adjustment for severity of illness.²¹ In a large Wales population study, the HFRS had only moderate ability for predicting inpatient, 6-month, and 1-year mortality in hospital and ICU patients.⁴¹ Conversely, a US study of 12,854 patients, using the single-center Medical Information Mart for Intensive Care (MIMIC-III) database, found that higher HFRS was associated with an increased risk of 28-day mortality.^40,42

In our study, we found that critically ill older adult hospitalizations receiving mechanical ventilation were at high-risk of poor outcomes, including prolonged hospitalization (41%), 30-day in-hospital mortality (44%), in-hospital mortality (45%), and 30-day emergency hospital readmission (20%). Unsurprisingly, palliative care utilization was very high at 26.8%, with higher use in the high-risk frailty groups. The overall readmission rate was high in the patients of this study, suggestive of current difficulties in transitions in care for these patients and potential room for quality improvement.

Prior studies of critically ill patients have established that frailty is associated with increased risks of mortality.^3,4 Counterintuitively, we found that the HFRS was inversely associated with mortality in the NRD (i.e., lower HFRS was associated with the highest risks of in-hospital mortality), suggestive of a spurious association. To ascertain this surprising and unnatural finding, we performed a post hoc analysis on the entire NRD population of older adults, independent of the receipt of mechanical ventilation, and found that the HFRS performed well on the whole population (i.e., higher HFRS was associated with the highest risks of in-hospital mortality in all older adults) (ESM eTable 13).

There may be some possible explanations for this unusual phenomenon, including selection biases and coding biases. In Gilbert et al’s original study, they validated the HFRS in a general hospitalized population to predict in-hospital mortality.¹⁶ Critically ill patients or mechanically ventilated patients represent a “sicker” population or surrogate endpoint, at higher risk of death compared to a general hospitalized population. Therefore, by limiting our population to mechanically ventilated patients, selection bias may have been introduced, potentially altering the true association of the HFRS and mortality. Coding biases may also occur as critically ill patients who had prolonged hospitalizations and/or survived their hospitalization may appear to more “frail,” as they accrue more ICD-10-CM secondary diagnoses captured in their medical records. In the NRD, most of the hospitalizations of older adults receiving mechanical ventilation were in the intermediate-risk frailty group, and most hospitalizations in the high-risk group had significantly more ICD-10-CM codes captured compared to hospitalizations in the low-risk group. Finally, frail patients with higher severity of illness or those with treatment limitations may choose less invasive treatments, introducing further selection bias. We did adjust for do-not-resuscitate status; however, this may not fully capture all treatment limitations.

These biases and differences in the ICU patient population from the original development cohort could potentially explain why the HFRS had mixed performances for predicting in-hospital mortality in an ICU patient population, as seen in this study and others.

Strengths and Limitations

Our study had several strengths including the use of a large multi-center dataset, comprising close to 650,000 weighted hospitalizations. To our knowledge, our study represents one of the largest studies of critically ill patients examining the use of the HFRS, allowing for generalizability of our findings to critically ill older adults receiving mechanical ventilation. Unlike prior external validation studies in critically ill administrative databases, we evaluated the HFRS to predict prolonged hospitalization and 30-day emergency hospital readmissions. Additionally, we assessed both model discrimination and calibration, allowing for confidence in the results presented. Finally, our study performed several sensitivity analyses to verify our findings.

However, our study has limitations. As discussed previously, selection bias may have occurred in our selection of a mechanically ventilated population. Additionally, as the HFRS is derived from a composite of ICD-10 codes, coding practices and biases may affect the relative prevalence of admission comorbidities, diagnoses, and treatments. Some important codes to the determination of the HFRS, such as dementia in Alzheimer’s disease (F00) or care involving the use of rehabilitation procedures (Z50), were undercoded (ESM eTable 3). This is similarly seen in other databases including the Centers for Medicare & Medicaid Services and National Inpatient Sample databases.^36,37 Other databases of critically ill patients may perform differently, depending on their coding practices.

Additionally, the NRD does not have sufficient information to determine ICU severity of illness, such as the sequential organ failure assessment (SOFA) or Acute Physiologic Assessment and Chronic Health Evaluation II (APACHE II) scores. We are therefore unable to verify whether the HFRS would perform better after adequate adjustment for severity of illness; however, other studies would suggest that the HFRS does not perform well even after adjustment for severity of illness.²¹ Likewise, the NRD does not capture detailed clinical information (i.e., patient weight, vasopressor dosing), and while it collects information on length of mechanical of ventilation, this information is often incomplete. Furthermore, it does not record out-of-hospital deaths, limiting our ability to only evaluate in-hospital mortality.

Nevertheless, these limitations highlight the difficulty in applying the HFRS to datasets of critically ill patients and further support our caution on avoiding the use of the HFRS to predict these outcomes. For now, other tools, such as the Hospital One-Year Mortality Risk Score (HOMR) may be better suited to predict in-hospital mortality in the critically ill older adult population than the HFRS.¹⁹

Clinical implications and future directions

Clinicians need to have accurate predictions of frailty and outcomes to identify patients who would benefit from early geriatric medicine referral, as well as to engage with patients and their families in shared decision-making, goals of care discussion, and end-of-life planning, and/or palliative care referral. Likewise, healthcare administrators need to have accurate estimates of the number of frail patients to plan and allocate healthcare services.

While the HFRS may have utility in non-ICU databases, our study demonstrates its limitations in critically ill patients. The mFI is a promising alternative; however, it needs further development and validation for use with ICD-10-CM codes.^15,43 Perhaps the better solution for clinicians, researchers, and administrators would be to adapt and transform existing databases for frailty research. With other well-validated frailty scores such as the CFS, there is a compelling argument for its integration into routine clinical practice and inclusion in data capture.

Conclusion

In this large nationally representative external validation study of older adults receiving mechanical ventilation, the HFRS could not be validated to predict in-hospital mortality. While the HFRS may predict prolonged hospitalization and 30-day emergency hospital readmissions, its use should be avoided in the critically ill. Further research with administrative databases is necessary to develop accurate, intuitive frailty scores in critically ill patients.

References

1.          Brummel NE, Bell SP, Girard TD, Pandharipande PP, Jackson JC, Morandi A, Thompson JL, Chandrasekhar R, Bernard GR, Dittus RS, Gill TM, Ely EW. Frailty and subsequent disability and mortality among patients with critical illness. Am J Respir Crit Care Med. 2017;196(1):64-72. doi:10.1164/rccm.201605-0939OC

2.          Bagshaw SM, Muscedere J. Is this intensive care unit patient frail?: Unraveling the complex interplay between frailty and critical illness. Am J Respir Crit Care Med. 2017;196(1):4-5. doi:10.1164/rccm.201612-2538ED

3.          Muscedere J, Waters B, Varambally A, Bagshaw SM, Boyd JG, Maslove D, Sibley S, Rockwood K. The impact of frailty on intensive care unit outcomes: a systematic review and meta-analysis. Intensive Care Med. 2017;43(8):1105-1122. doi:10.1007/s00134-017-4867-0

4.          Bagshaw SM, Stelfox HT, McDermid RC, Rolfson DB, Tsuyuki RT, Baig N, Artiuch B, Ibrahim Q, Stollery DE, Rokosh E, Majumdar SR. Association between frailty and short- and long-term outcomes among critically ill patients: A multicentre prospective cohort study. Cmaj. 2014;186(2):95-102. doi:10.1503/cmaj.130639

5.          Heyland D, Cook D, Bagshaw SM, Garland A, Stelfox HT, Mehta S, Dodek P, Kutsogiannis J, Burns K, Muscedere J, Turgeon AF, Fowler R, Jiang X, Day AG. The very elderly admitted to ICU: A quality finish? Crit Care Med. 2015;43(7):1352-1360. doi:10.1097/CCM.0000000000001024

6.          Rydingsward JE, Horkan CM, Mogensen KM, Quraishi SA, Amrein K, Christopher KB. Functional Status in ICU Survivors and Out of Hospital Outcomes. Crit Care Med. 2016;44(5):869-879. doi:10.1097/CCM.0000000000001627

7.          Sanchez D, Brennan K, Al Sayfe M, Shunker S-A, Bogdanoski T, Hedges S, Hou YC, Lynch J, Hunt L, Alexandrou E, Saxena M, Abel S, Lakshmanan R, Bhonagiri D, Parr MJ, Aneman A, Chroinin DN, Hillman KM, Frost SA. Frailty, delirium and hospital mortality of older adults admitted to intensive care: the Delirium (Deli) in ICU study. Crit Care. 2020;24(1):609. doi:10.1186/s13054-020-03318-2

8.          Brummel NE, Girard TD, Pandharipande PP, Thompson JL, Jarrett RT, Raman R, Hughes CG, Patel MB, Morandi A, Gill TM, Ely EW. Prevalence and course of frailty in survivors of critical illness. Crit Care Med. Published online 2020:1419-1426. doi:10.1097/CCM.0000000000004444

9.          Pugh RJ, Ellison A, Pye K, Subbe CP, Thorpe CM, Lone NI, Clegg A. Feasibility and reliability of frailty assessment in the critically ill: a systematic review. Crit Care. 2018;22(1):49. doi:10.1186/s13054-018-1953-9

10.       Rockwood K. A global clinical measure of fitness and frailty in elderly people. Can Med Assoc J. 2005;173(5):489-495. doi:10.1503/cmaj.050051

11.       Fried LP, Tangen CM, Walston J, Newman AB, Hirsch C, Gottdiener J, Seeman T, Tracy R, Kop WJ, Burke G, McBurnie MA. Frailty in Older Adults: Evidence for a Phenotype. Journals Gerontol Ser A Biol Sci Med Sci. 2001;56(3):M146-M157. doi:10.1093/gerona/56.3.M146

12.       Morley JE, Malmstrom TK, Miller DK. A simple frailty questionnaire (FRAIL) predicts outcomes in middle aged African Americans. J Nutr Health Aging. 2012;16(7):601-608. doi:10.1007/s12603-012-0084-2

13.       Rolfson DB, Majumdar SR, Tsuyuki RT, Tahir A, Rockwood K. Validity and reliability of the Edmonton Frail Scale. Age Ageing. 2006;35(5):526-529. doi:10.1093/ageing/afl041

14.       Clegg A, Bates C, Young J, Ryan R, Nichols L, Ann Teale E, Mohammed MA, Parry J, Marshall T. Development and validation of an electronic frailty index using routine primary care electronic health record data. Age Ageing. 2016;45(3):353-360. doi:10.1093/ageing/afw039

15.       Zampieri FG, Iwashyna TJ, Viglianti EM, Taniguchi LU, Viana WN, Costa R, Corrêa TD, Moreira CEN, Maia MO, Moralez GM, Lisboa T, Ferez MA, Freitas CEF, de Carvalho CB, Mazza BF, Lima MFA, Ramos G V., Silva AR, Bozza FA, Salluh JIF, Soares M. Association of frailty with short-term outcomes, organ support and resource use in critically ill patients. Intensive Care Med. 2018;44(9):1512-1520. doi:10.1007/s00134-018-5342-2

16.       Gilbert T, Neuburger J, Kraindler J, Keeble E, Smith P, Ariti C, Arora S, Street A, Parker S, Roberts HC, Bardsley M, Conroy S. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet. 2018;391(10132):1775-1782. doi:10.1016/S0140-6736(18)30668-8

17.       Gilbert T, Cordier Q, Polazzi S, Bonnefoy M, Keeble E, Street A, Conroy S, Duclos A, Gilbert T. External validation of the Hospital Frailty Risk Score in France. Age Ageing. Published online 2021:1-8. doi:10.1093/ageing/afab126

18.       Eckart A, Hauser SI, Haubitz S, Struja T, Kutz A, Koch D, Neeser O, Meier MA, Mueller B, Schuetz P. Validation of the hospital frailty risk score in a tertiary care hospital in Switzerland: Results of a prospective, observational study. BMJ Open. 2019;9(1):1-8. doi:10.1136/bmjopen-2018-026923

19.       McAlister F, Van Walraven C. External validation of the Hospital Frailty Risk Score and comparison with the Hospital-patient One-year Mortality Risk Score to predict outcomes in elderly hospitalised patients: A retrospective cohort study. BMJ Qual Saf. 2019;28(4):284-288. doi:10.1136/bmjqs-2018-008661

20.       Street A, Maynou L, Gilbert T, Stone T, Mason S, Conroy S. The use of linked routine data to optimise calculation of the Hospital Frailty Risk Score on the basis of previous hospital admissions: a retrospective observational cohort study. Lancet Heal Longev. 2021;2(3):e154-e162. doi:10.1016/s2666-7568(21)00004-0

21.       Bruno RR, Wernly B, Flaatten H, Schölzel F, Kelm M, Jung C. The hospital frailty risk score is of limited value in intensive care unit patients. Crit Care. 2019;23(1):8-9. doi:10.1186/s13054-019-2520-8

22.       Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement. J Clin Epidemiol. 2015;68(2):112-121. doi:10.1016/j.jclinepi.2014.11.010

23.       NRD Overview. Healthcare Cost and Utilization Project (HCUP). Published 2019. Accessed June 14, 2020. https://www.hcup-us.ahrq.gov/db/nation/nrd/nrddbdocumentation.jsp

24.       Kerlin MP, Weissman GE, Wonneberger KA, Kent S, Madden V, Liu VX, Halpern SD. Validation of Administrative Definitions of Invasive Mechanical Ventilation across 30 Intensive Care Units. Am J Respir Crit Care Med. 2016;194(12):1548-1552. doi:10.1164/rccm.201605-0953LE

25.       Ladha KS, Zhao K, Quraishi SA, Kurth T, Eikermann M, Kaafarani HMA, Klein EN, Seethala R, Lee J. The Deyo-Charlson and Elixhauser-van Walraven Comorbidity Indices as predictors of mortality in critically ill patients. BMJ Open. 2015;5(9):e008990. doi:10.1136/bmjopen-2015-008990

26.       Healthcare Cost and Utilization Project (HCUP). Agency for Healthcare Research and Quality (AHRQ). Clinical Classifications Software Refined (CCSR). Published 2020. Accessed December 31, 2020. https://www.hcup-us.ahrq.gov/toolssoftware/ccsr/ccs_refined.jsp

27.       HCUP Cost-to-Charge Ratio Files (CCR). Healthcare Cost and Utilization Project (HCUP). 2006-2009. Agency for Healthcare Research and Quality. Rockville, MD. Accessed November 14, 2019. https://www.hcup-us.ahrq.gov/db/state/costtocharge.jsp

28.       US Bureau of Labor. Consumer Price Index. Published 2019. Accessed November 14, 2019. https://www.bls.gov/cpi/

29.       Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338(jun29 1):b2393-b2393. doi:10.1136/bmj.b2393

30.       Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials – a practical guide with flowcharts. BMC Med Res Methodol. 2017;17(1):162. doi:10.1186/s12874-017-0442-1

31.       Huang Y, Li W, Macheret F, Gabriel RA, Ohno-Machado L. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Informatics Assoc. 2021;27(4):621-633. doi:10.1093/JAMIA/OCZ228

32.       Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. doi:10.1097/EDE.0b013e3181c30fb2

33.       Harrell FE. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer; 2001.

34.       Cox DR. Regression Models and Life-Tables. J R Stat Soc Ser B. 1972;34(2):187-202. doi:10.1111/j.2517-6161.1972.tb00899.x

35.       White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30(4):377-399. doi:10.1002/sim.4067

36.       Kwok CS, Zieroth S, Van Spall HGC, Helliwell T, Clarson L, Mohamed M, Mallen C, Duckett S, Mamas MA. The Hospital Frailty Risk Score and its association with in-hospital mortality, cost, length of stay and discharge location in patients with heart failure short running title: Frailty and outcomes in heart failure. Int J Cardiol. 2020;300:184-190. doi:10.1016/j.ijcard.2019.09.064

37.       Kundi H, Wadhera RK, Strom JB, Valsdottir LR, Shen C, Kazi DS, Yeh RW. Association of Frailty with 30-Day Outcomes for Acute Myocardial Infarction, Heart Failure, and Pneumonia among Elderly Adults. JAMA Cardiol. 2019;4(11):1084-1091. doi:10.1001/jamacardio.2019.3511

38.       Bonjour T, Waeber G, Marques-Vidal P. Trends in prevalence and outcomes of frailty in a Swiss university hospital: A retrospective observational study. Age Ageing. 2021;50(4):1306-1313. doi:10.1093/ageing/afaa278

39.       Hollinghurst J, Housley G, Watkins A, Clegg A, Gilbert T, Conroy SP. A comparison of two national frailty scoring systems. Age Ageing. 2021;50(4):1208-1214. doi:10.1093/ageing/afaa252

40.       Marshall D, Salciccioli J, Hatch R, Rowland M. Validation of the Hospital Frailty Risk Score in the ICU. EP. 305. J Intensive Care Soc. 2019;20 Supplem:230-231.

41.       Szakmany T, Hollinghurst J, Pugh R, Akbari A, Griffiths R, Bailey R, Lyons RA. Frailty assessed by administrative tools and mortality in patients with pneumonia admitted to the hospital and ICU in Wales. Sci Rep. 2021;11(1):1-10. doi:10.1038/s41598-021-92874-w

42.       Johnson AEW, Pollard TJ, Shen L, Lehman LH, Feng M, Ghassemi M, Moody B, Szolovits P, Anthony Celi L, Mark RG. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3(1):160035. doi:10.1038/sdata.2016.35

43.       Subramaniam A, Ueno R, Tiruvoipati R, Darvall J, Srikanth V, Bailey M, Pilcher D, Bellomo R. Defining ICD-10 surrogate variables to estimate the modified frailty index : a Delphi-based approach. Res Sq. Published online 2021:1-17. doi:10.21203/rs.3.rs-929245/v1

Declarations

Ethics Approval and consent to participate:

This study was reviewed by the Saskatchewan Health Authority Research Ethics Board (REB-20-77) and was considered exempt under the TCPS2, with a waiver of consent. 

Consent for publication:

Not applicable

Availability of data and materials:

The Nationwide Readmissions Database is available through the Healthcare Cost and Utilization Project (https://www.hcup-us.ahrq.gov/nrdoverview.jsp)

Competing Interests:

The authors declare that we have no conflict of interests.

Funding:

There were no grants/funding from any other public, private, or commercial interests.

Acknowledgements:

Not applicable

Sponsor:

Not applicable

Abbreviations

Tables

Table 1: Characteristics of the population

Abbreviations: interquartile range (IQR), standard deviation (SD)

^a Expressed as unweighted number and weighted percentage (%) unless otherwise stated. Weighted percentages were calculated using complex survey methods in Stata and used the weighted number of hospitalizations. 

^b A p-value <0.05 considered statistically significant.

^c As determined by the patient’s zip code

^d Among patient hospitalizations that survived their index admission (Unweighted total n=18,775 for low-risk, n=133,597 for intermediate-risk, n=50,610 for high-risk, n=202,982 total)

Table 2: Model performance of HFRS subcategory and outcome in mechanically ventilated older adults

Abbreviations: confidence interval (CI), hospital frailty risk score (HFRS), number (No.), odds ratio (OR)

^a Adjusted for age (continuous variable), Elixhauser-van Walraven comorbidity index score (continuous variable), do-not-resuscitate status, biological sex, insurance status, income quartile, year of study, hospital teaching status, hospital size, and admission diagnosis category. 30-day emergency readmissions include additional adjustment for hospital disposition.

^­b Total number of patient hospitalizations in analysis who survived index hospital admission.