Validation of the Hospital Frailty Risk Score among older adults receiving mechanical ventilation

Background: Older adults are increasingly being admitted to intensive care units, with frailty recognized as a risk factor for worse outcomes. The Hospital Frailty Risk Score (HFRS) was developed for use in administrative databases of older adults, but it has not yet been well-validated for critically ill patients. The objective of this study was to validate the HFRS to predict prolonged hospitalization, in-hospital mortality, and 30-day emergency hospital readmissions in critically ill patients. Methods: We selected index hospitalizations of older adults ( ≥ 75 years old) receiving mechanical ventilation, using the United States Nationwide Readmissions Database from January 1, 2016 to November 30, 2018. Frailty risk was determined by the HFRS using International Classication of Diseases, Tenth Revision Clinical Modication (ICD-10-CM) codes, and further subcategorized into low (score <5), intermediate (score 5-15), and high (score >15) risk for frailty. We evaluated the HFRS to predict prolonged hospitalization, in-hospital mortality, and 30-day emergency hospital readmissions, using multivariable logistic regression after adjustment for patient and hospital characteristics. Model performance was assessed using the c-statistic, Brier score, and calibration plots.


Introduction
Older adults (≥75 years old) are increasingly being admitted to intensive care units (ICUs). [1][2][3][4] In the United States (US), there were close to 27.8 million hospitalizations of older Medicare bene ciaries admitted to ICUs between 1996-2010. 5 Older adults represented close to 30% of all ICU admissions to Veterans Affairs hospitals during March-August 2020 of the coronavirus-2019 (COVID- 19) pandemic. 6 Despite critical care interventions, these patients are at-risk for death, readmission, and poor functional outcomes. 1 In a systematic review of older adult critically ill patients, the 6-month mortality and 1-year mortality of this population ranges from 21-58% and 33-72%, respectively. 7 It has been increasingly recognized that frailty is an independent risk factor for mortality, prolonged hospitalization, readmission, and poorer quality of life after discharge. 1,[8][9][10][11][12][13] It has been estimated that up to 24% of critically ill patients may be frail at baseline prior to ICU admission. 14 Thus, there is a need to better understand and characterize frailty in the critically ill older adult to engage in shared decision-making with patients and their families, inform them on post-ICU outcomes, and conduct research. Several tools have been developed to measure and characterize frailty depending on the conceptual framework of frailty, including phenotypic or cumulative de cit models. 15,16 Most studies in critical care have prospectively assessed frailty using the Canadian Study of Health and Aging Clinical Frailty Score (CFS). 11,14,17,18 However, established frailty scores, such as the CFS, Fried's frailty phenotype, Edmonton Frail Scale, or FRAIL scale, have limited use in administrative databases, as most databases do not collect the variables necessary to calculate these scores. 15,[18][19][20] Additionally, frailty assessments in the critically ill have been limited by feasibility and reliability due to the inherent nature of the population, often requiring input from family members, caregivers, and/or proxies. 17 As a result, there is an interest in the development and validation of frailty scores, using universally collected information in administrative databases. The electronic Frailty Index (eFI) was designed for use primary care electronic health records; however, many hospital administrative databases do not contain the laboratory information needed to calculate the eFI. 21 Other electronic frailty indices, such as the Veterans Affairs Frailty Index (VA-FI), Kim index, and Davidoff index, require additional codes speci c to US databases, including Current Procedural Terminology (CPT) and Healthcare Common Procedure Coding System (HCPCS) coding, limiting their generalizability to non-US databases. 22,23 A study of Brazilian ICUs assessed frailty using the modi ed Frailty Index (mFI); however, the measurement of functional capacity is not commonly captured or coded in most administrative databases. 24 Recently, the Hospital Frailty Risk Score (HFRS) was developed using International Classi cation of Diseases, Tenth Revision Clinical Modi cation (ICD-10-CM) codes, and it has been validated to predict the risk of mortality, prolonged hospitalization, and 30-day emergency hospital readmissions in older hospitalized patients. 25 It has since been externally validated in other hospitalized patient populations, 26-29 but its value in critically ill patients has been questioned in a single center study. 30 Therefore, there is a need to externally validate the use of the HFRS in large, generalizable administrative databases of critically ill patients. The primary goal of this study was to externally validate the HFRS among a nationally representative sample of older adults receiving mechanical ventilation.

Methods
This study was reported in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement. 31 As the Nationwide Readmissions Database (NRD) contains de-identi ed information, this study was granted an exemption by the Saskatchewan Health Authority Research Ethics Board (SHA-REB-20-77).

Data source
We obtained hospitalization information from the NRD between January 1, 2016 and November 30, 2018. The NRD is the largest all-payer US readmission database from the Healthcare Cost and Utilization Project (HCUP), including insured and uninsured patients. 32 It is constructed from state databases from 28 different states, and it samples >15,000,000 unique hospitalizations annually, representing >36,000,000 weighted hospitalizations. 32 The NRD accounts for 60% of the total US population and 59% of all hospitalizations, allowing for national estimates of readmissions. Linked visits can be identi ed through a linking variable in the database.

Study population
We included all hospitalizations of older adults (≥75 years old) receiving mechanical ventilation, as de ned by the ICD-10 procedure coding system (ICD-10-PCS) codes 5A1935Z, 5A1945Z, 5A1955Z, and 0BH17EZ. A prior equivalent ICD-9 coding algorithm had an overall 47.0% sensitivity and 99.6% speci city for detecting mechanical ventilation in all patients, but a sensitivity of 94.0% and speci city of 99.3% in medical patients speci cally. 33 We excluded hospitalizations of patients <75 years old, as the HFRS was originally derived from a population of adults ≥75 years old. We additionally excluded patients that left against medical advice and hospitalizations with missing information for length of stay, time to next visit, and December admissions, as the NRD would be unable to accurately follow these patients longitudinally beyond the calendar year. We also excluded hospitalizations of non-residents of the state, as the NRD does not have any linking state identi ers.

Measurements
The covariates in the NRD included age, biological sex, hospital characteristics (teaching status, size), income quartile, primary insurance status (Medicare, Medicaid, private insurance, self-pay, or other), Elixhauser-van Walraven comorbidity index score, 34 and comorbidities. The ICD-10-CM and ICD-10-PCS codes used to classify comorbidities are described in the Electronic Supplementary Material (ESM) eTable 1. We determined the primary reason for admission of the index hospitalization and readmission, using aggregate groups of the Clinical Classi cations Software Re ned (CCSR) developed by HCUP (ESM eTable 2). 35 The CCSR was developed to provide clinically meaningful categories of diagnoses, based on ICD-10-CM codes. Hospital costs were determined using total hospital charges multiplied by the all-payer cost-to-charge ratio. 36 Hospital costs were in ation-adjusted to 2018 US dollars (USD) using the US Bureau of Labor Consumer Price Index for medical care. 37

Frailty risk
We assessed the risk of frailty using the HFRS developed by Gilbert et al (ESM eTable 3). 25 The HFRS was derived from a development cohort of over >20,000 older adult patients, using 109 ICD-10 codes and ridge regression. The HFRS has been previously validated for use in hospitalized patients. 25 We classi ed patients as either low (score <5), intermediate (score [5][6][7][8][9][10][11][12][13][14][15], or high (score >15) risk for frailty, based on the original HFRS study and subsequent validation studies. [25][26][27] Outcome(s) We evaluated prolonged hospitalization, in-hospital mortality, and 30-day emergency hospital readmissions. We de ned prolonged hospitalization as a length of stay >10 days in hospital and only evaluated 30-day emergency hospital readmissions, similar to Gilbert et al. 25,26 In-hospital mortality included all-cause mortality at any point during patient hospitalization. We chose this outcome instead of 30-day mortality because the NRD only measures in-hospital mortality.

Statistical analysis
All statistical analyses were performed using Stata/MP 15.1 (College Station, Texas, USA). A two-sided pvalue <0.05 was considered statistically signi cant for all analyses. We accounted for the complex sampling design of the NRD using sampling weights provided by HCUP. 32 Descriptive statistics were presented for the index hospitalization by HFRS subcategory. Categorical variables were presented as unweighted numbers and weighted percentages. Continuous variables were presented as either means (standard deviation [SD]), or medians (interquartile range [IQR]), following testing for normality. Surveyspeci c Rao-Scott tests were used to compare nominal data. Survey-speci c linear regression was used to compare continuous data, using the geometric means for data without a normal distribution. Missing data was present in <1.5% of all patient visits.
We assessed the validity of the HFRS for predicting in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission, using unadjusted and adjusted logistic regression. For inhospital mortality and prolonged hospitalization, we performed adjustment for age, biological sex, income quartile, insurance status, do-not-resuscitate status, admission diagnosis, hospital characteristics, and year. For 30-day emergency hospital readmissions, we performed adjustment for the same variables, also including hospital disposition. Model discrimination was assessed with the c-statistic and calibration with the Brier score. 38,39 Calibration plots additionally were constructed.

Sensitivity analyses
We performed several sensitivity analyses to assess the robustness of our ndings. First, we re-evaluated our ndings using the HFRS as a continuous variable and using restricted cubic splines with ve knots. 40 Next, we performed survey-speci c Cox proportional hazards regression for in-hospital mortality and 30-day emergency hospital readmissions. 41 We then performed multiple imputation with chained equations, using 10 imputations, and repeated the primary analysis with the imputed dataset. 42 Subsequently, we derived 30-day in-hospital mortality, using hospitalization data from the NRD, and re-performed our primary analysis. Finally, we evaluated the total population of older adults in the NRD, independent of receipt of mechanical ventilation, to determine whether our ndings hold for the whole population.

Assessment of model performance
Model performance was assessed for in-hospital mortality, prolonged hospitalization, and 30-day emergency hospital readmission (  (Figure 2) visually demonstrate good calibration of the adjusted models.

Sensitivity analyses
Our sensitivity analyses are detailed in the ESM eTable 6-eTable 11. Re-analyzing our data using the HFRS as a continuous variable (ESM Table 6) or using restricted cubic splines with ve knots (ESM Table  7, eFigure 2, eFigure 3, and eFigure 4), yielded similar results for model t and calibration. We found similar results again when using a Cox proportional hazards multiple variable regression (ESM eTable 8), performing multiple imputation with chained equations (ESM eTable 9), and evaluating 30-day in-hospital mortality (eTable 10). When we evaluated a population of all older hospitalized adults from the NRD independent of receipt of mechanical ventilation (ESM eTable 11), we found that the HFRS predicted inhospital mortality, prolonged hospitalization, and 30-day hospital readmissions with a higher degree of discrimination and accuracy.

Discussion
We described the use of the HFRS to predict adverse outcomes in a large nationally representative US cohort of older adults receiving mechanical ventilation. Among this cohort, we found that the intermediate and high risk for frailty groups, as categorized by the HFRS, were associated with increased risks of prolonged hospitalization and 30-day emergency hospital readmissions, compared to the low risk for frailty group. However, they were inversely associated with in-hospital mortality when compared to the low risk for frailty group. Overall, the HFRS had moderate discrimination and accuracy in predicting these outcomes. The use of the HFRS either as a continuous variable or with splines did not provide additional value over using the HFRS subcategories of low, intermediate, and high risk.

Comparison with previous studies
Prior studies of the HFRS focused on validating its use in older adults admitted in hospitalized settings, demonstrating good calibration and discrimination. 26-28, [43][44][45][46] Recently, there has been interest in validating the HFRS in the use of administrative databases of critically ill patients. 30,47,48 A prior study of 1,498 patients in a tertiary German ICU evaluated the use of the HFRS for a combined endpoint of mortality and risk of readmission, and they found no association after adjustment for severity of illness scoring. 30 In a large population study of patients with pneumonia in Wales, Szakmany et al found that the HFRS had only moderate ability for predicting inpatient, 6-month, and 1-year mortality in hospital and ICU patients. 48 Conversely, a study of 12,854 patients from the Medical Information Mart for Intensive Care (MIMIC-III) database found that higher HFRS was associated with an increased risk of 28-day mortality. 47,49 In our study, we found that >40% of our cohort died in-hospital. Prior studies of critically ill patients have associated frailty with increased risks of mortality. 10,11 While the HFRS performed well on a patient population of all older adults independent of mechanical ventilation status, it did not perform as well in a population of older adults receiving mechanical ventilation. Counterintuitively, we found that the HFRS was inversely associated with mortality in the NRD (i.e., lower HFRS was associated with the highest risks of in-hospital mortality). There may be some explanations for this phenomenon, including potential selection bias of patients admitted to ICUs (i.e., frail patients with higher severity of illness may choose not to undergo mechanical ventilation) and/or coding biases.
Unlike prior studies, we evaluated the validity of the HFRS to predict prolonged hospitalization and 30-day emergency hospital readmissions. Patient hospitalizations with higher HFRS were associated with prolonged hospitalizations and higher risks of 30-day readmissions; however, the HFRS only had moderate discrimination and accuracy to predict these outcomes. Our ndings and other prior studies would suggest that the HFRS should be used with caution in administrative datasets of critically ill patients until better models or prediction scores of frailty can be developed speci cally for use in this patient population.

Strengths and Limitations
Our study had several strengths including the use of a large dataset, comprising close to 650,000 weighted hospitalizations of older adult patients receiving mechanical ventilation. To our knowledge, our study represents the rst study examining the use of the HFRS in a large representative study, allowing for generalizability to all older critically ill patients receiving mechanical ventilation. Additionally, we assessed both model discrimination and calibration, allowing for con dence in the results presented.
However, our study has some limitations. Coding biases may affect the relative prevalence of admission comorbidities, diagnoses, and treatments. The HFRS itself is derived from a composite of ICD-10-CM codes, which may be prone to coding biases. Patients that had prolonged hospitalization and/or survived their hospitalization may appear to be more "frail," if they have more ICD-10-CM secondary diagnoses. These coding biases could potentially explain the results seen for in-hospital mortality. While the HFRS may have utility in administrative databases, the HFRS is di cult to apply and calculate at the bedside, and it may not intuitively classify the risk of in-hospital mortality in all patient populations.
The mFI is a promising alternative; however, the mFI still needs further development and validation for use with ICD-10-CM codes. 24,51 Furthermore, it has become increasingly recognized that frailty may exist in younger critically ill patients. 8,9 Future research should be directed at developing frailty prediction scores that can be applied to a broad population of critically ill patients.

Conclusion
In this large nationally representative sample of older adults receiving mechanical ventilation, the HFRS may predict prolonged hospitalization and 30-day emergency hospital readmissions. Further research with administrative databases is necessary to develop accurate, intuitive, easy to use frailty scores in critically ill patients that predict their outcomes. was considered exempt under the TCPS2, with a waiver of consent. Figure 1 Flow chart of hospitalizations of older adults receiving invasive mechanical ventilation.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. Graphicalabstractv1.pptx