Detecting Potentially Ineffective Care in Critically Ill Patients in a Community Hospital: Assessing Sustained Predictive Value after Two Decades

Background: Eighteen years ago, we derived a formula to predict 100-day post-discharge mortality, utilizing the Acute Physiology and Chronic Health Evaluation III (APACHE III) data. This study was designed to reassess this formula when applied to a new cohort of patients, utilizing the updated predictive hospital mortality equations derived from APACHE IV. Methods: Compared with the 1995 ‒ 1997 cohort in our original study, this study included a cohort of intensive care unit patients from 2012 ‒ 2017, with similar demographics. Both cohorts included patients >18 years old admitted to and surviving at least ve days of intensive care in the Sarasota Memorial Hospital in Sarasota, Florida, USA. Results: In the recent cohort, the formula exhibited a specicity of 99.7%, sensitivity of 17.8%, false positive rate of 0.3%, and positive predictive value of 92.6%; applied to the original cohort, the formula exhibited values of 98.7%, 33.8%, 1.3%, and 93.3%, respectively. There was no statistical difference between the two databases, except in sensitivity. Conclusions: Potentially ineffective care can be predicted with nearly the same specicity and predictive value using the formula developed in the 2002 study. If these results are reproducible at other institutions, they could assist in patient/family and palliative care discussions.


Background
Acute Physiology and Chronic Health Evaluation (APACHE) is a prognostic evaluation system developed in 1981 for assessment of disease severity in intensive care units (ICUs) [1]. Currently, this system is in its fourth iteration. A higher APACHE score indicates higher disease severity. In 1995, Esserman et al. utilized data from APACHE III to derive a formula to predict mortality and avoid potentially ineffective care (PIC).
In this study, Esserman et al. developed the concept of PIC to refer to a combination of high resource utilization (top 25% of total hospital charges) and limited inpatient survivability within 100 days posthospital discharge [2]. The main purpose of this concept was to generate a model to identify patients with poor prognoses (speci cally, mortality) despite intensive treatment and high resource utilization.
In 2002, Fleegler et al., utilizing a community hospital database from 1995-1997, used a retrospectively derived formula implementing longitudinal and individual-speci c APACHE III hospital mortality risk prediction (APHM) to predict 100-day post-discharge mortality [3]. The retrospectively derived model was Afessa et al. [4] observed that if an Acute Physiology Score (APS) on the third day of the patient's admission into the ICU (APS3) was greater than that on their rst day in the ICU (APS1), and the APACHE III-predicted mortality was ≥ 80% on day 1, the patient would expire within 100 days post-discharge. Their study was limited to patients treated in the ICU for three days or more, who had an APACHE-predicted mortality of ≥ 80%. To the best of our knowledge, no other study has evaluated 100-day mortality using APACHE. More recently, some have suggested it is time for a re-evaluation of PIC and have proposed PIC as an outcome measure to evaluate quality of critical care delivery [5].
The fourth iteration of APACHE (APACHE IV), published in 2006, allows improved accuracy and inclusion of patients with a history of coronary artery reconstruction [6,7]. APACHE IV exhibits excellent discrimination (area under receiver operating characteristic curve = 0.88) and hospital mortality prediction (13.51%) statistically identical to that observed in the validation data set (13.55%; p = 0.76). The difference between the observed and mean predicted hospital mortality across risk deciles is 0.1-0.4%, except for the 70-80% decile (1.1%) and the 90-100% decile (1.6%). For most subgroups, the ratio of observed to mean predicted hospital mortality is near 1.0, and 90% of the standardized mortality ratios (SMR) within disease groups are not signi cantly different from 1.0. APACHE and APACHE IV have been considered a useful tool for decision-making and to benchmark performance across ICUs in the United States [3,7].
To the best of our knowledge, no studies have examined PIC prediction over 20 years utilizing the same predictive equation. This study was designed to examine reproducibility of the PIC model using APACHE IV, applying the same formula we derived in 2002, with a new cohort at the same medical institution almost two decades later [6].

Methods
This retrospective observational cohort study tested the hypothesis that the performance-sensitivity, speci city, false positive rate, and positive predicted value (PPV)-of a PIC model developed using a cohort of patients obtained from July 1995-June 1997 (using APACHE III) would be statistically equivalent when using APACHE IV-based APHM in a cohort of patients from July 2012-June 2017 [3].
A patient consent waiver was obtained from the Institutional Review Board as part of a continuing review amendment to include a validity replication cohort.
The sample included patients > 18 years old admitted to and surviving at least ve days of intensive care in medical-surgical (32 beds) and cardiac-surgical (20 beds) units at the 832-bed publicly owned acute care Sarasota Memorial Hospital, Sarasota, FL (the same institution as in our 2002 study).

Data collection
Data collected included age; APS1 and APS5, as well as APHMs; ICU and hospital discharge status; and vital status for the 100 days post-discharge. Inpatient data were obtained from the APACHE IV Outcomes Database (Cerner Corporation Kansas City, MO). If patients had more than one ICU stay during their admission, data from the rst hospital stay was used for this study. Our 2002 study utilized the last ICU admission on a given hospitalization since it was closest to the 100-day endpoint [3]. However, unlike APACHE III, APACHE IV did not include an adjustment co-e cient for readmissions. Therefore, only the initial ICU admission was recorded and utilized for the current data set.
To establish vital status between ICU day 5 and 100 days post-discharge, multiple methods were employed.
Patients were followed up from hospital admission until discharge to ascertain mortality between ICU day 5 and hospital discharge. Mortality between discharge and 100 days post-discharge was established in one of two ways.
Demographic data, including medical record numbers, were used to identify subsequent hospital admissions after incident ICU admission. This permitted tracking outcomes for patients treated subsequently or who died within the institution. Patient mortality was con rmed using an Obituaries Database (http://www.legacy.com) or the National Death Index, a centralized database of death record information led with the National Center for Health Statistics. Patients were excluded from analysis if vital status could not be established within 100 days post-hospital discharge.

Replication of PIC model
We applied the previously derived formula to evaluate the PIC model's performance using APACHE IVbased APHM [3]. Mortality within 100-days post-hospital discharge was predicted if the patient had either an APHM5 ≥ 0.92 or if the product of APHM1×APHM5 was ≥ 0.40 and there was an increase in APHM ≥ 0.10 between day 1 and day 5 after ICU admission.

Analytic strategy
The predictive performance of the PIC model was quanti ed according to several parameters, including sensitivity, speci city, false positive rate, and PPV. For the current dataset, data were pooled from 2012 to 2017 due to the small number of ICU patients that met the PIC criteria in a given year. The PIC model's performance was compared using the same formula, albeit adapted to APACHE III (1995-1997) and APACHE IV (2012-2017) hospital mortality predictions. Similarly, SMR tendencies were compared between APACHE III and APACHE IV. SMRs were computed as the actual mortality divided by estimated mortality, utilizing APACHE III and APACHE IV. Continuous variables, summarized as means (standard deviation), were compared through an analysis of variance. Categorical variables, summarized as counts/proportions (interquartile range), were compared using a chi-square test. Statistical analyses were performed using SAS 9.4 software (Cary, NC). A two-tailed p < 0.05 was considered signi cant.

Results
The updated cohort comprised 13,585 patients admitted to the ICU between July 1, 2012 and June 30, 2017. A total of 2,590 patients met the threshold requirement of a 5-day ICU stay for ICU day 1 and day 5 predictive criteria. Eighteen patients who were in the ICU for 5 days did not have a social security number and were excluded from the analysis. Patients who met both the PIC criteria were only counted once. Due to the small number of patients who met the PIC criteria in the updated cohort, a ve-year sample was chosen to enable su cient evaluation.
Demographics for the 1995-1997 cohort, as shown in Table 1, are shown separately in the original publication [3]. There was no statistically signi cant difference between both cohorts regarding APS, APHM, or SMR.   Abbreviations: APACHE, Acute Physiology and Chronic Health Evaluation Table 3 shows the results of the statistical analysis of the formula for predicting PIC, indicating the speci city, sensitivity, false positive rate, and PPV for both cohorts. The only difference that was statistically signi cant was the sensitivity parameter: 33.8% for the 1995-1997 cohort and 17.8% for the 2012-2017 cohort.
To the best of our knowledge, this is the rst study to evaluate the predictive value of PIC using the same formula after almost two decades utilizing updated APACHE predictions. It is noteworthy that the PPV of this PIC model utilizing APACHE III (93.3%) was sustained when applied to a more recent cohort utilizing APACHE IV (92.6%). These mortality predictions were in a replication population with a similar SMR (0.88) compared to the earlier cohort (0.91) [3].
For decisions regarding mortality prediction, a high speci city is required for use in family discussions regarding possible prognoses and outcomes. The high sensitivity of the predictive equation was maintained in the current study, in spite of the considerable time gap. Given the high speci city required for potential decision-making, a low sensitivity is expected. There was a signi cantly lower sensitivity in the updated database (from which we obtained the 2012-2017 cohort) compared to the original database (wherein we obtained the 1995-1997 cohort). The low sensitivity partially explains the smaller number of patients identi ed in any one-year period.
Since the PIC model was developed [3], many clinical features potentially in uencing mortality risk and end-of-life management have undergone change, including (a) continued modi cations in ICU design and process, with a focus on shared decision-making; (b) the progressive aging of the general population; (c) the changing of the cultural milieu regarding the process of dying (7); and physicians' estimate of intensive care bene ts [8][9][10].
In 2006, APACHE III was updated to APACHE IV. This update improved its accuracy and included patients with a history of coronary artery reconstruction [6,7]. APACHE IV exhibited excellent discrimination and hospital mortality prediction that was statistically identical to that observed for the validation data set (13.55%). Although it is expected that the updated prediction equation would maintain sensitivity, it is possible that the difference in sensitivity between the original and updated PIC prediction equation is related to the use of APACHE IV. The sensitivity in the current cohort is consistent with an earlier study [4] that used APACHE III APS3 to predict PIC, which demonstrated a sensitivity of 0.15, similar to that in our study (0.11). The lack of a formal palliative care service 18 years ago might explain the reduced sensitivity in the current data set. Nevertheless, we have demonstrated that by utilizing both APACHE IV APM1 and APM5 data, 100-day mortality can be predicted with a high PPV over a prolonged period at a single institution utilizing the same formulas, which had been derived from APACHE III on a cohort two decades earlier.
Widely employed prognostic models for assessing overall severity of illness in critically ill adults include APACHE, the Simpli ed Acute Physiology Score (SAPS), and the Mortality Probability Model (MPMoIII While our analysis represents mortality over a 100-day period, six-month morbidity and mortality have been analyzed for ICU patients receiving life-sustaining therapy. Factors that were associated with not returning to baseline included a higher APACHE III score and being non-white, elderly, recently hospitalized, with a history of transplantation, cancer, or neurologic or liver disease [12]. Some authors have also researched the frequency and cost of treatment perceived to be futile, but the point at which treatment is deemed futile has varied [13]. We utilized a 100-day mortality point, which could provide families with a concrete reference point.
Empirically based models estimating PIC in critically ill patients remain scarce. Furthermore, there has been a trend to embrace palliative care services. Clinical trends encouraging collaborative ICU providerpatient/surrogate decisions [14][15][16]  Integrating palliative with intensive care allows preferences of patients and families to be transparent, while also fostering dignity [15]. This integration of palliative care with ICU treatment also reduces the length and costs of hospital stay [10]. Patients who receive palliative care consultations are more likely to have a code status change, be transferred to hospice, and be treated with comfort measures [17,18]. Multiple studies have encouraged collaborative ICU provider-patient/surrogate decisions [13][14][15][16]. Since the PPV has statistical precision representing the "prospective" discriminant utility of PIC modeling, it could be used to complement the contextual, continuous, and evolving process of clinical decisionmaking [17]. This model may not have been pursued in the past due to the emphasis on autonomy. However, newer clinical approaches to the concept of autonomy suggest data from this model could be utilized in collaborative discussions [19,20].

Limitations
In this study, our PIC model had a signi cantly lower sensitivity, compared with the original study. The current study also had the limitation of being a single institution study.

Conclusion
In this study, we reassessed a formula to predict 100-day mortality using APACHE IV in a community hospital in Sarasota, FL. Originally, we used this formula on a 1995-1997 cohort using APACHE III in the same hospital. Our ndings showed that the formula has sustained predictive value, 18 years after we developed it. To the best of our knowledge, this is the rst study to evaluate the predictive value for PIC using the same formula at an interval of almost two decades. If replicated in other institutions, these predictive equations could contribute to ICU management shared decision-making models of care.

Declarations
Ethics approval and consent to participate A patient consent waiver was obtained from the Institutional Review Board as part of a continuing review amendment to include a validity replication cohort. The sample included patients > 18 years old admitted to and surviving at least ve days of intensive care in medical-surgical (32 beds) and cardiac-surgical (20 beds) units at the 832-bed publicly owned acute care Sarasota Memorial Hospital, Sarasota, FL (the same institution as in our 2002 study).