Development of PREPARE (PREdicting PAtients’ Long-Term Outcome for REcovery): A Practically Usable Prediction Model for Quality of Life of ICU Survivors


 BACKGROUND Though numbers of Intensive Care Unit (ICU) survivors have been increasing, using data on patient-reported long-term physical and mental functioning post-ICU during ICU admission is rare. Individualised information about long-term quality of life (QoL) supports ICU physicians’ decision making and empowers patients to better manage their disease during ICU admission and recovery and rehabilitation. We aimed to develop a prediction model for ICU survivors’ change in QoL one year after ICU admission. METHODS This is a sub-study of an ongoing multicenter prospective cohort study (MONITOR-IC study), in which long-term outcomes of ICU patients are measured up to five years after ICU admission. Adult patients admitted ≥12hrs to the ICU between July 2016 and January 2019 were included. Moribund patients were excluded. Multivariable linear regression and best subsets regression analysis (SRA) were used for building the prediction model. Change in QoL after one year was quantified using the EuroQol five-dimensional (EQ-5D-5L) questionnaire (Dutch range: -0.446–1), and Short-Form 36 (SF-36, range: 0-100). Models were internally validated RESULTS Data on 1308 ICU survivors was used to build the PREdicting PAtients’ long-term outcome for Recovery, PREPARE, prediction model. The best model contained 33 predictors, using the EQ-5D. Explained variance (R2) was 58.0%. Using SRA, we reduced the number of predictors to 5 (R2=55.3%): QoL before admission; sex; Clinical Frailty Scale; a cerebral embolism, occlusion, bleeding or infarction prevalent at or within one hour of admission; and having been admitted to the ICU from the operating room, from the same hospital’s nursing ward. The prediction model using the EQ-5D to quantify QoL had better predictive performance than the best SF-36 model (R2=40.6%).CONCLUSIONS We developed PREPARE, a prediction model for ICU survivors’ QoL one year after ICU admission that is practically usable due to the small number of predictors, measurable within the first 24 hours of admission. The next step is to test and evaluate the use of this prediction model in conversations between ICU physicians and patients and their families to ultimately improve patient outcomes. TRIAL REGISTRATION ClinicalTrials.gov: NCT03246334 on August 11, 2017.

. Few models consider long-term health outcomes (11)(12)(13), let alone long-term QoL (14). Even rarer is a clear path towards its use in clinical practice. Therefore, the aim of the current study was to develop a model to predict the QoL of ICU survivors one year after ICU admission, with the clear goal of its use in clinical practice.

Study design and population
Data were obtained through an ongoing multicenter prospective cohort study (MONITOR-IC study), in which long-term outcomes of ICU patients are measured up to ve years after ICU admission (ClinicalTrials.gov: NCT03246334). ICU patients aged 16 years or older were excluded when they had been admitted to the ICU for less than 12 hours, had a life expectancy of less than 48 hours, or could not read or speak the Dutch language. The MONITOR-IC study protocol has been previously published and can be referred to for further detail (15).
For this ancillary study, data of patients who were admitted to a university hospital between July 2016 and January 2019 were used. The population included medical, planned and emergency surgery admission types. The follow-up period was one year.
The study has been approved by the research ethics committee of the Radboud University Medical Centre, CMO region Arnhem-Nijmegen (number 2016-2724). All patients, or their legal representative, provided written informed consent.

Data collection
Patients were asked to complete self-administered questionnaires regarding their health status before and one year after ICU admission. In the case of unplanned ICU admissions, patients were asked to recall their health situation before admission and complete the questionnaire retrospectively as soon as possible after admission. Patients with a planned ICU admission lled in the baseline questionnaire at the outpatient clinic before ICU admission, while patients with unplanned ICU admissions were asked to ll in the questionnaire retrospectively as soon as possible after ICU admission. E-mail and telephone reminders were used in the case of non-response for 4 or 6 weeks, respectively. After 90 days of nonresponse, patients were excluded. If patients were unable to ll in either questionnaire, this task was performed by their proxies.

Outcome measure
The outcome measure, change in QoL one year after ICU admission, was quanti ed using the EQ-5D-5L questionnaire (16), as speci cally calibrated for the Dutch population (17), and the Short-Form 36 (SF-36) questionnaire (18). The SF-36 is a staple of ICU QoL research (19). Its 36 questions are used to describe both the physical and the mental health aspects of QoL, compiled into the physical component summary (PCS) and the mental component summary (MCS). Both the PCS and MCS range from 0 -100, a higher score means a better QoL. The EQ-5D is generally used in health technology assessment studies. It consists of ve questions each highlighting different aspects of QoL. The Dutch EQ-5D-5L range is -0.446 -1 (17), a higher score means a better QoL. Both QoL questionnaires were included in order to compare the resulting models on performance (explained variance, adjusted R 2 ).

Predictors
Candidate predictors consisted of PROMs and other variables that were part of the questionnaire lled in by the patients or proxies as part of the MONITOR-IC study, and of medical patient characteristics collected in the electronic health record (EHR). Two of the PROMs used as a candidate predictor in this study were frailty, measured using the Dutch Clinical Frailty Scale (CFS) (20), and the Visual Analog Scale (VAS). Examples of other questionnaire variables are marital status and education level. All PROMs and other variables collected in the MONITOR-IC database can be found in the study protocol (15). EHR data included information about the rst 24 hours of a patient's admission to the ICU, as well as hospital and ICU length of stay. This was supplemented with the Acute Physiology and Chronic Health Evaluation IV (APACHE IV) score and physiological factors like blood pressure. All of these variables were collected by the Dutch National Intensive Care Evaluation registry (NICE: https://stichting-nice.nl/), with standardized de nitions as speci ed in their data dictionary.
A preselection of possible predictors from the MONITOR-IC database was executed based on expert opinion (21). The full list of candidate predictors can be found in the Additional Files (Additional File 1).

Statistical analysis
In order to satisfy the regression assumption of normal dependent variable distribution, the outcome measure was de ned as change in QoL after one year, i.e. QoL one year after ICU admission minus QoL before ICU admission (at baseline). To produce both a statistically sound and practically useable prediction model for ICU survivors' change in QoL after one year, multivariable linear regression analysis was applied to nd the model that best t the data, followed by best subsets regression analysis (SRA).
To assess the bias in predictions, we visualised the predicted changes in EQ-5D scores versus the actual changes in EQ-5D scores in a calibration plot, which can be found in the Additional File (Additional File 2).
Linear regression modelling: choosing the best t model To obtain the model that best t the data with respect to Akaike's Information Criterion (AIC) (22), a stepwise selection procedure was performed. This model is subsequently referred to as the best t model.
The adjusted R 2 (a-R 2 ) and mean square error (MSE) were calculated to indicate the predictive performance of the model.
Best subsets regression modelling: choosing the best practical model SRA was used to identify the best practical model for use in daily clinical practice, and to compare its performance to the best t model. We considered a 'practical' model to be easy to use and thus small in terms of the number of predictors needed, while still providing an adequate prediction performance. SRA is a means for model selection that consists of testing all possible combinations of predictor variables while restricting the number of variables allowed in the prediction model. The R function regsubsets from the R package leaps was used to identify different best models of different sizes, ranging from the best one-variable model up to the best ten-variable model (23). The best practical model was chosen based on a-R 2 differences of <0.5% in pairwise comparisons based on an increasing number of variables to balance the number of variables in the model and the increase in the amount of variation explained in the model.

Internal validation
For all models, we performed interval validation through bootstrap sampling (24). For each of the 2000 bootstrap samples, we t the model on the bootstrap dataset and predicted the outcomes on the full dataset and calculated the R 2 and MSE from these predictions. We then calculated the mean R 2 and MSE across bootstrap samples as well as standard errors (SEs). Good internal validation would result in mean bootstrap R 2 's that are similar to the model a-R 2 , and low SEs.
Complete case analysis was applied, meaning that the prediction model has explicitly been developed using survivor data.
Some additional analyses were explored and can be viewed in Additional Files 3 and 4. These include a preliminary analyses using the SF-36 questionnaire to quantify QoL and a prediction model including con rmed deaths.
The variance in ation factor (VIF) was used to identify multicollinearity in the models. Variables with VIF>5 were removed from the model.

Results
A total of 2291 patients were admitted to the university hospital between July 2016 and January 2019. Of these, 780 (34.0%) were excluded because their one-year outcome measurements were missing, either due to death (n= 184, 8.0%) or other, unknown reasons (n= 596, 26.0%), resulting in a dataset of 1511 patients, of whom 1308 (57.1%) records had no other missing non-outcome data ( Figure 1).

Baseline patients characteristics
The median age of the study population was 65.0 years (interquartile range [IQR] = 57-71) and the majority was male (67.9%). Their median QoL, as quanti ed using the EQ-5D describing the period before ICU admission, was 0.8 (IQR= 0.7-0.9). Over two thirds of the sample were admitted for planned surgery (72.7%). The median APACHE-IV score was 48.0 (IQR=38-60). All baseline study population characteristics can be found in Table 1.

Quality of life questionnaire
Two practical prediction models were developed using the SF-36, to describe the change in PCS and MCS score. The best change in PCS model had an a-R 2 = 40.6%, and the best change in MCS model had an R 2 = 38.5%, more details can be found in the Additional File (Additional File 3). The EQ-5D yielded better results in terms of predictive performance (a-R 2 = 55.3%) and was further preferred to the SF-36 due to the smaller size of the questionnaire. This means an advantage when used in clinical practice. It was therefore selected for further analysis.
While our main goal was to develop a prediction model for ICU survivors, we explored the option the EQ-5D questionnaire provides to give con rmed deaths a value of '0'. Due to the increased heterogeneity in the outcome measure variation, this yielded a lower predictive performance (a-R 2 = 35.0). See Additional File 4 for full results.
Best t model (EQ-5D) A total of 69 candidate predictors (excluding variable categories) were included in the linear regression analysis (Additional File 1). Stepwise selection resulted in a model with 34 predictors as the best tting model (lowest AIC). Minimum thrombocyte level in rst 24 hours was removed due to evidence of multicollinearity, and the model was re t with 33 predictors. The strongest predictor in this best t model was the baseline EQ-5D QoL score (β = -0.79, 95% CI: -0.83 --0.75), a higher score was associated with a lower change in QoL after one year. Other negatively associated predictors were being admitted with a cerebrovascular accident (CVA) (which includes a cerebral embolism, occlusion, bleeding or infarction prevalent before or within one hour of admission) (β = -0.11, 95% CI: -0.18 --0.04)), chronic respiratory insu ciency (β = -0.15, 95% CI: -0.24 --0.06) and a higher frailty score (β = -0.01, 95% CI: -0.02 --0.01). Male sex was associated with a higher change in EQ-5D score after one year (β = 0.05, 95% CI: 0.03 -0.07). The full summary of the best t regression model is depicted in Table 2.
Best subsets regression analysis If, through SRA, the number of predictors in the model was restricted to one, the model was most likely to include the baseline EQ-5D QoL score. The a-R 2 ranged from 51.1% for the smallest model, to an a-R 2 value of 56.2% for the largest model. The a-R 2 from all models were similar to their bootstrap mean R 2 , showing good internal validation. The predictive performance of the model in terms of a-R 2 improved by less than 0.5% when we moved from the best ve-variable model to the best six. The balance struck by the model that was restricted to ve predictors between a high mean a-R 2 , a low mean square prediction error, and a small number of relatively easily accessible variables was therefore deemed the most e cient, and was selected as the nal model. There was no evidence of multicollinearity, with variance in ation factors <2 for all variables across the 10 models. The full summary of the best subsets regression analysis can be found in Table 3.

Best practical model analysis
The ve predictors most likely to be included in the best subsets model restricted to ve predictors were studied with linear regression analysis. Baseline EQ-5D QoL score was negatively associated with the outcome (β = -0.76, 95% CI: -0.81 --0.72), as were frailty score (β = -0.02, 95% CI: -0.03 --0.01) and having been admitted with a cerebral vascular accident (CVA) (β = -0.14, 95% CI: -0.20 --0.07). Male sex and being admitted from the operating room from the nursing ward within the same hospital were positively associated with the outcome (β = 0.05, 95% CI: 0.03 -0.07) and β = 0.07, 95% CI: 0.05 -0.10), respectively. This best practical model, named PREPARE (PREdicting PAtient's long-term outcome for REcovery) had an a-R 2 of 55.3%. A visual demonstration of the best practical model can be viewed in Figure 2. The red dot demarcates the predicted change in QoL after one year for a ctional patient.
The calibration plot shows that the relationship between the predicted changes in EQ-5D and the actual changes in EQ-5D is linear with slope = 1 (95% CI: 0.95 -1.05) and intercept = 0 (95% CI: -0.01 -0.01), indicating that the model is well calibrated. The calibration plot can be viewed in the Additional Files (Additional File 2).

Discussion
We have developed a prediction model for change in ICU survivors' QoL one year after ICU admission that has both considerable predictive value and is well calibrated, as supported by internal validation. The small number of predictors, all available within the rst 24 hours of admission, increases the feasibility of using the PREPARE model in clinical practice. Moreover, the prediction model based on the lessburdensome ve-question EQ-5D questionnaire had better predictive value than the longer SF-36 QoL questionnaire, which is more commonly used in the ICU (19).
Prediction models focusing purely on quality of life, and not on mortality, are rare in critical care. A timely and continuous conversation about long-term outcomes is an indispensable part of future intensive care medicine for expected ICU survivors. Using a prediction model as an enrichment to ICU professionals' knowledge enables them to provide patients with a more objective prediction of their long-term outcomes. This in turn empowers ICU patients and their families to be more involved in clinical decision-making and helps to manage their expectations for their long-term recovery, facilitating an individualized, preparatory approach instead of a reactionary one.
As far as we are aware, only Oeyen et al. have developed a series of increasingly practically usable prediction models for the general ICU population's long-term QoL (14). Although our overall conclusions are similar, the major difference between our studies is the inclusion of patients who died in the year after admission, an approach we rejected due to our focus on good survivorship. We also had more planned surgical patients and we incorporated more physiological patient characteristics within the rst 24 hours of admission. We consider this study an elaboration on their foundational work, with a focus on good survivorship and a proposed method of prediction model visualisation for incorporation of the results in clinical practice.
Past implementation of prediction modelling in ICU practice has been impeded by doubts about model accuracy (14,36). Even though any prediction for long-term health can only ever be used as a supplement to ICU professionals' knowledge and experience, these fears are understandable. Though we consider the predictive performance of our PREPARE model to be more than acceptable, medical professionals familiar with prediction modelling might think our model performance inferior to the predictive performance measurements of other oft used prediction models in ICU practice (SAPS III aROC: 0.86 (10), APACHE IV aROC: 0.88 (9) predict patients' chance to die in hospital). Considering the di culty of predicting a long-term PROM such as QoL, we deem the current achieved model performance based only on variables available within the rst 24 hours of admission to be a substantive step in the right direction. All the same, the CIs of our model predictions are still wide, and the model performance can be improved upon. For instance by using a larger pool of patient data by extracting more detailed patient data from the EHR, instead of only data from the rst 24 hours. Whether inputting new data pertaining to the whole ICU admission will make the model less practical remains to be seen. Perhaps the gains in model performance will be su ciently large to offset this potential setback.
Like other studies detailing the prediction of physical long-term health post ICU admission, we have shown that among our predictors the pre-ICU QoL is by far the most important predictor for long-term QoL (used as the sole predictor a-R 2 = 51.1%) (37)(38)(39). Conversations about pre-ICU health status are already an important part of many ICU physician's manner of providing care. Doing so with the added help of a more standardized set of questions and data on hundreds of former patients will only make these conversations more valuable and objective.
Talking about both the opportunities and the disadvantages of an ICU admission in a frank manner is important to prepare patients and families for the road ahead (5). Stressing the importance of pre-ICU health can be an important source of comfort and preparedness. Our study is new evidence supporting the importance of all aspects of pre-ICU health in ICU treatment decision making. The signi cance of pre-ICU health status extends to another one of our ndings: the relative unimportance of age. Age is generally one of the most crucial factors considered when performing triage (40,41). While it was associated with our outcome in the full model, our results show that baseline QoL and frailty score are more important predictors of long-term QoL than age, which is supported by literature (42). This is a large prospective cohort study based on ICU patients' past and long-term health. Though second of its kind, it demonstrates the possibility of achieving a good predictive performance combined with a reduction of the number of predictors needed, compared to its predecessor in purpose (14).
This study has some limitations. There is a possible bias in the relatively healthier ICU patient group that agreed to participate in the MONITOR-IC study, and by extension, are included in the prediction model. More than two thirds of the study population were admitted due to planned surgery. Generally, these patients are more prepared for an ICU stay and have better outcomes. Even though our prediction model is for ICU survivors only, this sample may be slightly better off than the general ICU survivor population. This could have caused our predictions to be slightly overoptimistic. Moreover, though proxies lling in the questionnaire when patients were unable to might have introduced a small underestimation of their QoL scores (43), we believe that excluding these patients would have led to bias as well (44). We consider our choice to use complete case analysis on ICU survivors to be best due to the improved model performance, though we realize that it is not always clear who will survive long-term. However, we still believe that the new information the model provides will be a valuable contribution to the improved ICU care of today, with its shifting focus towards good survivorship.
Lifting a model from the pages of theory and putting it into clinical practice is not an easy feat. In order to further model implementation, the next step is external validation. While the overrepresentation of planned ICU admissions in our sample might pose a challenge, the small number of predictors needed for the model has caused us to consider this an achievable goal. A pilot study using the prediction model in existing conversations between ICU physicians and patients and their families is conducted at the moment. In this pilot study, ICU physicians are asked to judge patients' likeliness of long-term survival. If physicians are con dent in this matter, the change in QoL prediction is introduced into a conversation with family, in order to start looking ahead to the future while still on the ICU. If both the external validation, using data from secondary centers to further improve model generalisability, and the pilot study provide conclusive evidence to the model's value in practice, a large trial will be conducted to evaluate the effect of using QoL data in ICU treatment on patient outcomes and experiences.

Conclusions
In conclusion, we have developed the PREPARE model, a prediction model for ICU survivors' change in QoL one year after ICU admission that is usable in clinical practice due to the small number of predictors, all available on the rst day of admission. The explained variance of 55.3%, as supported by internal validation, marks the model as a potential helpful complement to ICU professionals' knowledge and expertise in conversations with patients and their families. In future, long-term QoL information should be integrated in ICU care to better inform ICU decision making and to aid ICU survivors in managing their long-term health expectations.

Consent for publication
Not applicable.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors declare that they have no competing interests.

Funding
Funding for this study was acquired through Zorginstituut Nederland to last author Dr. Marieke Zegers (2018026879). Zorginstituut Nederland was not involved in the design of the study, nor with the data collection, analysis, interpretation or writing of the manuscript.
Authors' contributions NW, MvDB, HvdH and MZ contributed substantially to the conception of the work. Each author contributed substantially to the design of the work. NW, MvdB and MZ contributed substantially to the acquisition of the data. NW and JR contributed substantially to the analysis of the data. Each author contributed substantially to the interpretation of the data. NW drafted the manuscript. Each author contributed substantially to the revision of the work.
Each author approved the submitted version. Each author agrees to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.   One previous admission to the ICU -0.05 -0.09 --0.01 0.02 Table 3. Best subsets regression analysis summary