On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation

doi:10.21203/rs.3.rs-3938204/v1

Download PDF

Research Article

On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation

https://doi.org/10.21203/rs.3.rs-3938204/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background.

Time-to-event dynamic predictions are defined as the probability to survive until a defined time horizon given being event-free at landmark times and given available predictive variables at such prediction times. From two different mathematical formulations, dynamic predictions can either predict the survival probability until a final time horizon or until the end of a sliding horizon window. We aim to illustrate the need to clearly define the time horizon to correctly interpret the prognostic performances.

Methods.

First, following the PRISMA, CHARMS and TRIPOD recommendations, we conducted a systematic review of articles concerning dynamic predictions to assess how the time horizon was reported in the literature. Second, using a sample of 2,523 kidney recipients, we assessed the prognostic capacities of the Dynamic predictions of Patient and kidney Graft survival (DynPG) using either a final time horizon or a sliding horizon window.

Results.

Of 172 references retrieved about dynamic predictions, 102 articles were included in the systematic review. We notably observed that 71 (69.6%) used a sliding horizon window to assess the prognostic performance while 18 (17.7%) used a final time horizon. We also identified 13 articles (12.7%) where the time horizon was not defined clearly (or at all). Our concrete application in kidney transplantation shows that discrimination and calibration are not the same when comparing the two time horizon definitions. On one hand, for a 5-year sliding horizon window, the discrimination slightly increased as the landmark times increased, and we also observed that DynPG is reasonably well calibrated, particularly for the earliest landmark times. On the other hand, for an 11-year final time horizon, the discrimination was high for the earliest landmark times and increased over time, while the calibration plot revealed predictions were underestimated for the earliest landmark times and overestimated for later ones.

Conclusions.

Our systematic review identified a clear heterogeneity in the time horizon definition used, and an absence of a clear time horizon definition in a part of published articles. Our study advocates for improving the reporting when studying dynamic prediction scoring systems since the prognostic performances and interpretation differ according to the time horizon definition.

Time-to-event dynamic predictions

landmark times

horizon window

time horizon

discrimination

calibration

In the personalized medicine era [1], time-to-event dynamic predictions are becoming more widespread. They are defined as the probability to survive until a defined time horizon given being event-free at the time of making a prediction (landmark time) and given available predictive variables at such prediction times [2–5]. Dynamic predictions enable to take into account the valuable information consisting of the entire marker trajectory known at landmark time and have shown their importance in improving time-fixed predictions [6–8]. The dynamic predictions are updated predictions whenever additional longitudinal data becomes available during the patient follow-up.

The time horizon deserves precisions in reporting since it can be defined in several ways [3, 6]. One objective could be to predict the survival probability until a final time horizon. For instance, Teramukai et al. were interested in dynamically predicting cardiovascular endpoints of hypertensive patients until 3 years after inclusion using repeated on-treatment blood pressure measurements [9]. An alternative objective could be to predict the survival probability until the end of a sliding horizon window [6, 10, 11]. Ben-Hassen et al. proposed to dynamically predict dementia from longitudinal neurocognitive tests on a 5-year sliding horizon window [12]. The literature on dynamic predictions is growing. But, to our knowledge, a clear definition of these two objectives does not exist yet and the comparison between the two approaches have not been investigated.

To be useful, prognostic scores require good prognostic performances [13]. They can be studied through global performances using time-dependent Brier Score or R²-curve [10, 14], discrimination property using time-dependent AUC of ROC curves [14, 15] and calibration property using time-dependent calibration plots [16]. Since the estimated dynamic predictions would not be the same given the definition of the time horizon (final time horizon or sliding horizon window), this is of major importance when assessing the prognostic performances.

The objective was to illustrate that the prognostic performances and their interpretations differ given the two time horizon definitions. We presented the mathematical framework to obtain dynamic predictions according the two definitions (Section 2). We then conducted a systematic review of articles concerning dynamic predictions to objectively assess how the time horizon was reported in the literature (Section 3). We illustrated that prognostic performances of dynamic predictions of kidney graft failure differ between the time horizon definitions (Section 4). Finally, section 5 offers a discussion and conclusions.

2.1 Notations

Let’s assume a learning sample of $n$ independent and identically distributed patients with the observed data $\left\{{Y}_{i},{t}_{i}, {T}_{i},{\delta }_{i},{A}_{i};i=1,\dots , n\right\}$. Here, ${Y}_{i}=\left\{{Y}_{ij};j=1,\dots ,{n}_{i}\right\}$ corresponds to the set of ${n}_{i}$ longitudinal marker values measured at the corresponding times ${t}_{i}=\left\{{t}_{ij};j=1,\dots ,{n}_{i}\right\}$ in individual $i$. Let’s consider ${T}_{i}=\text{min}\left({T}_{i}^{*}, {C}_{i}\right)$ with ${T}_{i}^{*}$ the true time-to-event, ${C}_{i}$ the censoring time for subject $i$, and ${\delta }_{i}=1\left\{{T}_{i}^{*}\le {C}_{i}\right\}$ the event indicator function taking 1 if the event is not censored and 0 otherwise. We note ${A}_{i}=\left\{{A}_{ik};k=1,\dots ,K\right\}$ the $K$ baseline variables. From a learning sample of $n$ patients, we can estimate a prediction model of parameters $\theta$. Among those, landmarking and joint modeling of longitudinal and survival data are popular approaches [5, 17, 18].

2.2. Dynamic predictions definitions

The dynamic prediction ${\pi }_{l}$ for a patient $l$ is defined as the probability of being event-free until a time horizon $u$ given being event-free at the landmark time $s \left(s<u\right)$, given baseline variables ${A}_{l}$ and the longitudinal marker history available at time $s$ (i.e. ${\stackrel{\sim}{Y}}_{l}\left(s\right)=\left\{{Y}_{l}\left({s}_{1}\right),\dots ,{Y}_{l}\left({s}_{p}\right);0\le {s}_{1}<\dots <{s}_{p}<s\right\}$):

${\pi }_{l}\left(s,u;\theta \right)=\text{Pr}\left({T}_{l}^{*}>u|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\theta \right)$ $\left(3\right)$

We report two different uses of such generic definition of dynamic predictions [2–5, 7], that depend on the definition of time horizon $u$ and that would not have the same clinical objective.

Dynamic predictions given a final time horizon. The objective would be to dynamically predict the survival probability ${\pi }_{l}$ for a patient$l$ until a fixed final time horizon $U$ (Fig. 1). It corresponds to the probability to not suffer the event on a horizon window defined as the delay between the landmark time$s$ and the final time horizon $U$:

$${\pi }_{l}\left(s,U;\theta \right)=\text{Pr}\left({T}_{l}^{*}>U|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}(s);\theta \right)$$

Using the estimated parameters $\widehat{\theta }$ of the prediction model, the dynamic predictions considering a final time horizon can be estimated as the ratio between the survival probability until the final time horizon$U$ and the survival probability until the time of making prediction $s$:

$${\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>s|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}$$

Following this, the horizon window $\left\{U-s\right\}$ is progressively reduced when landmark time $s$ increases (Fig. 1). We note that for a fixed ${\stackrel{\sim}{Y}}_{l}\left(s\right)$, the smaller the difference between $s$ and $U$, the bigger ${\widehat{\pi }}_{l}\left(s,U\right)$ will be. Indeed, ${\pi }_{l}\left(s,U;\theta \right)$ is bounded with:

$$\underset{s\to 0}{\text{lim}}{\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(0);\widehat{\theta }\right)}{1} = Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(0);\widehat{\theta }\right)$$

$$\underset{s\to U}{\text{lim}}{\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(U);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(U);\widehat{\theta }\right)} = 1$$

Dynamic predictions given a sliding horizon window. Some authors have proposed an alternative definition of the dynamic predictions [2, 10, 11]. The target of inference of a patient $l$ is to predict the survival probability until the end of a horizon window of length $\varDelta t$:

$${\pi }_{l}\left(s,s+\varDelta t;\theta \right)=\text{Pr}\left({T}_{l}^{*}>s+\varDelta t|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}(s);\theta \right)$$

Following this definition, the horizon window $\varDelta t$ is thus sliding when updating the prediction, i.e. the landmark time increases (Fig. 2). Using the estimated parameters $\widehat{\theta }$ of the prediction model, the dynamic predictions given a sliding horizon window can be decomposed as the ratio between the survival probability until the end of the horizon window $s+\varDelta t$ and the survival probability until the time of making prediction $s$:

$${\widehat{\pi }}_{l}\left(s,s+\varDelta t;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>s+\varDelta t|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>s|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}$$

2.3. Accuracy measures for dynamic predictions

Following these two definitions, the horizon window is either reduced or sliding as the landmark time increases. Therefore, the at-risk population at landmark times would be identical given the two definitions, but the incidence rate on the prediction window would not. Consequently, the prognostic performances and the corresponding interpretation will differ.

The global prognostic performance of dynamic predictions can be assessed by measuring the prediction error [19] with the popular Brier Score metric, for instance. The time-dependent expected Brier Score is defined as follows [14, 17]:

$$BS\left(s,u\right)=E\left[\left(1\left({T}^{*}>u\right)-\pi \left(s,u\right)\right)²|{T}^{*}>s\right]$$

The Brier Score is a mean square error term. The closer the prediction is to the observation, the closer the Brier Score is to 0. When considering the final time horizon, it means that the survival probability until a final time horizon is expected to be close to the observed event indicator after the final time horizon $\left(BS\left(s,U\right)=E\left[\left(1\left({T}^{*}>U\right)-\pi \left(s,U\right)\right)²|{T}^{*}>s\right]\right)$. When considering the sliding horizon window, it means that the survival probability after an horizon window $\varDelta t$ is expected to be close to the observed event indicator after this horizon window $\left(BS\left(s,s+\varDelta t\right)=E\left[\left(1\left({T}^{*}>s+\varDelta t\right)-\pi \left(s,s+\varDelta t\right)\right)²|{T}^{*}>s\right]\right)$. In a dynamic context, one limit of the Brier Score is that it is sensitive to the marginal event probability that could change as the landmark time increases. van Houwelingen et al. proposed studying the relative error reduction [19], also named R²-curve [10]:

$${R}^{2}\left(s,u\right)=1-\frac{BS\left(s,u\right)}{B{S}_{0}\left(s,u\right)}$$

where $BS\left(s,u\right)$ is the Brier Score obtained using the model of interest and $B{S}_{0}\left(s,u\right)$ the Brier Score of a reference model (Kaplan-Meier estimator for instance) allowing to assess the global performance of dynamic predictions whatever the evolution of the marginal event probability along the landmark times. Since the Brier Score computation would be impacted by the consideration of the final time horizon or the sliding horizon window, the R²-curve would also not be the same between the two approaches.

The discrimination is the ability of a prognostic tool to order the risk between subjects. To assess discrimination, the well-known Area Under the Receiver Operating Characteristics Curve (AUC) [14, 15] can be explicitly defined in a dynamic context as:

$$AUC\left(s,u\right)=\text{Pr}\left({\pi }_{l}\left(s,u\right)>{\pi }_{{l}^{{\prime }}}\left(s,u\right)|{T}_{l}^{*}>u,{s<T}_{{l}^{{\prime }}}^{*}<u\right)$$

When considering the final time horizon, the AUC corresponds to the probability that a subject $l$ at-risk at landmark time$s$ and who would not suffer the event before the final time horizon $U$ would have a higher survival prediction than a subject ${l}^{{\prime }}$ at-risk at landmark time$s$ and who would suffer the event before the final time horizon $U$ $\left(AUC\left(s,U\right)=\text{Pr}\left({\pi }_{l}\left(s,U\right)>{\pi }_{{l}^{{\prime }}}\left(s,U\right)|{T}_{l}^{*}>U,{s<T}_{{l}^{{\prime }}}^{*}<U\right)\right)$. When considering the sliding horizon window, the AUC corresponds to the probability that a subject $l$ at-risk at landmark time$s$ and who would not suffer the event before the end of the horizon window $s+\varDelta t$ would have a higher survival prediction than a subject ${l}^{{\prime }}$ at-risk at landmark time$s$ and who would suffer the event between $s$ and $s+\varDelta t$ $\left(AUC\left(s,s+\varDelta t\right)=\text{Pr}\left({\pi }_{l}\left(s,s+\varDelta t\right)>{\pi }_{{l}^{{\prime }}}\left(s,s+\varDelta t\right)|{T}_{l}^{*}>s+\varDelta t,{s<T}_{{l}^{{\prime }}}^{*}<s+\varDelta t\right)\right)$.

The calibration property assesses the ability of a prognostic tool to provide a prediction close to the observed outcome. Usually assessed through calibration plots, the calibration is described by comparing the predicted values within subgroups (defined from quantiles of predictions) to the observed survival probabilities (computed using the Kaplan–Meier estimator for instance). When considering the final time horizon, a good calibration means that, for any given $x$ value, we expect that among all subjects who are predicted to survive with a $x$ % probability, $x$ out of 100 will not experience the event before the final time horizon. When considering the sliding horizon window, a good calibration means that, for any given $x$ value, we expect that among all subjects who are predicted to survive with a $x$ % probability, $x$ out of 100 will not experience the event before the end of the horizon window.

3.1. Search strategy

We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement - PRISMA (Web supplementary materials Table S1) [20]. We searched the Medline database on 25th August 2022, without date or language restriction. The search equation used is reported in Appendix 1 of Web supplementary materials.

3.2. Study selection

As this systematic review questioned the definition and the choice of time horizon, we included all articles focusing on the development or the validation of individual dynamic prediction scoring systems as well as methodological papers concerning dynamic predictions. Two reviewers (LC - ED) independently screened references by title and abstract. All eligible texts were assessed independently by the two reviewers (LC - ED) and discussions resolved any discrepancies. Our non-inclusion criteria were reviews or meta-analyses, full texts not found or conference abstracts and editorials.

3.3. Data extraction

Each study was randomly allocated to two of the five reviewers (LC – VB – PR – SD – ED). Following the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) [21], we predefined a standardized form for data extraction. Whether it was a methodological article or an article concerning the development or the validation of dynamic predictions, for each eligible article, we collected the author’s name, year and journal, the data source, the study design, the inclusion and exclusion criteria, the sample size, the population characteristics, the predicted outcomes, the prediction times (i.e. the time when one calculates the prediction), the time horizon (i.e. the end of the prediction time window), the predictive tools and the development details (such as the methodology used, the variables of the scoring systems). For each article, the time horizon was qualified as either a final time horizon or a sliding horizon window from an explicit equation, explicit texts and/or a clear graphic representing individual dynamic prediction, or qualified as unclear information in case of lack of such explicit information or contradictory information. The TRIPOD statement recommends to develop a prognostic score from a learning sample and to validate the prognostic performances from an independent internal and/or external validation sample to avoid reporting the prognostic performances on the learning sample only, which would lead to overestimating the performances [22, 23]. We thus extracted information on internal and external validations as well. Whenever necessary, any discrepancies in data extraction were resolved by discussion with another reviewer to reach a consensus.

3.4. Results

The search identified 172 unique articles. Screening of titles and abstracts identified 135 papers eligible for full-text review. As detailed on the flow diagram (Fig. 3), 33 articles were excluded. Finally, we included 102 articles (Appendix 2 of Web supplementary materials).

Dynamic predictions appear to be a relatively recent research topic in biostatistics with 100 (98%) articles published after 2010. Fifty-one (50%) of the retained articles are methodological studies that proposed new modeling approach. This systematic review confirmed the dominance of joint modeling for longitudinal and survival data (n = 42, 41%) and landmarking (n = 44, 43%) to develop dynamic predictions. To a lesser extent, we identified the emergence of machine learning approaches including deep learning (n = 14, 14%), with random survival forest [24–26] or neural networks [27, 28] for instance. All articles referring to machine learning approaches were published after 2018. This will probably increase in the future with the rapid development of such modeling approaches. Among the applied articles (n = 51, 50%), we also observed that 12 studies (24%) only presented a development modeling of dynamic prediction without validation of their predictive performance on an independent data set (Table 1). This does not respect the TRIPOD recommendations about the right process to develop and validate prognostic performances. Nevertheless, 28 articles included internal validation, 5 an external validation and 5 both types of validation, leading to 76% of the applied articles in agreement with the TRIPOD.

Table 1

Description of the articles included in the systematic review
	All included articles N = 102	Methodological articles N = 51	Applied articles N = 51
Time prediction horizon definition
Final time horizon	18 (18%)	5 (10%)	13 (25%)
Sliding horizon window	71 (69%)	44 (86%)	27 (53%)
Unclear	13 (13%)	2 (4%)	11 (22%)
Published after 2010	100 (98%)	51 (100%)	49 (96%)
Main statistical analyses for model development
Joint model	42 (41%)	26 (51%)	16 (32%)
Landmarking	44 (43%)	18 (35%)	26 (52%)
Machine Learning	14 (14%)	6 (12%)	8 (16%)
Others	9 (9%)	5 (10%)	4 (8%)
Global performance	46 (45%)	32 (74%)	14 (28%)
Brier Score	28 (61%^a)	21 (66%^a)	7 (%^a)
Prediction error	20 (43%^a)	12 (38%^a)	8 (57%^a)
Others	1 (2%^a)	1 (3%^a)	0 (0%^a)
Discrimination	84 (82%)	34 (79%)	50 (100%)
AUC	65 (77%^a)	31 (91%^a)	34 (68%^a)
C-index	18 (21%^a)	3 (9%^a)	15 (30%^a)
Others	6 (7%^a)	0 (0%^a)	0 (0%^a)
Calibration	24 (24%)	6 (14%)	18 (36%)
Calibration plot	19 (79%^a)	5 (83%^a)	14 (78%^a)
Calibration slope	2 (8%^a)	0 (0%^a)	2 (11%^a)
Heuristic shrinkage factor	5 (21%^a)	0 (0%^a)	5 (28%^a)
Others	2 (8%^a)	1 (17%^a)	1 (6%^a)
Type of study
Development alone	-	-	12 (24%)
Development & Internal validation	-	-	28 (55%)
Development & External validation	-	-	5 (10%)
Development, Internal & External validation	-	-	5 (10%)
External validation alone	-	-	1 (1%)
^a Percentage among the count of articles reporting the prognostic ability

Among the included articles, we identified 18 (17.7%) articles defining dynamic predictions using the final time horizon, 71 (69.6%) articles using sliding horizon windows and 13 (12.7%) articles that were unclear about the time horizon definition (Table 1). Methodological articles mainly used a sliding horizon window approach to consider dynamic predictions (n = 44, 86%), and to a lesser extent final time horizon (n = 5, 10%) while only 2 (4%) articles did not clearly define the considered approach. This repartition was more heterogeneous when studying the applied articles: 27 (53%) used sliding horizon windows, 13 (25%) used the final time horizon and 11 (22%) appeared unclear as to which approach was used.

Beyond the description of the time horizon considered, our systematic review allows to appraise the reporting quality on dynamic predictions. As clearly recommended in the literature [13], assessing prognostic performances is of major importance for prediction tools to propose useful scores. Discrimination capacity is well reported (n = 84, 82%) with AUC as the principal performance indicator. Global performances (n = 46, 45%) with a Brier Score are less often provided, as are calibration (n = 24, 24%) with a calibration plot. Details of such prognostic performances are also not homogeneous between methodological and applied articles (Table 1).

4.1. Context

In kidney transplantation, patients are particularly interested by their kidney graft survival, before any risk of adverse outcomes or infections [29]. In this context, we developed and internally and externally validated ‘Dynamic predictions of Patient and kidney Graft survival’ (DynPG) for kidney recipients alive with a functioning graft at 1-year post-transplantation [16, 30]. The main outcome was the delay from 1-year post-transplantation to patient and kidney graft failure defined as the first event between return-to-dialysis, pre-emptive retransplantation and death with a functioning graft. Such dynamic predictions can be obtained from six baseline variables: recipient age, graft rank, cardiovascular histories, pretransplantation anti-HLA class I immunization, serum creatinine at 3-months post-transplantation, occurrence of acute rejection in the first year post-transplantation, and also the complete longitudinal serum creatinine trajectory available at the time of prediction. Our aim was to provide to patients and physicians an information on the possible future on a sliding horizon window of 5 years. In line with this clinical objective, the DynPG was defined as the probability of being graft failure-free over the next 5 years after the landmark time, for each prediction time from 1 to 6 years post-transplantation. We retained 6 years post-transplantation as the maximal landmark time since there were 178 patients still at risk of patient and kidney graft failure at 11 years post-transplantation in the validation sample.

We aimed to illustrate that the prognostic performances would not be the same under the assumptions of a sliding horizon window or a final time horizon. While a sliding horizon window may be relevant to monitor the patient risk along his/her follow-up with always the same horizon window of 5 years, the clinical objective is different under the prism of a final time horizon. Considering the final time horizon of 11 years post-transplantation, the dynamic predictions aim to predict with good confidence until this final time horizon. Benefiting from the previously estimated joint model and the same internal validation sample, we compared the prognostic performances of the DynPG under the assumption of a 5-year sliding horizon window and under the assumption of a 11 years final time horizon.

4.2. Study population

Data were extracted from the French multicentric observational and prospective DIVAT cohort (Données Informatisées et VAlidées en Transplantation; www.divat.fr, CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). All participants gave informed consent. The inclusion criteria were adult recipients who received a first or second renal graft transplanted after January 2000 from a living or heart-beating deceased donor, alive with a functioning graft at 1 year post-transplantation and maintained under tacrolimus and mycophenolate. The extracted DIVAT cohort data consisted of a learning set of 2,749 patients, initially used to estimate the shared-random joint model, and an internal validation set of 2,589 patients. Due to missing data for retained predictors, 66 patients were excluded from the analysis of the validation set (n = 2,523). More details on this validation sample are reported in Fournier et al. [16].

4.3. Results

Under the 5-year sliding horizon window setting, the results were as previously published [16]. The global prognostic performance of the DynPG appeared relatively stable along the landmark times with R² values ranging from 14% (95% CI 7–21%) to 15% (95% CI -2–33%) at 1 and 6 years post-transplantation, respectively (Fig. 4A). The discrimination slightly increased along the prediction times with the AUC values ranging from 0.72 (95% CI 0.67–0.78) to 0.76 (95% CI 0.68–0.85) at 1 and 6 years post-transplantation (Fig. 4B). Since the discrimination performances increased and the global prognostic performances remained stable, the calibration properties decreased for the late landmark times (Figure S1).

Under the 11 years final time horizon assumption, the global prognostic performances of the DynPG were at a higher level for the earliest landmark times, but decreased along the landmark times. We estimated R² values of 22% (95% CI 12–32%), 15% (95% CI -2–31%) and − 2% (95% CI -41–37%) at 1, 6 and 10 years post-transplantation, respectively (Fig. 5A). The discrimination was high for the earliest landmark times and increased over times with AUC values of 0.76 (95% CI 0.70–0.82), 0.76 (95% CI 0.67–0.84) and 0.88 (95% CI 0.79–0.98) at 1, 6 and 10 years post-transplantation, respectively (Fig. 5B). On the other hand, we observed a very poor calibration plot (Figure S2): the predictions appeared underestimated for the earliest landmark times and overestimated for the late ones.

The number of at-risk patients at landmark times are the same for the two time horizon definitions. However, the number of observed events and the number of censored subjects on each window differs between the two approaches due different horizon windows, resulting in a number of at-risk patients at the end of the horizon window that are not comparable (Figures S3, S4). The incidence rate on the prediction window, that is either reduced or sliding given the landmark time increases, are thus not the same between the two approaches. Consequently, the dynamic predictions cannot be compared and they tell a different story. For instance, under the sliding horizon window assumption, at 1 year post-transplantation, we estimated a 72% probability that the predicted survival of a patient who actually experienced a graft failure within the 5 years was lower than that of a patient who did not (Fig. 4 – part B). Under the final time horizon assumption, at 1 year post-transplantation, we estimated a 76% probability that the predicted survival of a patient who actually experienced a graft failure before 11 years post-transplantation was lower than that of a patient who did not (Fig. 5 – part B). In terms of calibration properties, we may reasonably accept that DynPG was sufficiently well calibrated for the earliest landmark times under the 5-year sliding horizon window assumption (Figure S1). In contrast, under the 11-year final time horizon assumption, the calibration is quite poor (Figure S2).

In this study, we distinguished two types of time prediction horizon - final time horizon or sliding horizon window - for dynamic predictions. We conducted a systematic review that stated the heterogeneity of the used time prediction horizons in the literature about dynamic predictions. While the two definitions are similar, a specific definition is of major importance since the prognostic performances obtained are different given the nature of the prediction window. This is also well illustrated by our concrete application in kidney transplantation.

The concept of P4-medicine (predictive, preventive, personalized and participatory) is now largely developed in the literature, but still difficult to apply in clinical practice [1]. Dynamic predictive tools can be beneficial for a such health policy and promote shared medical decision making, provided that the prognostic performances are sufficiently good. While dynamic predictions are updated predictions whenever additional information is available during the patient follow-up [2, 6], their associated prognostic performances depend on the at-risk population at the time of making a prediction and can evolve as the landmark time increases. In this work, we insist on the fact that prognostic performances are also related to the prediction window. We showed that the incidence rates of the event would not be identical between the two time horizon definitions due to the nature of the window that can be either reduced or sliding, resulting in prognostic performances that are not comparable. The corresponding interpretations of prognostic performances should therefore be formulated with caution since they do not tell the same story in the two contexts.

In our opinion, the dynamic predictions obtained with the sliding horizon window framework can be used to follow the patient health evolution accross the landmark times and thus present interesting properties for individualized and personalized medicine [16, 30]. With a fixed length of prediction window, it allows to appreciate the prognostic performances all along the follow-up. In our concrete application in kidney transplantation, the ‘Dynamic predictions of patient and kidney graft survival’ has a good discrimination property useful for stratified medicine. The good calibration properties from 1-year until 6-year landmark times are important point for personalized care. We argued that this dynamic predictive tool can be useful for instance to inform kidney transplant recipients of their prognosis over the next 5 years [16]. This may result in increased patient adherence to their treatment, to increase patient empowerment, as patients recognize having an active role in their chronic disease evolution [31, 32]. This could also help patients to better manage their feelings of uncertainty about the survival of their transplant in the not-too-distant future.

When considering the final time horizon (i.e. reducing window), the comparison of prognostic performances across the landmark times is difficult since the lengths of the prediction window are not the same. Despite the bounded character of dynamic predictions for a final time horizon, it is important to note that such predictions are not always monotonic along the landmark times because of the actualization of the longitudinal marker. The amelioration or deterioration of the prognosis between two landmark times is not necessarily due to a difference in health state characterized by the new marker measurement. This is noised by the mathematical artefact brought by the reduction of the prediction window. Since it is difficult to know to what extent the prognosis evolution is due to the reduction of the window rather than to the marker evolution, a comparison cannot be done between predictions realized at two different landmark times. For this reason, the final time horizon approach does not seem suited to follow the patient health evolution through survival probabilities and cannot be envisaged to support patient personalized care. In our kidney transplantation application, the poor calibration property across the landmark times for a final time horizon of 11 years post-transplantation do not allow to consider that individual survival probability are correctly predicted. The more the prediction time approaches 11 years, the more we predicted patient and kidney survival probabilities close to 1, while the observed event repartition appear not. This inevitably results in poor calibration. Such an approach cannot be used to personalize the kidney recipient taking care based on the predicted patient and kidney graft survival probabilities. Nevertheless, in a context in which the objective is not the personalization of patient care, but rather the stratification of the studied population, considering a final time horizon may be of interest. This should be clearly stated in the clinical objective. In kidney transplantation, we may envisage to guide patients through the care organization given their stratified survival probabilities. In our application, we observed an increase of the discrimination performance of the DynPG up to 11 years post-transplantation. We may consider to better organize the healthcare visit by reducing the patient visit schedule or consider telemedicine consultations for recipients at higher patient and kidney graft survival probabilities and by reinforcing the patient follow-up for patients with lower survival predictions [33, 34]. All the same, we recognize that it may be difficult to determine a final time horizon of clinical interest, particularly in the context of chronic disease. In our application, the horizon of 11 years is debatable from a patient-centered perspective should difficulty be bounded by a terminal horizon. This may be more appropriate in a public health perspective where efficient allocation of resources could provide economic benefits. Other clinical contexts may be of interest when a final time horizon is already known as for medical devices, with for instance the prediction of aortic valve duration or the prediction of hip replacement duration up to a maximum announced by the supplier [35].

Considering these methodological differences between the two time horizon definitions, the studies’ clinical objective should be clearly anticipated to correctly assess the prognostic performances. Indirectly, the heterogeneity that we observed in the reporting of the time horizon window in our systematic review reveals the lack of clarity regarding the clinical objectives justifying dynamic predictions development. Our systematic review also informed about the methodological quality of dynamic predictions. A non negligible proportion of the literature did not sufficiently respect the TRIPOD recommendation [22] with a lack of validation studies for instance. We also noted that discrimination properties were mainly reported and referred to the stratification of the studied population, but calibration metrics that are essential for personalized care were not reported in numerous studies.

In the literature, numerous developments concern improvements of the joint modeling of the longitudinal and the survival processes and thus participate to the improvement of dynamic predictions performances. For instance, considering competing risks [36, 37], to refine the longitudinal evolution with flexible modeling [38, 39], or incorporating multiple longitudinal markers [36, 40], would help to better predict the clinical event of interest. The mathematical notations introduced and the following concrete application concerns the case of one longitudinal marker (e.g. serum creatinine) and one clinical event (e.g. patient and kidney graft survival). Nevertheless, notations can be adapted to more complex modeling situations. Our conclusion about the attention to bring to time prediction windows would still be valid to broader modeling frameworks.

In conclusion, our study identified that the choice of the time prediction window is crucial and should depend on the clinical objectives. Our work advocates for clarifying the clinical objective of the dynamic predictions and improving the reporting when studying dynamic prediction scoring systems since prognostic performances and their interpretation will differ given the time horizon definition.

95% CI: 95% Confidence Interval; AUC: Area Under the Curve; CHARMS: CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies; DIVAT: Données Informatisées et VAlidées en Transplantation; DynPG : Dynamic predictions of Patient and kidney Graft survival; HLA: Human Leukocyte Antigen, PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; ROC: Receiver Operating Characteristic ; TRIPOD: Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis statements.

Ethics approval and consent to participate

Data confidentiality was ensured following the recommendations of the French commission for data protection (Commission Nationale Informatique et Liberté, CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). The present study was approved by the DIVAT scientific committee (Données Informatisées et VAlidées en Transplantation; www.divat.fr).

Consent for publication

Not applicable

Availability of data and materials

The data underlying this article could be shared on reasonable request to the corresponding author.

Competing interests

Lucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was partly supported by SÊMEIA and also by the French National Research Agency [grant number ANR PRC 2022 JM-QALYs].

Authors' contributions

Lucas Chabeau : methodology, formal analysis, systematic review, writing - original draft, review and editing.

Vincent Bonnemains : methodology, systematic review, writing - review and editing.

Pierre Rinder : methodology, systematic review, writing - review and editing.

Magali Giral : scientific coordinator of the DIVAT cohort, clinical mentorship, data collection, writing - review and editing.

Solène Desmée: methodology, systematic review, writing - review and editing.

Etienne Dantan: conceptualization, methodology, formal analysis, systematic review, writing - original draft, review and editing.

Acknowledgments

We thank members of the DIVAT consortium* and the clinical research assistant team (S. Le Floch, A. Petit, J. Posson, C. Scellier, V. Eschbach, K. Zurbonsen, C. Dagot, F. M’Raiagh, V. Godel, X. Longy, P. Przednowed). We are also grateful to Roche Pharma, Novartis, Sanofi and Astellas laboratories for supporting the DIVAT cohort (www.divat.fr) as the CENTAURE foundation (www.fondation-centaure.org).

* Données Informatisées et VAlidées en Transplantation, DIVAT Cohort Collaborators (Medical Doctors, Surgeons, HLA Biologists)

Nantes : Gilles Blancho, Julien Branchereau, Diego Cantarovich, Agnès Chapelet, Jacques Dantal, Clément Deltombe, Lucile Figueres, Claire Garandeau, Magali Giral, Caroline Gourraud-Vercel, Maryvonne Hourmant, Georges Karam, Clarisse Kerleau, Aurélie Meurette, Simon Ville, Christine Kandell, Anne Moreau, Karine Renaudin, Anne Cesbron, Florent Delbos, Alexandre Walencik, Anne Devis ; Paris-Necker : Lucile Amrouche, Dany Anglicheau, Olivier Aubert, Lynda Bererhi, Christophe Legendre, Alexandre Loupy, Frank Martinez, Rébecca Sberro-Soussan, Anne Scemla, Claire Tinel, Julien Zuber ; Nancy : Pascal Eschwege, Luc Frimat, Sophie Girerd, Jacques Hubert, Marc Ladriere, Emmanuelle Laurain, Louis Leblanc, Pierre Lecoanet, Jean-Louis Lemelle ; Lyon E. Hériot : Lionel Badet, Maria Brunet, Fanny Buron, Rémi Cahen, Sameh Daoud, Coralie Fournie, Arnaud Grégoire, Alice Koenig, Charlène Lévi, Emmanuel Morelon, Claire Pouteil-Noble, Thomas Rimmelé, Olivier Thaunat ; Montpellier : Sylvie Delmas, Valérie Garrigue, Moglie Le Quintrec, Vincent Pernin, Jean-Emmanuel Serre.

Flores M, Glusman G, Brogaard K, Price ND, Hood L. P4 medicine: how systems medicine will transform the healthcare sector and society. Pers Med. 2013;10:565–76.
Proust-Lima C, Blanche P. Dynamic Predictions. Wiley StatsRef: Statistics Reference Online. John Wiley & Sons, Ltd; 2016. pp. 1–6.
Andrinopoulou E-R, Harhay MO, Ratcliffe SJ, Rizopoulos D. Reflection on modern methods: Dynamic prediction using joint models of longitudinal and time-to-event data. Int J Epidemiol. 2021;50:1731–43.
Asgari S, Khalili D, Zayeri F, Azizi F, Hadaegh F. Dynamic prediction models improved the risk classification of type 2 diabetes compared with classical static models. J Clin Epidemiol. 2021;140:33–43.
Rizopoulos D. Joint Models for Longitudinal and Time-to-Event Data: With Applications in R. CRC; 2012.
Rizopoulos D. Dynamic predictions and prospective accuracy in joint models for longitudinal and time-to-event data. Biometrics. 2011;67:819–29.
Barrett J, Su L. Dynamic predictions using flexible joint models of longitudinal and time-to-event data. Stat Med. 2017;36:1447–60.
Li K, Luo S. Dynamic predictions in Bayesian functional joint models for longitudinal and time-to-event data: An application to Alzheimer’s disease. Stat Methods Med Res. 2019;28:327–42.
Teramukai S, Okuda Y, Miyazaki S, Kawamori R, Shirayama M, Teramoto T. Dynamic prediction model and risk assessment chart for cardiovascular disease based on on-treatment blood pressure and baseline risk factors. Hypertens Res Off J Jpn Soc Hypertens. 2016;39:113–8.
Fournier M-C, Dantan E, Blanche P. An R2 -curve for evaluating the accuracy of dynamic predictions. Stat Med. 2018;37:1125–33.
Asar Ö, Fournier M-C, Dantan E. Dynamic predictions of kidney graft survival in the presence of longitudinal outliers. Stat Methods Med Res. 2021;30:185–203.
Ben-Hassen C, Helmer C, Berr C, Jacqmin-Gadda H. Five-Year Dynamic Prediction of Dementia Using Repeated Measures of Cognitive Tests and a Dependency Scale. Am J Epidemiol. 2022;191:453–64.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiol Camb Mass. 2010;21:128–38.
Blanche P, Proust-Lima C, Loubère L, Berr C, Dartigues J-F, Jacqmin-Gadda H. Quantifying and comparing dynamic predictive accuracy of joint models for longitudinal marker and time-to-event in presence of censoring and competing risks. Biometrics. 2015;71:102–13.
Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17:53.
Fournier M-C, Foucher Y, Blanche P, Legendre C, Girerd S, Ladrière M, et al. Dynamic predictions of long-term kidney graft failure: an information tool promoting patient-centred care. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2019;34:1961–9.
Ferrer L, Putter H, Proust-Lima C. Individual dynamic predictions using landmarking and joint modelling: Validation of estimators and robustness assessment. Stat Methods Med Res. 2019;28:3649–66.
Goldstein BA, Pomann GM, Winkelmayer WC, Pencina MJ. A comparison of risk prediction methods using repeated observations: An application to Electronic Health Records for Hemodialysis. Stat Med. 2017;36:2750–63.
van Houwelingen HC, Putter H. Dynamic Prediction in Clinical Survival Analysis. CRC; 2012.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9. W64.
Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11:e1001744.
Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73.
Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–7.
Devaux A, Genuer R, Peres K, Proust-Lima C. Individual dynamic prediction of clinical endpoint from large dimensional longitudinal biomarker history: a landmark approach. BMC Med Res Methodol. 2022;22:1–14.
Yang T, Yang Y, Jia Y, Li X. Dynamic prediction of hospital admission with medical claim data. BMC Med Inf Decis Mak. 2019;19(Suppl 1):18.
Lin J, Li K, Luo S. Functional survival forests for multivariate longitudinal outcomes: Dynamic prediction of Alzheimer’s disease progression. Stat Methods Med Res. 2021;30:99–111.
Fahimi F, Guo Y, Tong SC, Ng A, Bing SOY, Choo B et al. A Vital Signs Telemonitoring Programme Improves the Dynamic Prediction of Readmission Risk in Patients with Heart Failure. AMIA Annu Symp Proc AMIA Symp. 2020;2020:432–41.
Zeng Z, Tang X, Liu Y, He Z, Gong X. Interpretable recurrent neural network models for dynamic prediction of the extubation failure risk in patients with invasive mechanical ventilation in the intensive care unit. BioData Min. 2022;15:21.
Howell M, Wong G, Rose J, Tong A, Craig JC, Howard K. Patient Preferences for Outcomes After Kidney Transplantation: A Best-Worst Scaling Survey. Transplantation. 2017;101:2765–73.
Lenain R, Dantan E, Giral M, Foucher Y, Asar Ö, Naesens M, et al. External Validation of the DynPG for Kidney Transplant Recipients. Transplantation. 2021;105:396–403.
Gordon EJ, Butt Z, Jensen SE, Lok-Ming Lehr A, Franklin J, Becker Y, et al. Opportunities for shared decision making in kidney transplantation. Am J Transpl Off J Am Soc Transpl Am Soc Transpl Surg. 2013;13:1149–58.
Vandecasteele SJ, Kurella Tamura M. A patient-centered vision of care for ESRD: dialysis as a bridging treatment or as a final destination? J Am Soc Nephrol JASN. 2014;25:1647–51.
Jörres A, John S, Lewington A, ter Wee PM, Vanholder R, Van Biesen W, et al. A European Renal Best Practice (ERBP) position statement on the Kidney Disease Improving Global Outcomes (KDIGO) Clinical Practice Guidelines on Acute Kidney Injury: part 2: renal replacement therapy. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2013;28:2940–5.
Foucher Y, Meurette A, Daguin P, Bonnaud-Antignac A, Hardouin J-B, Chailan S, et al. A personalized follow-up of kidney transplant recipients using video conferencing based on a 1-year scoring system predictive of long term graft failure (TELEGRAFT study): protocol for a randomized controlled trial. BMC Nephrol. 2015;16:6.
Evans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A. How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet. 2019;393:647–54.
Andrinopoulou E-R, Rizopoulos D, Takkenberg JJ, Lesaffre E. Combined dynamic predictions using joint models of two longitudinal outcomes and competing risk data. Stat Methods Med Res. 2017;26:1787–801.
Suresh K, Taylor JMG, Spratt DE, Daignault S, Tsodikov A. Comparison of joint modeling and landmarking for dynamic prediction under an illness-death model. Biom J Biom Z. 2017;59:1277–300.
Yang M, Luo S, DeSantis S. Bayesian quantile regression joint models: Inference and dynamic predictions. Stat Methods Med Res. 2019;28:2524–37.
Desmée S, Mentré F, Veyrat-Follet C, Sébastien B, Guedj J. Nonlinear joint models for individual dynamic prediction of risk of death using Hamiltonian Monte Carlo: application to metastatic prostate cancer. BMC Med Res Methodol. 2017;17:105.
Li N, Liu Y, Li S, Elashoff RM, Li G. A flexible joint model for multiple longitudinal biomarkers and a time-to-event outcome: With applications to dynamic prediction using highly correlated biomarkers. Biom J Biom Z. 2021;63:1575–86.

Competing interest reported. Lucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplementarymaterial.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation

Status:

Version 1

Abstract

Background.

Methods.

Results.

Conclusions.

Figures

1. Background

2. Methods

2.1 Notations

2.2. Dynamic predictions definitions

2.3. Accuracy measures for dynamic predictions

3. Systematic review of dynamic predictions

3.1. Search strategy

3.2. Study selection

3.3. Data extraction

3.4. Results

4. Application

4.1. Context

4.2. Study population

4.3. Results

5. Discussion

6. Conclusions

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1