External validation of a prognostic model for 1-year survival after fragility hip fracture: A retrospective cohort study

doi:10.21203/rs.3.rs-2003197/v1

Download PDF

Article

External validation of a prognostic model for 1-year survival after fragility hip fracture: A retrospective cohort study

https://doi.org/10.21203/rs.3.rs-2003197/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Background

To scientifically manage fragility hip fracture patients, we have developed a survival prediction model for predicting 1-year survival after fragility hip fracture (SFHF). This study aims to validate the model.

Methods

This was a retrospective cohort study conducted in the orthopaedics department at a regional medical centre in central China. Fragility hip fracture patients who were consecutively admitted to the orthopaedics department from January 2016 to October 31, 2021. We evaluated the performance of an SFHF model (including the full model and the simplified model) from the following three aspects: (1) Discrimination. We presented the concordance (c) index of the model, including Harrell's c-index and Uno's c-value. Overall performance was measured using Nagelkerke R² values. (2) Calibration. The calibration plot method was used to evaluate the calibration of the model. (3) Decision curve analysis (DCA) was used to judge whether the model had clinical value in the validation population.

Results

A total of 877 (≥ 50 years) (out of 1132) eligible fragility hip fracture patients were included in this study. Among them, 47 patients were lost to follow-up. Among the patients who were successfully followed up, 87 died within 1 year of the fracture. After applying simple imputation to deal with missing values, the final effective sample size was 93 cases. The 1-year mortality rate after fracture was 10.6%. The Harrell’s c-index values of the full and simple SFHF models were 0.764 (standard error, 0.024) and 0.763 (0.024), respectively. Uno’s c-values were 0.765 (0.024) and 0.763 (0.024), respectively. The Nagelkerke R² values were 0.144 and 0.144, respectively. The calibration plot showed that the model had good calibration between predicted and actual values. DCA showed that the model was clinically useful within a threshold range of risk of death of 0.03–0.38.

Conclusion

Our study preliminarily confirmed that the SFHF model has good accuracy and generalizability in predicting the one-year survival rate of fragility hip fracture patients, and that it has good clinical value. This predictive model may be considered for local clinical practice.

External validation

Prognostic model

Fragility hip fracture

Survival

Background and Objectives

For the purpose of scientifically stratifying the management of hip fractures in elderly individuals, and at the same time identifying high-risk patients to guide doctors and patients to jointly make decisions about whether to operate, we previously developed a prognostic model to predict 1-year survival after fragility hip fracture (SFHF)(1). This prognostic Cox proportional hazards model was developed from a retrospective cohort study in a secondary care hospital in China. Of the 735 eligible elderly hip fracture patients in the study, 68 died within 1 year of fracture. Its simplified version includes 8 preoperative clinical indicators (age, albumin, sex, serum creatinine, malignancy, hypertension, ability to live independently, and cardiovascular and cerebrovascular diseases) that can be quickly ascertained within 48 hours after admission; the addition of whether to choose surgical treatment gives a total of 9 predictor variables. Using the nomogram derived from this model, a weighted score for each variable can be quickly obtained for a patient after admission. Based on the total score, the 1-year survival rate of the patient with surgery (surgery = 1) or conservative treatment (surgery = 0) can be obtained simultaneously before determining the treatment plan. The model showed good discriminative ability (Harrell’s c-statistic = 0.814 (95% confidence interval (CI )0.762–0.865)) and no significant overfitting (c = 0.795, internal validation by 1000 bootstrap repetitions). External validation is essential if the model is to be ultimately used for clinical decision-making (2–5). For this purpose, we designed and conducted another retrospective cohort study to examine the accuracy and generalizability of this model in a relatively related new population(5, 6).

In recent years, predictive models that also target fragility hip fracture patients and focus on 1-year mortality (or 1-year survival) after fracture are not uncommon. External validation of multiple similar models on the same dataset can provide potential model users with more information, such as which model looks better. However, here, we did not perform such a validation study. This is for the following reasons: 1. The purposes and contexts of model development are varied. As you can imagine, our own developed models are best suited to our own local environment. Therefore, we are most concerned with the performance of our model on external validation. This is probably why more model validation research is carried out by developers themselves. 2. This study was specifically designed to validate our own model. Therefore, in terms of predictor definitions, outcome definitions, and data measurement and collection, the heterogeneity between these validation data and our own development data is naturally minimal compared to the development data for other models. In the presence of such systematic biases, objective model comparison study results cannot be obtained. This is a fact(5, 7). 3. A more appropriate study design for a model comparison study is a prospective cohort study, since certain predictors can be collected specifically for the target model. However, this is a retrospective cohort study and cannot collect variables such as “poor handgrip strength”(8), “not being able to drive”(9), and “difficulty preparing meals”(9). As a result, validation of models containing these predictors cannot be achieved. 4. Because of the different contexts in which similar models were developed, some of the predictors in the models are not meaningful in certain settings. For example, “long-term care residence”(10) and “living in an institution”(11) are meaningless in most areas of China because it is not common for elderly individuals to live in nursing facilities. In China, most elderly people live alone or with a spouse or children. It does not make sense to validate such a model. Therefore, this study only validates our self-developed SFHF model.

This study followed the 2015 TRIPOD recommendations(12).

Data Source

We designed and conducted a single-centre, retrospective longitudinal cohort study. We calculated survival for each patient in this cohort using the SFHF model and compared it with actual observations to determine whether the model is a good tool for predicting 1-year survival after fracture for these individuals. When necessary, the model is updated to better fit this population. This study included all patients with hip fracture who were consecutively admitted to the Orthopaedics Department of Shanxi Bethune Hospital in China from January 1, 2016, to October 31, 2021. Follow-up of these patients was then completed via telephone interviews from March 14 to 29, 2022. The preoperative individual characteristics of each patient were obtained from the patient's medical records. This study was approved by the Medical Ethics Committee of Shanxi Bethune Hospital. Informed consent was waived by the Medical Ethics Committee of Shanxi Bethune Hospital. All methods in this study followed the Declaration of Helsinki.

Participants

Shanxi Bethune Hospital is located in Taiyuan, the capital city of Shanxi Province in Central China. It is a comprehensive teaching hospital and a tertiary regional referral hospital. As in the developmental study, we identified our eligible participants by the following: 1. Patients ≥ 50 years old; 2. Patients with low-energy fractures (referring to fractures caused by a patient falling from a standing height or lower), excluding patients with pathological, high-energy fractures, periprosthetic fractures after previous hip replacement surgery, and reoperation due to failure of internal fixation for hip fracture, regardless of whether the patient had a primary or secondary fracture; and 3. Patients with hip fractures including femoral neck, intertrochanteric and subtrochanteric fractures.

Unlike in the developmental study, eligible patients could have other low-energy fractures at the same time, such as wrist fractures, femoral neck fractures, or vertebral compression fractures, which result from falls that lead to hip fractures. For the same patient who was hospitalized twice for hip fractures on different sides during the study period, the most recent hospitalization data were selected.

All patients were considered for surgical treatment after admission, and routine preoperative preparations were completed as soon as possible. If the patient suffered from obvious medical diseases that affected the surgical treatment, the surgery was postponed, and the relevant departments were requested to assist with diagnosis and treatment. Surgical treatment was given when the patient's overall condition was stable. In the developmental study, participants underwent osteodistraction immediately after hospitalization, unless it was determined that surgery would be performed soon. In this study population, osteodistraction was not performed unless it was clear that the patient could not undergo surgery in the short term or that surgery was not considered.

As in the developmental study, different surgical approaches were implemented depending on the fracture site. For femoral neck fractures, cannulated screw fixation and total hip or hemi-hip replacement were selected according to different age ranges. Intramedullary nails were used for intertrochanteric and subtrochanteric fractures.

All eligible patients were included in this study. In the case of individuals with missing data, we used advanced statistical methods to impute to avoid serious bias caused by simple deletion.

Outcome

As in the developmental study, our event of interest was all-cause mortality within 1 year (365 days) after fracture. We first collected demographic characteristics and preoperative clinical indicators for each patient, followed by telephone interviews.

Similar to developmental research, we also developed an interview strategy to increase the interview rate and reduce loss to follow-up(1).

Predictors

As in the developmental study, we collected 24 individual patient indicators. Except for whether surgery was performed, the remaining 23 variables were all preoperative characteristics. They included demographic characteristics such as age, sex and medical insurance; fracture-related characteristics such as fracture site, type of fracture, days from fracture to hospitalization, days from hospitalization to surgery, and length of stay (LOS); medical history information such as ability to live independently (ALI), lung disease (LD), cardiovascular and cerebrovascular disease (CCD), kidney disease (KD), malignancy (MAL), hypertension (HYP), diabetes, and mean arterial pressure (MAP); laboratory-related factors such as partial pressure of oxygen (PaO2), haemoglobin (Hb), serum creatinine (SC), fasting blood sugar (BS), albumin (ALB), and total protein (TP); and treatment-related factors such as osteodistraction and surgery (SUR).

The definition and measurement time of each indicator can be found in the developmental study (1). Here, we describe only the variables that differ from those in developmental studies. In this study, we redefined CCD. In the developmental study, CCD positivity was defined as previous diagnoses of myocardial infarction, cerebral infarction, cerebral haemorrhage, or extremity thrombosis, or patients who were previously undiagnosed but were identified as having an infarct or thrombosis during the admission examination. In this study, to better capture patients at high risk of cardiovascular and cerebrovascular diseases, we also judged patients with coronary heart disease who were not diagnosed with myocardial infarction as CCD positive. Regarding LD, positive patients also included patients with previously diagnosed chronic bronchitis and chronic obstructive pulmonary disease at baseline.

It should be noted that because the prefracture ALI was included in the telephone interview, when the ALI description in the medical history collection was different from the results obtained from the telephone interview, the results of the telephone interview were used.

To ensure the reliability of data collection, we checked the original data again when necessary to avoid human error or find a reasonable explanation for unreliable data. For example, we encountered extreme values when data cleaning (such as extremely high or low blood sugar values) or situations that should have been the same but were inconsistent (such as the inconsistency between the days from hospitalization to surgery and the number of people undergoing surgery).

Sample Size

There are few studies on the required sample size for validation studies, let alone validation studies on models of survival data. Some empirical studies have shown that for external validation of prognostic models, a minimum of 100 the effective sample size is needed, and the ideal effective sample size is 200 or more (13–15). If this criterion is followed, for this study to achieve an unbiased and accurate estimation of the performance of the prognostic model, the study population should have at least 100 deaths. Given the relatively fixed number of hip fracture patients admitted each year, the possible sample size can only be met by expanding the time frame of the study. However, going back too far will lead to difficulties in follow-up, which will lead to a decrease in the accuracy of the outcome information. Therefore, we had to strike a balance between ensuring the credibility of the follow-up results and expanding the study time frame. Under this premise and referring to the information of approximately 10% mortality in 1 year in the developmental study sample, we collected a dataset with a sample size of approximately 1000 to obtain 100 deaths.

Missing Data

We estimated missing values using advanced multivariate model imputation techniques. Although studies have shown that the use of multiple imputation (MI) methods approximates the obtained data to the true values, they are complex and sometimes unnecessary(16, 17). Some studies have shown that when the loss to follow-up rate is less than 10%, there is no significant difference between MI and other simple estimation methods(18). Empirical studies show that nonstatistician-friendly single imputation (SI) results in model prediction studies are not significantly different from MI results (19). Therefore, we chose SI to handle missing data (using the mice package in R). When implementing SI, all 14 variables entered into the model were referenced.

Statistical Analysis Methods

In the model developmental study(1), we clearly reported the mathematical equation of the SFHF model and the baseline hazard at 1 year after low-energy hip fracture. Therefore, we can easily obtain the prognostic index (PI) of the model and use it to calculate the one-year survival rate of each individual in the validation study. Although our developmental study provided a nomogram developed based on a simple model, we declined to use the nomogram to calculate the survival of patients in the validation cohort because it was inefficient and prone to error.

In the model developmental study, we provided the full model and the reduced model; therefore, we evaluated both models.

Traditional Measures

We evaluated the performance of the SFHF model by examining its discrimination and calibration on an external validation dataset (20).

Specifically, for this study, discrimination refers to the ability of the model to distinguish patients with shorter survival times after hip fracture (predicting patients as high risk) from those with longer survival times (predicting patients as low risk)(21, 22). We gave Harrell's c-index and Uno's c-value, the latter being more suitable for the censored survival data model(5, 21–23). The value of c ranged from 0.5 to 1, with 0.5 indicating that the model has no discriminative ability and 1 indicating perfect discrimination. Generally, a c value greater than 0.7 indicates that the discrimination of the model is acceptable.

Calibration refers to the degree of agreement or consistency between the 1-year survival rate predicted by the model and the actual 1-year survival rate of the patient, that is, the accuracy of the prediction. (24) Some studies have compared the pros and cons of mean calibration, weak calibration, moderate calibration and strong calibration and concluded that model development and validation research should focus on moderate calibration because it ensures that clinical decisions based on the model will not lead to harm(15). Therefore, we gave the calibration plot of the model to judge the calibration of the model(24). The specific method was to calculate the 1-year survival rate of each individual in the validation data using the SFHF model. The cohort was divided into ten equal groups according to the magnitude of the survival rate, after which the average predicted survival rate (as the x-axis) and the actual survival rate (as the y-axis) of each group were obtained; the bootstrapping procedure of 1 000 repetitions was used to obtain corrected values, which were added to a calibration plot for intuitive comparison. If each group of patients had a perfect prediction, their respective values would fall on the standard 45-degree straight line in the graph.

We also grouped the validation cohort by prognostic index size and plotted Kaplan‒Meier curves superimposed with Kaplan‒Meier curves of the developmental cohort. The discrimination and calibration performance of the model can also be obtained intuitively from this figure. If the survival curves for the risk groups in the validation cohort are separated to the same extent as the developmental data, then the discriminative power of the model is preserved in validation; if the degree of separation between the sets of curves decreases, it indicates that the model is less discriminative and vice versa. The model has good calibration if the sets of curves from the two studies are intertwined and follow the same trend. Otherwise, the calibration is poor.

We also calculated the overall performance measure, the Nagelkerke R², to see the model's ability to explain the variation in results in the validation dataset(25). Its value ranged from 0 to 1. The larger the value was, the greater the ability of the model to explain the variation and the higher the prediction accuracy of the model(26).

Utility Measures

Using metrics such as model discrimination and calibration to judge the performance of a model is more of a statistical perspective that does not provide information on whether the model has clinical value(27–29). The original intention of developing and validating prognostic models was to provide references for clinical decision-making. In this study, the original intention of the SFHF model was to identify patients at high risk of dying within 1 year after fragility fracture and to treat such patients conservatively in an attempt to reduce the mortality rate among these patients. Whether the application of the model can achieve this clinical goal with the traditional model performance measures mentioned earlier cannot answer this question. Decision curve analysis (DCA), which has emerged in recent years, can link the model with clinical consequences and answer the most basic and most important question of whether the use of the model can promote clinical development(27, 28, 30). Therefore, we performed DCA to judge whether the SFHF model had clinical value in the validation population.

Decision curve analysis can tell us within which probability threshold range a model is valuable and give the magnitude of the net benefit(NE)(28, 31). The threshold probability means that under the probability of risk, for example, the expected benefit of choosing an intervention is equal to the expected benefit of rejecting the intervention(27, 32). A model is considered clinically valuable if it achieves a higher net benefit than the default strategy (treat all or treat none) within a reasonable threshold. Among them, whether this threshold range is reasonable depends on how much risk an individual is willing to take on a certain intervention.

It is important to stress here that if the validation results showed poor calibration in the SFHF model, then we simply revised the model to make it better suited to the new environment(33, 34). Otherwise, we did not update the model.

Risk Groups

Another purpose of establishing the SFHF model was to group patients with hip fractures according to the prognostic index to achieve clinical stratified management of such patients. The grouping standard was still inconclusive, and the patients were mostly divided into 3 or 4 groups based on clinical needs(4, 35). In the developmental study, we divided patients into low-, intermediate- and high-risk groups, each with equal numbers. In the validation study, however, we regrouped the developmental data in an attempt to maximize between-group differences and minimize within-group differences(5, 35). Our grouping method was as follows: according to the size of the prognostic index, in percentiles, the cut-off points were 0.15, 0.5, and 0.9. This resulted in a worst prognosis group of 10% of the total sample, close to the 1-year mortality rate (9.25%) shown in the developmental study. In this study, we also divided the validation cohort into 4 groups according to the abovementioned grouping method based on the prognostic index obtained by the SFHF model in the validation population.

Development Versus Validation

To help readers understand the differences in case mix between the two studies more clearly, we provided special explanations for the differences in setting, patient inclusion criteria, predictors, and outcome definitions and measurements between the validation and developmental studies.

The dataset for the developmental study came from Fenyang Hospital, a comprehensive teaching hospital for secondary care, in Luliang city, Shanxi Province, China (located in western central Shanxi). Among the patients admitted, the majority (approximately 80%) were from rural areas. The dataset for the validation study came from Shanxi Bethune Hospital in China, a level 3 regional referral hospital located in Taiyuan, the provincial capital, which belongs to the comprehensive teaching hospital of Shanxi Medical University. The rural population accounted for approximately 60% of the patients treated. Therefore, the two hospitals are not only geographically different but also have substantially different levels.

Among the predictors, we redefined CCD and LD (see above for details), expanded the range of diseases determined as positive results, and made disease definitions more truly reflect individual patient characteristics. All variables were coded in the same way as in the developmental study. SCs with extreme values were also winsorized (creatinine values > 99th percentile were contracted to 99th percentile values). We also made slight adjustments to the patient inclusion criteria. The developmental study excluded patients with hip fractures coexisting with other fractures. In the validation study, eligible patients were allowed to combine fragility fractures of other sites caused by the same trauma that caused the hip fracture, regardless of whether the fracture had been operated on. These combined fractures are the most prone to fragility fractures, including distal radius, proximal humerus, and vertebral compression fractures. The reason for this adjustment was because this condition is not uncommon in the clinic.

There was no difference between the two studies in the outcome of interest, which was all-cause mortality within 1 year of fracture.

In general, compared with the developmental study, this validation study differed considerably in the research setting and differed slightly in the inclusion criteria and the definition of predictors but did not differ in the target results.

Participants

This study ultimately included 877 eligible patients with fragility hip fractures, 47 of whom were lost to follow-up. Among the 830 patients who were successfully followed up, 87 died within 1 year (shown in Figure. 1). The follow-up time ranged from 7 days to 2295 days, with a median follow-up time of 829 days. Of course, because we focused on survival or death within 1 year (365 days) after fracture, follow-up beyond 1 year was truncated to 365 days. The characteristics of the eligible participants are shown in Table 1.

Table 1

Participant Characteristics
Characteristics	Missing Values, n (%)	Value
Sociodemographic characteristics
Mean age (years)	0	76.3 (SD, 9.6) (range 50–97)
Male	0	298 (34.0%)
Medical insurance	0
Employee medical insurance (EMI)		301 (34.3%)
Non-EMI		576 (65.7%)
Fracture-related
Fracture site	0
Femoral neck		447 (51.0%)
Intertrochanteric		414 (47.2%)
Intertrochanteric		16 (1.8%)
Fracture type	0
Primary		696 (79.4%)
Secondary		181 (20.6%)
Fracture to admission (d)	0	4.0 (SD, 13.2) (range 0-216)
Admission to surgery (d) (n = 769)	0	4.6 (SD, 3.5) (range 0–38)
Length of stay (LOS)	0	12.1 (SD, 9.1) (range 1-130)
Medical history
Diabetes	2 (0.2%)	190 (21.7%)
Hypertension (HYP)	2 (0.2%)	440 (50.3%)
Malignancy (MLA)	0	62 (7.1%)
Kidney disease (KD)	0	18 (2.1%)
Lung disease (LD)	28 (3.2%)	191 (22.5%)
Ability to live independently (ALI) (no = 0, yes = 1)	0	110 (12.5%)
Cardiovascular and cerebrovascular disease (CCD)	1 (0.1%)	477 (54.5%)
Clinical indicators
Blood sugar (BS) (mmol/L)	166 (18.9%)	6.7 (SD, 2.4) (range 3.2–25.1)
Serum creatinine (SC) (µmol/L) *	53 (6.0%)	79.3 (SD, 41.7) (range 26.5-378.3)
Haemoglobin (Hb) (g/L)	20 (2.3%)	116.6 (SD, 20.2) (range 47.0-218.0)
Albumin (ALB) (g/L)	43 (4.9%)	35.3 (SD, 4.4) (range 18.9–52.3)
Mean arterial pressure (MAP) (mmHg)	0	100.0 (SD, 14.3) (range 59.3-147.3)
Partial pressure of oxygen (PaO2) (mmHg)	289 (33.0%)	72.5 (SD, 20.3) (range 25.8-184.4)
Total protein (TP) (g/L)	103 (11.7%)	63.1 (SD, 6.6) (range 41.1–93.5)
Treatment
Osteodistraction	1 (0.1%)	96 (11.0%)
Surgery	0	769 (87.7%)
Note. SD: standard deviation
* Value before SC was winsorized: 82.1 (SD, 64.9) (range 26.5-854.6).

To help readers clearly understand the difference between the validation cohort and the developmental cohort, we list the characteristics of the two cohorts in a comparison table (shown in Table 2). Because the level of the validation research hospital was higher than that of the developmental research hospital, the patients admitted by the former were more complicated and serious. This can be seen from the fact that the proportion of patients with secondary hip fractures and patients with malignancies in the validation study was substantially larger than that in the developmental study.

Table 2

Comparison of Participant Characteristics between the Developmental and Validation Cohorts
Characteristic	Developmental Cohort (n = 735)	External Validation Cohort (n = 877)
Setting
Mean age (years)	74.8 (SD, 9.5) (range 50–103)	76.3 (SD, 9.6) (range 50–97)
Male	279 (38.0%)	298 (34.0%)
EMI	101 (13.7%)	301 (34.3%)
Femoral neck	305 (41.5%)	447 (51.0%)
Intertrochanteric	413 (56.2%)	414 (47.2%)
Secondary fracture	46 (6.3%)	181 (20.6%)
Admission to surgery (d)	5.5 (SD, 3.3) (range 1–45)	4.6 (SD, 3.5) (range 0–38)
LOS	13.0 (SD, 6.4) (range 1–52)	12.1 (SD, 9.1) (range 1-130)
Predictors
HYP	357 (48.6%)	440 (50.3%)
MLA	18 (2.4%)	62 (7.1%)
ALI	107 (14.6%)	110 (12.5%)
CCD	295 (40.1%)	477 (54.5%)
SC (µmol/L)	70.3 (SD, 23.8) (range 26.0–190.8)	79.3 (SD, 41.7) (range 26.5-378.3)
ALB(g/L)	37.9 (SD, 4.1) (range 21.8–48.2)	35.3 (SD, 4.4) (range 18.9–52.3)
Surgery	637 (86.7%)	769 (87.7%)
Outcome
Death within 1 year	68	87
Lost to follow-up	11 (1.5%)	47 (5.4%)

Model Performance

To make it easier for readers to understand the differences in the predictions obtained by the SFHF model in the two cohorts, we show a comparison of the distributions of the PIs from the two studies. PI is the weighted sum of the variables in the model, where the weights are the regression coefficients (4, 5, 12). The comparison of the PI distribution of the two studies is shown in Table 3 and Figure. 2. There are no obvious outliers in the validation data, and the PI distribution in the validation study is wider than that in the developmental study.

Table 3

Comparison of PI distributions in the developmental and validation studies
Measurement	Developmental study		Validation study
	FULL	LASSO	FULL	LASSO
Mean	-1.250	-1.241	-0.736	-0.740
Median	-1.293	-1.286	-0.835	-0.859
Range of values	-4.071, 2.563	-3.963, 2.566	-3.785, 3.890	-3.693, 3.841
Range	6.634	6.529	7.675	7.534
SD	1.211	1.194	1.278	1.253
Note. SD: standard deviation

Discrimination

Harrell's c-index, Uno's c-value and the Nagelkerke R² of the SFHF model are shown in Table 4.

Table 4

Discriminative performance measurements of the SFHF model
Measurement	Developmental			Internal Validation (B = 1000)		External Validation
	Harrell’s c-index (SE)	Uno’s c-value (SE)	R²	Harrell’s c-index	R²	Harrell’s c-index	Uno’s c-value	R²
Full model	0.816 (0.025)	0.817 (0.025)	0.188	0.789	0.147	0.764 (0.024)	0.765 (0.024)	0.144
LASSO model	0.814 (0.026)	0.815 (0.025)	0.187	0.795	0.158	0.763 (0.024)	0.763 (0.024)	0.144

Calibration

Calibration plots of the full and simplified SFHF models in the validation data are shown in Figure. 3 and Figure. 4.

As shown in Fig. 3, most of the validation study populations were concentrated in the high survival direction. The difference in survival between the group with the lowest predicted survival rate (mean predicted survival rate = 0.579) and the group with the second lowest survival rate (0.808) was the most significant. The difference between groups decreased as predicted survival rates increased. The differences between the 5 groups with the highest survival rates were basically the same. Confidence intervals for predicted risks for all groups covered this 45° ideal line.

Figure 4 shows that the calibration map features of the LASSO (Least absolute shrinkage and selection operator) model are basically the same as those of the full model.

Risk Groups

After obtaining the prognosis of each patient, we divided the population into four groups according to the aforementioned grouping method and plotted the Kaplan‒Meier curves of each group. We overlapped the survival curves of the two studies for comparison (shown in Figure. 5). The specific survival rates for each risk group in the two studies are shown in Table 5.

Table 5

Survival rates for the respective risk groups in the developmental and validation studies
Risk group	Developmental data				Validation data
	N (risk)	N (mean)	Survival	SE	N (risk)	N (mean)	Survival	SE
Very good	95	0	1	0	111	2	0.985	0.011
Good	211	9	0.964	0.012	242	14	0.953	0.012
Bad	219	30	0.891	0.019	261	43	0.873	0.018
Very bad	38	29	0.594	0.059	51	34	0.610	0.052

Figure 5 shows that, in general, the survival curves of each group in the validation study are in good agreement with those in the developmental study. Specifically, in the very good and bad groups, it seems that the validation curves were systematically lower than the developmental curves, but in the good and very bad groups, the curves of the two studies were almost identical. The degree of separation between the respective curves of the two studies was similar, and the degree of separation between the curves of the good group and the bad group in the validation study was even better than that in the developmental study.

Clinical usefulness

The DCA of SFHF in the validation cohort is shown in Figure. 6.

Figure. 6 shows that when the risk threshold is in the range of 0.03 ~ 0.38, the net benefit of using SFHF for clinical decision-making is better than the default treat all or treat none strategies. Between 0.38 and 0.50, the net benefit becomes unstable. Compared with the treat none strategy, the corresponding net benefit range of using the SFHF model is 0.105 (risk threshold = 0; at this time, using the SFHF model is equivalent to adopting the treat all strategy) ~ 0 (0.38). Compared with the treat all strategy, the net benefit value of using the SFHF model ranges from 0 (0.03) to 0.046 (0.11, at this time the net benefit of the treat all strategy is 0). The decision curves of the full model and the simple model are basically the same.

Model Updating

Given the good performance of model calibration in calibration plots and survival curve plots, we did not consider updating the SFHF model.

Limitations

Any study has certain limitations, and this research is no exception. First, due to the inherent deficiencies in the retrospective study design, there was a lack of initiative in the collection of patients' individual characteristics. Some predictors were not available, which resulted in lost opportunities to evaluate some similar models. At the same time, the measurement of data cannot be standardized in advance as in prospective studies; this affects the accuracy of the measurement of various variables to a certain extent, thereby affecting the reliability of the verification results. Second, the effective sample size of this study was 93 deaths. This number was close to the currently considered minimum requirement of at least 100 valid events for external validation studies, but it is still far from the ideal requirement (the ideal valid sample size is 200 or more). This somewhat affects the robustness of the findings. Third, compared with other outcomes, such as postoperative function and pain, all-cause mortality is relatively objective, so retrospective follow-up of death is reliable. However, for patients who have been deceased for 3 years or more, retrospective telephone follow-up for the specific time of death is difficult, and its accuracy is still not satisfactory compared with the follow-up results of prospective studies. This also affects the reliability of the study. Fourth, this study has geographic/broad validation, and it is a moderate-intensity study (5, 36). However, a validation study performed by a developer must have a certain degree of bias compared with a study organized by others. Fifth, although the population of the validation study came from other institutions, the two institutions are geographically adjacent, and there is a small overlap in the patients who are admitted. This situation affects the evaluation of the transportability of the model. This needs to be performed with great caution if the model is to be considered for use in the wider region of China. Sixth, the validation study showed that the Nagelkerke R² was 0.144, showing that there is still much room for improvement in the ability to explain outcome variation. Such a result can be expected; after all, we only used preoperative characteristics that were easily and quickly available in the short term after admission as predictors and excluded intraoperative and postoperative characteristics (such as intraoperative or postoperative complications) that accounted for more variation in mortality outcomes(37). Seventh, there may be patients' families and doctors who are more concerned about the risk of death in a shorter period of time after fracture, such as six months or three months after fracture, which can provide more sufficient reasons for the decision to avoid surgery. Risk predictions for these time points were less robust due to the smaller effective sample size in shorter time periods. Therefore, it is not recommended to use this model to predict the risk of death in a shorter period of time.

Interpretation

As mentioned earlier, from the perspective of the difference in setting and definition of individual predictors between the two studies, this study has geographic or broad validation, so the research results are moderately strong. Therefore, we focused on the transportability of the model rather than the reproducibility. From the above research results, it can be concluded that the performance of the simple LASSO model is basically indistinguishable from that of the full model. Therefore, here, we only explain the results of the LASSO model. From the PI comparison of the two studies, the SFHF model provided a similar amount of prognostic information on the two sets of data (2). According to the grouping method of this study, the prognostic information value of the model in the developmental study was mortality in the highest risk group and mortality in the lowest risk group, that is, 0.392-0 = 0.392. In the validation study, it was 0.386 − 0.015 = 0.371. The amount of prognostic information available in developmental studies was largely maintained in the validation data. From this perspective, the accuracy and generalizability of the model are confirmed. In terms of discrimination, although Harrell's c-index changed from the original greater than 0.8 (good) to 0.763 (acceptable), judging from the numbers, the discrimination declined. However, given the marked differences in the case mix between the two studies, we have good reason to believe that the SFHF model showed good discriminative power in validation. To further confirm the good discriminative power of the SFHF model, we also provided Uno's c-statistic, which is more applicable to survival data with censored outcomes. From the comparison of Uno’s c-value with Harrell's c-index, the discriminative power of the SFHF model was very stable in validation (see Table 4). In addition, the good discrimination is also visualized in Fig. 5, where the survival curves of the high-risk population in the validation study were clearly separated from those of the other groups, confirming the good ability of the model to identify the high-risk population. Some scholars believe that the best indicator for judging the usefulness of a model is the ability to separate different risk groups. If we follow this standard, we can say that the SFHF model has been successfully validated(35).

From the calibration point of view, the predicted probability of the SFHF model is relatively accurate. The patients were divided into 10 groups according to the predicted risk probability obtained from the model, and compared with the actual observation results of each group, there was no significant difference between the two. This is reflected in the fact that the predicted risk confidence intervals for each group covered the ideal standard line for perfect prediction. Likewise, the accuracy of the prediction can be visually judged in Fig. 5. In Fig. 5, although the very good and bad groups seem to have slightly systematically high predictions, in the high-risk group we are most concerned about, the survival curves of the two groups of data are intertwined with a high degree of agreement. Overall, the survival curves of the respective risk groups in the two studies were in good agreement, confirming good calibration of the SFHF model.

Research shows that accurate models are not necessarily useful(26, 27, 38). Therefore, it is insufficient to conclude that a model has been successfully validated based solely on statistical accuracy. In terms of clinical usefulness, the SFHF model has great practical value in externally validated populations. In DCA, when considering the risk threshold between 0.03 and 0.38, the net benefit of the model was better than that of the treat everyone or treat no one strategies. When we predict a patient's 1-year risk of death to be low, i.e., less than 0.05, it is clear that doctors and patients will not hesitate to choose surgery when choosing treatment. At this risk level, it is clear that surgery will benefit most patients. Conversely, when the risk of death is obvious, i.e., greater than 0.5, we will not hesitate to choose conservative treatment. It is when the risks lie in the middle that doctors and patients are confused. Our DCA results show that it is within these risk thresholds that clinical decision-making based on our model is highly valuable. For example, if we set the 1-year risk of death threshold at 10%, if the patient's predicted risk of death exceeds 10%, conservative treatment is considered; otherwise, surgery is considered compared with the option of surgical treatment for all patients (treat none), NE = 0.049. This means that the net result of making decisions based on the SFHF model is that out of 100 patients, we conservatively treated 4.9 patients who would die within 1 year, compared to treating all patients with surgery and surgery-treated patients who would not die within 1 year. In contrast to the conservative treatment strategy for all patients (treat all), the model's NE = 0.049 − 0.010 = 0.039, calculated using the net benefit formula 0.039×100/(0.1/0.9) = 35.1. This means that using the predictive model, conservative treatment decreased by 35.1% among patients who did not die within 1 year, whereas the number of patients who would die within 1 year of surgical treatment would not increase(27). If compared with surgery for all patients as the default strategy, then the NE difference obtained by choosing the decision-based model, ΔNE = 0.049-0 = 0.049, the test trade-off is 1/0.049 = 20.4. If we are willing to apply the model to 20 patients to find a patient who will die within 1 year of a fracture, then the model is valuable(28). Unfortunately, we cannot provide a reasonable risk threshold here, as it varies from individual to individual and involves the need to reasonably evaluate all possible outcomes(28).

Overall, the reproducibility and generalizability of the SFHF model in this external study were confirmed, both by traditional metrics and by clinical usefulness. At this point, we can say that the SFHF model is a model that was initially successfully validated. Unlike the Li 2021(37) and Endo 2018(39) models that use intraoperative or postoperative indicators as predictors, our model only includes preoperative individual characteristics that can be quickly and easily obtained after admission. Additionally, different from the model in Ma 2018(40), which requires multidisciplinary collaboration to obtain accurate predictors, the SFHF model does not require the assistance of multidisciplinary consultation, and the predictors can be independently determined by an orthopaedic surgeon 48 hours after admission. This outstanding feature of convenience, speed, and accuracy provides the most fundamental guarantee for the model to enter clinical practice in the future. There is no complex scoring system in the predictors of the model, avoiding the use of the model as a predictor as in the Hjelholt 2022(41), Trevisan 2021(7), Li 2021(37), and Söderqvist 2009(42) studies. No interaction variable is introduced, which is easy to understand and explain.

Implications

The initial verification of the SFHF model was successful, confirming that the strategy we chose to develop the model based on a small sample is feasible. That is, according to the principles and purposes of modelling, combined with professional knowledge and existing research results, the number of input variables is controlled instead of using a data-driven variable selection strategy to avoid overfitting. Of course, based on the limited strength of this validation study, we cannot yet conclude that the SFHF model has been fully validated. Therefore, in future research, on the one hand, we should urgently further increase the validation strength. For example, nondevelopers should validate the model using a more heterogeneous, sufficient and relevant population from other regions of China (e.g., eastern and southern China, etc.) to further evaluate the generality and accuracy of the SFHF model. On the other hand, given the effectiveness of the preliminary validation, we may consider conducting model impact studies in our hospital to investigate whether the clinical application of the SFHF model can reduce 1-year mortality after hip fracture among elderly individuals compared with not using the model (3, 43). In addition, in the study of prognostic factors, this model can also be used as a reference for multivariate adjustment analysis.

From the results of the study, we can know that : from a statistical point of view, the SFHF model has good discrimination and calibration, that is, the model has certain accuracy and generalizability; from a clinical point of view, the model has clinical Value. Therefore, we can consider applying this model to the management of elderly patients with hip fracture, and formulate treatment plans accordingly, with the expectation of reducing 1-year mortality in this population.

SFHF: Survival after fragility hip fracture; DCA: Decision curve analysis; CI: Confidence interval; LOS: length of stay; ALI: Ability to live independently; LD: lung disease; CCD: Cardiovascular and cerebrovascular disease; KD: Kidney disease; MAL: Malignancy; HYP: Hypertension; MAP: Mean arterial pressure; PaO₂: Partial pressure of oxygen; Hb: Haemoglobin; SC: Serum creatinine; BS: Blood sugar; ALB: Albumin; TP: Total protein; SUR: Surgery; MI: Multiple imputation; SI: Single imputation; PI : Prognostic index ; LASSO: Least absolute shrinkage and selection operator ; NE : Net benefit.

Ethics approval and consent to participate

This study was approved by the Medical Ethics Committee of Shanxi Bethune Hospital. Informed consent was waived by the Medical Ethics Committee of Shanxi Bethune Hospital.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used in this study are available from the corresponding authors upon reasonable request, if required by the reader.

Competing interests

The authors declare that they have no competing interests.

Funding

This study did not receive any funding.

Authors' contributions

In this study, Hairui Fu participated and was responsible for all the research process, Feixiong Li and Fangfang Wei were responsible for data collection, follow-up and verification, and participated in the writing of the paper. Weihan Qu and Xuan Zhang participated in the verification and analysis of the data, Bin Liang provided expertise and study design guidance, and Dou Wu and Qiang Liu provided support for the study in terms of ethics approval, hospital department coordination, statistics, and clinical knowledge.

Acknowledgments

The authors thank teachers from the hospital's medical record room, information centre, and orthopaedic reference room for their help during the data collection process. Thanks to my wife, Xingying Li, for the help during the phone interview. Thanks to my colleagues Enzhe Zhao and Liming He for their help during the information verification process.

Fu H, Liang B, Qin W, Qiao X, Liu Q. Development of a prognostic model for 1-year survival after fragile hip fracture in Chinese. J Orthop Surg Res. 2021 Nov 27;16(1):695.
Altman DG, Royston P. What do we mean by validating a prognostic model? Statist Med. 2000 Feb 29;19(4):453–73.
Moons KGM, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009 Jun 4;338:b606.
Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol. 2013 Dec;13(1):33.
Ramspek CL, Jager KJ, Dekker FW, Zoccali C, van Diepen M. External validation of prognostic models: what, why, how, when and where? Clinical Kidney Journal. 2021 Feb 3;14(1):49–58.
Justice AC. Assessing the Generalizability of Prognostic Information. Ann Intern Med. 1999 Mar 16;130(6):515.
Trevisan C, Gallinari G, Carbone A, Klumpp R. Efficiently stratifying mid-term death risk in femoral fractures in the elderly: introducing the ASAgeCoGeCC Score. Osteoporos Int. 2021 Oct;32(10):2023–31.
Menéndez-Colino R, Gutiérrez Misis A, Alarcon T, Díez-Sebastián J, Díaz de Bustamante M, Queipo R, et al. Development of a new comprehensive preoperative risk score for predicting 1-year mortality in patients with hip fracture: the HULP-HF score. Comparison with 3 other risk prediction models. Hip Int. 2021 Nov;31(6):804–11.
Cenzer IS, Tang V, Boscardin WJ, Smith AK, Ritchie C, Wallhagen MI, et al. One-Year Mortality After Hip Fracture: Development and Validation of a Prognostic Index. J Am Geriatr Soc. 2016 Sep;64(9):1863–8.
Jiang HX, Majumdar SR, Dick DA, Moreau M, Raso J, Otto DD, et al. Development and initial validation of a risk score for predicting in-hospital and 1-year mortality in patients with hip fractures. J Bone Miner Res. 2005 Mar;20(3):494–500.
van de Ree CL, Gosens T, van der Veen AH, Oosterbos CJ, Heymans MW, de Jongh MA. Development and validation of the Brabant Hip Fracture Score for 30-day and 1-year mortality. Hip Int. 2020 May;30(3):354–62.
Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration. Ann Intern Med. 2015 Jan 6;162(1):W1–73.
Steyerberg EW. Patterns of External Validity. In: Steyerberg EW, editor. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]. Cham: Springer International Publishing; 2019 [cited 2022 Jun 22]. p. 367–97. (Statistics for Biology and Health). Available from: https://doi.org/10.1007/978-3-030-16399-0_19
Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Statistics in Medicine. 2016;35(2):214–26.
Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology. 2016 Jun;74:167–76.
Harel O, Zhou XH. Multiple imputation: review of theory, implementation and software. Statistics in Medicine. 2007;26(16):3057–77.
Janssen KJM, Donders ART, Harrell FE, Vergouwe Y, Chen Q, Grobbee DE, et al. Missing covariate data in medical research: To impute is better than to ignore. Journal of Clinical Epidemiology. 2010 Jul 1;63(7):721–7.
Imputations of Missing Values in Practice: Results from Imputations of Serum Cholesterol in 28 Cohort Studies | American Journal of Epidemiology | Oxford Academic [Internet]. [cited 2022 Jun 23]. Available from: https://academic.oup.com/aje/article/160/1/34/107200?login=true
Missing Values | SpringerLink [Internet]. [cited 2022 Jun 23]. Available from: https://link.springer.com/chapter/10.1007/978-3-030-16399-0_7
D’Agostino RB, Nam BH. Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures. In: Handbook of Statistics [Internet]. Elsevier; 2003 [cited 2022 Jun 23]. p. 1–25. (Advances in Survival Analysis; vol. 23). Available from: https://www.sciencedirect.com/science/article/pii/S0169716103230017
Pencina MJ, D’Agostino Sr. RB, Song L. Quantifying discrimination of Framingham risk functions with different survival C statistics. Statistics in Medicine. 2012;31(15):1543–53.
McLernon DJ, Giardiello D, Calster BV, Wynants L, Geloven N van, Smeden M van, et al. Assessing performance and clinical usefulness in prediction models with survival outcomes: practical guidance for Cox proportional hazards models [Internet]. medRxiv; 2022 [cited 2022 Jun 30]. p. 2022.03.17.22272411. Available from: https://www.medrxiv.org/content/10.1101/2022.03.17.22272411v1
Uno H, Cai T, Pencina MJ, D’Agostino RB, Wei LJ. On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statist Med. 2011 May 10;30(10):1105–17.
Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016 Aug 1;25(4):1692–706.
Steyerberg EW. Evaluation of performance. In: Clinical prediction models: A practical approach to development, validation, and updating [Internet]. Cham: Springer International Publishing; 2019. p. 277–308. Available from: https://doi.org/10.1007/978-3-030-16399-0_15
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures. Epidemiology. 2010 Jan;21(1):128–38.
Vickers AJ, Elkin EB. Decision Curve Analysis: A Novel Method for Evaluating Prediction Models. Med Decis Making. 2006 Nov;26(6):565–74.
Van Calster B, Wynants L, Verbeek JFM, Verbakel JY, Christodoulou E, Vickers AJ, et al. Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators. European Urology. 2018 Dec 1;74(6):796–804.
Vergouwe Y, Steyerberg EW, Eijkemans MJC, Habbema JDF. Validity of prognostic models: when is a model clinically useful? Seminars in urologic oncology. 2002 May;20(2):96–107.
Vickers AJ, Cronin AM. Traditional Statistical Methods for Evaluating Prediction Models Are Uninformative as to Clinical Value: Towards a Decision Analytic Framework. Seminars in Oncology. 2010 Feb 1;37(1):31–8.
Decision Analysis for the Evaluation of Diagnostic Tests, Prediction Models, and Molecular Markers: The American Statistician: Vol 62, No 4 [Internet]. [cited 2022 Jun 23]. Available from: https://www.tandfonline.com/doi/abs/10.1198/000313008X370302
Localio AR, Goodman S. Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision Making. Ann Intern Med. 2012 Aug 21;157(4):294–5.
Steyerberg EW. Updating for a New Setting. In: Steyerberg EW, editor. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]. Cham: Springer International Publishing; 2019 [cited 2022 Jun 23]. p. 399–429. (Statistics for Biology and Health). Available from: https://doi.org/10.1007/978-3-030-16399-0_20
Validation, calibration, revision and combination of prognostic survival models - van Houwelingen – 2000 - Statistics in Medicine - Wiley Online Library [Internet]. [cited 2022 Jun 23]. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/1097-0258(20001230)19:24%3C3401::AID-SIM554%3E3.0.CO;2-2
Altman DG. Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer. Cancer Investigation. 2009 Jan;27(3):235–43.
Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012 May 1;98(9):691–8.
Li Y, Chen M, Lv H, Yin P, Zhang L, Tang P. A novel machine-learning algorithm for predicting mortality risk after hip fracture surgery. Injury. 2021 Jun;52(6):1487–93.
Vickers AJ. Decision Analysis for the Evaluation of Diagnostic Tests, Prediction Models, and Molecular Markers. The American Statistician. 2008 Nov;62(4):314–20.
Endo A, Baer HJ, Nagao M, Weaver MJ. Prediction Model of In-Hospital Mortality After Hip Fracture Surgery. J Orthop Trauma. 2018 Jan;32(1):34–8.
Ma M, Lu H, Zhang P, Zhang D, Fu Z, Jiang B. Formulating a preoperative risk scoring system for elderly patients with hip fracture. Chinese Journal of Orthopaedic Trauma. 2018;1031–7.
Hjelholt TJ, Johnsen SP, Brynningsen PK, Knudsen JS, Prieto-Alhambra D, Pedersen AB. Development and validation of a model for predicting mortality in patients with hip fracture. Age Ageing. 2022 Jan 6;51(1):afab233.
Söderqvist A, Ekström W, Ponzer S, Pettersson H, Cederholm T, Dalén N, et al. Prediction of mortality in elderly patients with hip fractures: a two-year prospective study of 1,944 patients. Gerontology. 2009;55(5):496–504.
Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine. 2006;144(3):201–9.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

External validation of a prognostic model for 1-year survival after fragility hip fracture: A retrospective cohort study

Status:

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

Introduction

Methods

Data Source

Participants

Outcome

Predictors

Sample Size

Missing Data

Statistical Analysis Methods

Traditional Measures

Utility Measures

Risk Groups

Development Versus Validation

Results

Participants

Model Performance

Discrimination

Calibration

Risk Groups

Clinical usefulness

Model Updating

Discussion

Limitations

Interpretation

Implications

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1