Data source
The SEER database (http://www.seer.cancer.gov), covering approximately 28% of the US population, is used to identify the patients in our study. The SEER database can provide information freely to registered researchers, including patient demographics, primary tumor site, tumor stage, surgical treatment, patient survival data, and so on. We obtained the permission to access the database after submitting a SEER Research Data Agreement form through e-mail. The software of SEER*Stat (version 8.3.5) was used to extract the data, and our user name was 11697-Nov2018.
Study population
The patients with metastatic RCC were identified between 2010 and 2016 using the software of SEER*Stat. Key patient eligibility criteria were as following: (1) The RCC patients who had distant metastasis at diagnosis between 2010 and 2016 were enrolled. (2) RCC was identified using morphology codes (8050/3, 8260/3, 8310/3, 8317/3,8318/3, and 8319/3) based on International Classification of Diseases for Oncology codes (3rd edition). (3) RCC was the first and only primary diagnosis. (4) All the diagnoses of RCC were confirmed by histological examination. (5) Complete follow-up data of the RCC patients could be obtained. The exclusion criteria included: age < 18 years at diagnosis, unknown follow-up data, and unknown information about race, marital status, Fuhrman grade, tumor size, tumor stage, lymph node status, metastasis, and surgery. Autopsy or death certificate cases were also excluded. Finally, a total of 2315 patients with metastatic RCC were included in this cohort. The flow diagram for patient selection was presented in Figure 1.
Measurements of variables
For each patient, the demographic and clinical variables were recorded, including age at diagnosis, race (black, white, other), sex (male, female), marital status (married, unmarried), histologic subtype (clear cell renal cell carcinoma, CCRCC; papillary renal cell carcinoma, PRCC; chromophobe renal cell carcinoma, CHRCC; sarcomatoid renal cell carcinoma, SRCC; collecting duct renal cell carcinoma, CDRCC), Fuhrman grade (grade Ⅰ, grade Ⅱ, grade Ⅲ, grade Ⅳ, unknown), Tumor classification (T1, T2, T3, T4, TX), Lymph node status (N0, N1, NX), sarcomatoid feature (yes, no, unknown), cancer-directed surgery (recommended and performed, recommended but not performed, not recommended), bone metastasis (yes, no), brain metastasis (yes, no), liver metastasis (yes, no), lung metastasis (yes, no), survival time, and vital status. The AJCC Cancer Staging Manual (7th edition, 2010) was employed to evaluate the tumor stages.
Ascertainment of the outcome
The primary outcome of this study was OS, which was defined as survival from diagnosis of metastatic RCC to death due to any cause. OS was ascertained based on the code “vital status” in the SEER database.
Statistical analysis
The selected patients were randomly assigned to a training set and a validation set at a ratio of 1:1. Descriptive statistics was initially performed to describe the baseline characteristics of the patients in training and validation sets. Continuous variables with normal distribution were shown as mean (standard deviation), and non-normal continuous variables were presented as median (interquartile range). Categorical variables were summarized in terms of frequency and percentages. For the training set, univariable and multivariable Cox regression modeling were performed to generate crude and adjusted hazard ratios (HRs) for identifying the significant prognostic factors of OS. The selection of prognostic factors was carried out using a backward stepwise process with the Bayesian information criterion. The proportional hazards assumption of Cox regression modeling was assessed with the use of Schoenfeld residuals.
Nomograms, graphic tools to quantify risks and calculate the probability of clinical events by scoring the involved factors, had been demonstrated to generate more precise prediction than the conventional AJCC staging system in several types of cancers [10, 11]. In the current study, the nomogram for predicting 1-, 3-, and 5-year OS was formulated based on the results of multivariable Cox regression model.
Discrimination and calibration, important properties in the evaluation of model performance, were both assessed in our study. C-index was applied to evaluate the discriminative ability of the nomogram, which depicted the probability of the predicted risk was higher for a random patient having an event than for a random patient not having an event. After comparing the predicted probability of events for all possible pairs of patients, C-index was 0.5 if the model could not discriminate the patients with and without events. Conversely, C-index was 1 if the probability predicted by the model was always higher for patients with events than those without events [12]. Calibration plot, the best method to visually exhibit the relationship between the predicted risk and the actual risk, was adopted in this study [12]. Calibration plots fall on a 45-degree diagonal line, reflecting excellent absolute risk estimates. NRI and IDI were usually used to assess and quantify the improvement in risk prediction between the new and old models [13]. The NRI was based on reclassification tables separately composed of patients with and without events and could quantify the correct reclassification in categories. The NRI could be calculated by adding the percentage of patients with events who were correctly reclassified to the percentage of patients without events who were correctly reclassified [12]. The IDI could reflect the improvement of sensitivity and specificity, and it also could be viewed as an integrated difference in Youden’s indices [13]. Calculating the IDI required adding the increased probability predicted by the new model compared to the old model for patients with events to the decreased probability predicted by the new model compared to the old model for patients without events [13]. NRI and IDI were both employed to compare the discriminative ability between the new model and the AJCC staging system in the current study. DCA is a method for evaluating the benefits of a diagnostic test across a range of patient preferences for accepting risk of undertreatment and overtreatment to facilitate decisions about test selection and use [14]. Unlike the sensitivity, specificity, and area under the curve, DCA could directly assess the utility of clinical risk prediction models for decision making [15]. Herein, DCA was conducted to evaluate the clinical use of the nomogram through quantifying the net benefit in comparison with the AJCC staging system.
All statistical tests were performed using R software (version 3.5.2, http://www.r-project.org/). All tests were two-sided, and the significance level was set at P<0.05.