3.1 Patient characteristics
The cohort consisted of 162 patients with RC, including 57 females (35.2%) and 105 males (64.8%), with a mean age of 63.12 ± 9.95 years. Of those, 54 had LNM and 108 had non-LNM. The cohort was randomly divided into a training cohort (n = 114) and a validation cohort (n = 48) according to 7:3 ratio. The clinical characteristics of the 162 patients in the training and validation cohorts are summarized in Table 1. There were no statistical differences in age (P = 0.335), sex (P = 0.389), or N stage (P = 1.000) between the training and the validation cohorts.
3.2 Radiomic feature selection and Radscore building
Of all texture features, 101 features were selected on the basis of the 114 patients in the training cohort using the LASSO logistic regression model. When log(lambda) was -4.004, the AUC value corresponding to the LASSO model was the highest (Figs. 2A and B), and 12 potential predictors with nonzero coefficients were retained, including SurfaceVolumeRatioshape, Medianfirstorder, Kurtosisfirstorder, Energyfirstorder, Imc2glcm, SmallAreaHighGrayLevelEmphasisglszm, GrayLevelNonUniformityNormalizedglrlm, MajorAxisLengthshape, SmallAreaLowGrayLevelEmphasisglszm, LargeAreaHighGrayLevelEmphasisglszm, GrayLevelNonUniformityNormalizedglszm, Coarsenessngtdm. Then, the Radscore for each patient was calculated according to the following formula:
Radscore=-0.7394-0.3407*SmallAreaHighGrayLevelEmphasisglszm-0.2482*GrayLevelNonUniformityNormalizedglrlm-0.0758*MajorAxisLengthshape-0.0729*SmallAreaLowGrayLevelEmphasisglszm-0.0695*LargeAreaHighGrayLevelEmphasisglszm-0.0483*GrayLevelNonUniformityNormalizedglszm+0.1241*Medianfirstorder+0.1536*SurfaceVolumeRatioshape+0.2077*Coarsenessngtdm+0.2258*Kurtosisfirstorder+0.2304*Imc2glcm+0.3735*Energyfirstorder
In the training cohort (Radscore = -0.436 vs -0.891) and the validation cohort (Radscore = -0.501 vs. -0.859), the Radscore of RC patients with LNM was significantly higher than that of non-LNM patients (training cohort: P < 0.001; validation cohort: P = 0.003). The Radscores of the two groups are shown as violin plots in Figure. 3A and B.
FIGURE 2. The LASSO algorithm and 10-fold cross-validation were used to extract the optimal subset of radiomic features. A. Optimal feature selection according to AUC value. When the value log (lambda) increased to 0.018, the AUC reached the peak corresponding to the optimal number of radiomic features. B. LASSO coefficient profiles of the 101 radiomic features. The vertical line was drawn at the value selected by 10-fold crossvalidation, where the optimal lambda resulted in 12 nonzero coefficients. LASSO: least absolute shrinkage and selection operator AUC: area under receiver operating characteristic curve
FIGURE 3. Violin plot of Radscore for LNM and non-LNM patients in training (A) and validation (B) sets. The thick black line in the middle represents the median. The black line running up and down through the violin diagram represents the range from the smallest non-outlier value to the largest non-outlier value. LNM: lymph node metastasis
3.3 Performance and clinical utility of the prediction models
The performance of the three models in predicting LNM in patients with RC was evaluated by ROC curves and compared using the DeLong test. The performance of the prediction models to identify LNM is shown in Figure 4A. The MRI-based model, Radscore model, and complex model all performed well in discriminating LNM, with AUC values of 0.882, 0.728 and 0.902, respectively. The Delong test showed that the AUC value of the complex model was significantly higher than that of the MRI-based model (P = 0.001) and Radscore model (P < 0.001), while the MRI-based model had a higher AUC than the Radscore model; however, the difference was not significant (P = 0.159).
Comparisons of the clinical utility of the models were performed using DCA. The results revealed that the complex model outperformed the MRI-based model and Radscore model in a wide threshold range (Figure 4B). Therefore, the complex model was the most reliable clinical management tool for predicting LNM in patients with RC.
FIGURE 4. ROC curves and DCA of the three prediction models. A. ROC curves for the three prediction models in differentiating lymph node metastasis in the training set. The green line indicates MRI reported model, the blue line indicates Radscore model, the purple line indicates the complex model. B. DCA of the three prediction models in the training set. The Y-axis and the X-axis represent the net benefit and threshold probability respectively. The green line indicates MRI reported model, the blue line indicatesRadscore model, the purple line indicates the complex model, the red oblique line indicates the hypothesis that all patients were lymph node metastasis, the horizontal brown line represents the hypothesis that all patients were non-lymph node metastasis. ROC: receiver operating characteristic MRI: magnetic resonance imaging DCA:decision curve analysis Radscore: radiomics signature score
3.4 Individualized nomogram construction and validation
Considering the complex model's ability to predict LNM, we developed a nomogram to represent the individual prediction based on the training cohort, and to visualize the prediction results and the proportion of each factor (Figure 5A). The AUC of the model in the training cohort (n = 114) was 0.902 (95% CI: 0.848−0.957), with a sensitivity of 0.798, a specificity of 0.868, and an accuracy of 0.842. The AUC of the model in the validation group (n = 48) was 0.891 (95% CI: 0.799−0.983), with a sensitivity of 0.812, a specificity of 0.843, and an accuracy of 0.833. The nomogram exhibited good agreement between the predicted and observed values of the training and validation sets (Figure 5B and C). The Hosmer-Lemeshow goodness of fit test showed that there was no significant difference between the predicted and observed values in either the training cohort (χ2 = 6.533, P = 0.588) or the validation cohort (χ2 = 9.116, P = 0.333), thus, indicating a good fit.
FIGURE 5. Development and performance of a nomogram. A. Nomogram based on MRI reported andRadscore. Calibration curves of the nomogram in the training (B) and validation (C) sets. The horizontal axis is the predicted incidence of LNM. The vertical axis is the observed incidence of LNM. The gray line on the diagonal is the reference line, indicating that the predicted value is equal to the actual value and the blue line is the calibration curve.Radscore: radiomics signature score LNM: lymph node metastasis