2.1 Study Population Characteristics
Of 345 patients in the training and validation cohorts, the baseline information of clinical characteristics among these patients was summarized. 141 (40.7%) died during the 24-month follow-up period. Most subjects were male (242,70.1%) and 103 (29.9%) were female. There are slightly more non-smokers (158,45.8%) than smokers (187, 54.2%). Patients with driver genomic alterations ac-counted for 33.6%. All characteristics are presented in Table 1.
Table 1. Patient Characteristics, Stratified by Death Status Within 24 Months of the Index Encounter.
|
No. (%)
|
|
Characteristic
|
Alive (n = 204)
|
Died (n = 141)
|
Age (years)
|
|
|
<65
|
127(62.3)
|
91(64.5)
|
≥65
|
77(37.7)
|
50(35.5)
|
Gender
|
|
|
Male
|
148(72.5)
|
94(66.7)
|
Female
|
56(27.5)
|
47(33.3)
|
ECOG PS score
|
|
|
0-1
|
186(91.2)
|
115(81.6)
|
2-3
|
18(8.8)
|
26(18.4)
|
Smoking
|
|
|
Never
|
186(91.2)
|
115(81.6)
|
Current/previous
|
18(8.8)
|
26(18.4)
|
Histology
|
|
|
Squamous
|
58(28.4)
|
43(30.5)
|
Non-squamous
|
146(71.6)
|
98(69.5)
|
Driver oncogene alteration
|
|
|
Mutant
|
52(25.5)
|
64(45.4)
|
Wild-type / Unknown
|
152(74.5)
|
77(54.6)
|
Line of immunotherapy
|
|
|
<2
|
101(19.5)
|
47(33.3)
|
≥2
|
103(50.5)
|
94(66.7)
|
Location of metastasis
|
|
|
Brain
|
27(13.2)
|
27(13.2)
|
Liver
|
28(13.7)
|
35(24.8)
|
Bone
|
55(27.0)
|
53(37.6)
|
Lung/pleura
|
75(36.8)
|
73(51.8)
|
2.2 Algorithm Variable Importance
The top 10 variables in terms of variable importance for the 3 algorithms are shown in Table 2. The top predictors shared across all models were mutation patterns, prognostic nutrition index (PNI) and derived neutrophil-to-lymphocyte ratio (dNLR).Lung/pelura metastasis,Inflammatory factors, dNLR, PNI and driver oncogene alteration served as irreplaceable weights at decision trees ranches (Figure 1) .SHAP is used to further explain the impact of features on theXGBoost model, it showed that high dNLR, current/previous smoking and high LMR at baseline, bone metastasis,positive driver gene mutation have remained independent risk aspects for OS. (Figure 2).
Table 2. Variable Importance in Descending Order of Feature Importance for Decision Trees , Random Forest and XGBoost*.
Algorithm
|
Variable *
|
Algorithm
|
Variable *
|
Decision Trees
|
dNLR
|
Random Forest
|
Driver oncogene alteration
|
PNI
|
dNLR
|
Driver oncogene alteration
|
PNI
|
Lung/pelura metastasis
|
TNM stage
|
NLR
|
ECOG PS score
|
White blood cells
|
Age
|
Monocytes
|
Line of immunotherapy
|
LMR
|
PLR
|
A/G
|
NLR
|
Liver metastasis
|
Smoking
|
XGBoost
|
dNLR
|
|
|
Smoking
|
|
|
LMR
|
|
|
Bone metastasis
|
|
|
Driver oncogene alteration
|
|
|
Liver metastasis
|
|
|
PLR
|
|
|
PNI
|
|
|
Lung/pelura metastasis
|
|
|
Age
|
|
|
* Variable importance is ranked by importance rating for the decision trees, random forest and XGBoost models.
Abbreviations: NLR, neutrophil-lymphocyte ratio;dNLR, derived NLR;PNI, Prognostic nutritional
index;LMR, Lymphocyte-Monocyte Ratio;PLR, Platelet-Lymphocyte Ratio.
2.3 Model Performance
Algorithm discrimination and other performance metrics in the validation set are presented for each model in Table 3. The random forest and XGBoost models had higher PPV (70.4% and 60.9%, respectively) than the decision trees model (55.8%). AUC values were calculated for all models. The top two most accurate prediction models were derived from the random forest and XGBoost. The AUC of the two models was 0.736 and 0.724, respectively. Despite hyperparameter tuning, the random forest displayed evidence of overfitting, with AUCs in the training set of 0.99.
Table 3. Performance Metrics of Machine Learning Models
Algorithm
|
Positive Predictive Value a
|
ROC-AUC a
|
Accuracy
|
F1 Score
|
Decision Trees
|
0.704 b
|
0.709
|
0.709 b
|
0.559
|
Random Forest
|
0.558
|
0.736 b
|
0.660
|
0.624
|
XGBoost
|
0.609
|
0.724
|
0.699
|
0.644 b
|
a Coprimary performance metric.
b Refers to the best-performing model(s) for each performance metric.
1.4 Predicted probability of NSCLC overall survival
Given the importance results of the ML algorithm model, mutation patterns, PNI and dNLR were selected as variables for the construction of immune risk model in NSCLC. By integrating these factors, we used nomogram algorithm to predict the probability of OS of 6, 12 and 18 months in NSCLC (Figure 3). The nomogram indicated dNLR contributed slightly less to patients’ prognosis compared with other factors. C-index were conducted for the probability of OS of 6, 12 and 18 months, and the C-index was 0.694. Furthermore, we assessed and validated its effectiveness by time-dependent ROC analysis (Figure 4) and calibration plots (Figure 5). The AUC values for OS of 6, 12 and 18 months were 0.692, 0.672, and 0.679, respectively, in the training cohort, and 0.701, 0.759, 0.834, respectively, in the validation cohort,which highlighted that the predictive model performed well in predicting overall survival of patients with advanced NSCLC receiving ICI therapy.
Based on predictive scores, the cut-off values were determined by regrouping all subjects in the entire NSCLC cohort, as well as the training and validation cohorts into three subgroups with distinct prognosis. Survival analyses showed that in the entire NSCLC, training, and validation cohorts , the low-risk groups in all cohorts were cor-related with higher OS compared with high-risk groups ((Figure 6,P<0.001 in the entire NSCLC cohort, P<0.001 in the training cohort, and P<0.001 in the validation cohort, respectively).