DOI: https://doi.org/10.21203/rs.3.rs-37934/v1
Background: Preoperative prognostication of clinical and surgical outcome in patients with neurosurgical diseases can improve the risk stratification, thus can guide in implementing targeted treatment to minimize these events. Therefore, the author aims to highlight the development and validation of predictive models determining neurosurgical outcomes through machine learning algorithms using logistic regression.
Methods: Logistic regression (enter, backward and forward) and least absolute shrinkage and selection operator (LASSO) method for selection of variables from selected database can eventually lead to multiple candidate models. The final model with a set of predictive variables must be selected based upon the clinical knowledge and numerical results.
Results: The predictive model which performed best on the discrimination, calibration, Brier score and decision curve analysis must be selected to develop machine learning algorithms. Logistic regression should be compared with the LASSO model. Usually for the big databases, the predictive model selected through logistic regression gives higher Area Under the Curve (AUC) than those with LASSO model. The predictive probability derived from the best model could be uploaded to an open access web application which is easily deployed by the patients and surgeons to make a risk assessment world-wide.
Conclusions: Machine learning algorithms provide promising results for the prediction of outcomes following cranial and spinal surgery. These algorithms can provide useful factors for patient-counselling, assessing peri-operative risk factors, and predicting post-operative outcomes after neurosurgery.
The treatment of neurosurgical diseases have been revolutionized due to greater stride observed in terms of systematic therapy, radiotherapy, and surgical techniques 1–3,4,5,6,78,9. The multimodal management has resulted in achieving better clinical and functional outcomes 10–125,13,14. However, surgery for neurosurgical diseases such as spinal metastases remains the main stay of treatment for achieving durable local tumor control and spinal stabilization over other radiation and interventional techniques 15. Despite the technical expertise and surgical care, the incidence of complications in the postoperative period is not uncommon 10. As an example, the complication rates following surgery for metastatic spine disease ranges from 10-52% 16–19, 3-10% for lumbar degenerative spondylolisthesis 20, 4-11.1% for spinal osteotomies 21,22, and upto 70% for adult spinal deformity 23,24. These adverse events (AEs) include both medical and surgical issues, thus impairs the quality of life 25. Understanding of the factors that influence the postoperative outcomes is of utmost importance not only for surgical decision making, but for patient counseling as well since the benefits of improving ambulation and pain need to be weighed against the risks of postoperative AEs following surgery 10.
Furthermore, with an exponentially increasing costs associated with spinal surgery; there are several cost containment and quality improvement measures 26. The most common among all these is the introduction of the concept of “capitalized bundle care” 27, which combines the hospital facility and professional charges for necessary care for a defined longitudinal period of care. Thus, a small percentage of unanticipated event can consume a large percentage of a service line’s capitated revenue 28. To achieve financial sustainability, it is of utmost importance to stratify the risks of unplanned events 27,28. This risk stratification can help in addressing the modifiable factors in the high-risk patients, which is ultimately important to negotiate for complex bundled payments.
The understanding of postoperative outcomes laid a foundation in numerous quality-reporting metrics. This is because of the fact that the perioperative events negatively impact the clinical and patient reported outcomes 29. There exist need to develop well-defined prognostic algorithms to stratify the risks in the patients undergoing surgery for brain and spinal diseases. Developing an appropriate criterion in an interdisciplinary setting will eventually lead to avoid complications and reduce the financial burden associated with it. Although studies have applied machine learning algorithm to predict postoperative outcomes using machine learning algorithms 30–33. However, to date, a few studies focused on using logistic regression for developing and validating the machine learning algorithm 34,35. Therefore, the purpose of this study was to understand, (i) how to develop a set of predictive factors for postoperative outcomes following spine surgery using machine learning algorithms and (ii) to create an open accessible user-interface to prognosticate postoperative outcomes which is easily accessible to individuals all over the world.
STUDY DESIGN AND DATA SOURCES:
Databases, preferably big data analytics like multi-institutional clinical registries 36,37,38 have been used for developing the predictive algorithms. The de-identified data from large national database usually get exemption from review by institutional review board.
GUIDELINES:
The Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis (TRIPOD) and JMIR Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research 39,40 are followed.
SPLIT-SAMPLE APPROACH: TRAINING VERSUS VALIDATION SET:
Based on the split proportions in the previous literature 41, the data is randomly divided into a training dataset (70%) and a validation dataset (30%). A set of predictive models for various types of postoperative outcomes like AEs, extended length of hospital stay etc. for the training dataset is made using, (i) generalized linear regression model with a logit link function (logistic regression model) , and (ii) least absolute shrinkage and selection operator (LASSO) regularization method. The performance of the prediction models developed using the training dataset is evaluated through the validation dataset.
PATIENT SELECTION AND COHORT FEATURES:
The variables for each eligible patient as a potential predictor of postoperative outcome following brain and spine surgery is extracted from the included dataset. The missing data could be imputed using multiple imputation with chained equations from large administrative databases 32.
APPROACHES TO BUILD THE PREDICTIVE MODEL:
The first approach in building the predictive model is the logistic regression. Based upon the Akaike Information Criterion 41, a forward and backward stepwise selection procedure is conducted. A natural cubic spline method is used to determine the non-linearity of the continuous variables 42. The second approach is based on the penalized regression model to obtain shrinkage estimators for the regression coefficients using LASSO method 43. Using LASSO, there is shrinkage of regression coefficients for some variables to zero since it uses a regularization method and shrinkage estimator to impose a constraint on the model parameters. Furthermore, a 10-fold cross validation is used to find a tuning parameter for each predictive model 34. An absolute value of the z-statistic for each model is used to evaluate the importance of included variable.
PERFORMANCE EVALUATION OF THE PREDICTIVE MODEL:
The discrimination of the predictive model is assessed using the receiver operator characteristic (ROC) area under the curve (AUC) on both the training and validation dataset. Furthermore, the calibration is assessed by plotting the observed incidence of each postoperative outcome against the incidence of the model-predicted probability. When the predicted effect for the model is equivalent, we expect the predictions to be closer to a 45βΈ° diagonal line. Overall model performance is further assessed using the Brier score, which is the mean squared error between the predicted probability and the observed outcome of each model. The Brier score ranges between 0 and 1. A Brier score value of 0 shows a perfect fit.
Furthermore, a simulation study to evaluate the influence of sample size on the performance of prediction models for the assessment of postoperative outcomes is also important. Therefore, a random subset of data is selected from a varying sample size and repeated the model fitting procedure for calculating the predictive ability of overall complications using the logistic regression to calculate the AUC. Furthermore, decision curve analysis is performed to determine the best model for clinical management using net benefit over a range of probability thresholds.
PREDICTING POSTOPERATIVE NEUROSURGICAL OUTCOMES:
The receiver operating characteristics of postoperative outcomes based on the training data set are shown in Figure 1, with an AUC value corresponding to >0.70. The ROC curve for predicting postoperative neurosurgical outcomes is better with logistic regression (seen in Figure 1) compared to AUC for predicting outcomes with LASSO (Figure 2, AUC <0.70). This demonstrates that the performance of the predictive model based upon LASSO is a little bit lower than that using logistic regression. Furthermore, a calibration plot showing predicted probability and observed outcome in good agreement are illustrated in Figure 3 (Brier score: ~0.00).
VALIDATION AND SENSITIVITY ANALYSIS:
The remaining dataset (30%) is used for validation of the performance of this prediction model. The AUCs of the validation set are comparable to those of the training data (AUC >0.70).
Furthermore, a simulation study is performed by increasing the sample size to determine the influence of increasing the sample size on AUCs. However, it does not change beyond a particular sample size, thus shows there is no improvement in the accuracy as the sample size is increased.
DECISION CURVE ANALYSIS:
The decision curve was further plotted as illustrated in Figure 5, which shows that our prediction model provided a higher net benefit in the management of patients compared to changing management for no patients or for all patients undergoing brain or spine surgery over all thresholds. The overall predictive probability of the model along with an estimated risk is calculated from the decision curve analysis.
Furthermore, the variable importance for the prediction model for the postoperative outcome is also plotted as shown in Figure 6.
PREDICTIVE PROBABILITY OF POSTOPERATIVE OUTCOMES:
The predictive probability is calculated from the coefficients in the logistic regression. This can be further uploaded to an open access web application for risk stratification for the physicians and the patients.
This study evaluates the utility of machine learning algorithms using logistic regression to identify a set of predictive variables for prognosticating postoperative outcomes following spine surgery. The predictive model with the highest performance across the discrimination, calibration and decision analysis is selected. Furthermore, an open access web application could be created using the coefficients in the logistic regression. The predictive probability of postoperative outcome is one of the most important aspects of clinical decision making. Although various surgical decision-making algorithms such as the NOMS criteria exist 3, the ability to predict other postoperative outcomes in this patient population still requires development and further study. Instruments such as this can serve as helpful decision-making tools when faced with the question of whether to operate. While surgeons may recognize neurologic impairment as a result of cranial and spinal surgery, the most challenging aspect of deciding upon surgery is often the determining whether the patient will survive long enough to benefit from surgery along with postoperative clinical outcomes.
There are several existing studies that have explored predicting factors of postoperative outcomes using National Surgical Quality Improvement Program (NSQIP) and claims database 29,34,35,44.. Since the purpose of this study is to create a clinical decision tool, therefore it is of utmost importance to calibrate the model. Therefore, our study assessed the calibration slope and intercept both numerically and graphically. Unlike previous studies 32, we used logistic regression for accurate prediction as it explains the independent effect of individual risk factor on the postoperative outcomes.
There are several limitations of this study, which include, (i) Large administrative datasets such as NSQIP, provide patient- and surgical characteristics that likely vary for the patients for which this predictive model will ultimately be generated. As such, the clinicians should be cautious while interpreting the predictive probabilities derived from this analysis using multi-institutional clinical registries, (ii) Sometimes the AUC could not be >0.80, this is most likely due to large numbers of missing variables from the large multi-institutional datasets. Alternative machine-learning algorithms based upon other analyses like tree-base, Bayes-point machine, neural network or deep-learning methods may enhance the predictive accuracy, (iii) The LASSO regression does not show better performance than logistic regression. This is most likely due to the fact that the penalized regression approach does better when the number of observations are relatively smaller compared to the number of features, which is different from our study 34,35.
Despite these limitations, our study achieved the primary objective of developing a machine learning algorithm for predicting postoperative outcomes following cranial and spinal surgery. The model performed well across discrimination, calibration and decision-analysis. Furthermore, we created a predictive probability and developed an open access web application for risk stratification. With further development, instruments like this could foreseeably be integrated into electronic medical record systems to provide real-time risk assessments to aid clinical decision making at the bedside.
Machine learning algorithms provide a novel computational model that predicts the postoperative outcomes following cranial and spinal surgeries. The implementation of these algorithms as an accessible tool may improve the assessment of postoperative clinical outcomes, thus can enhance the risk modelling strategies, quality assessment, patient counselling and bed management.
Competing interests: The authors declare no competing interests.