Machine Learning Algorithms Using Logistic Regression for Predicting Neurosurgical Outcomes

DOI: https://doi.org/10.21203/rs.3.rs-37934/v1

Abstract

Background: Preoperative prognostication of clinical and surgical outcome in patients with neurosurgical diseases can improve the risk stratification, thus can guide in implementing targeted treatment to minimize these events. Therefore, the author aims to highlight the development and validation of predictive models determining neurosurgical outcomes through machine learning algorithms using logistic regression.

Methods: Logistic regression (enter, backward and forward) and least absolute shrinkage and selection operator (LASSO) method for selection of variables from selected database can eventually lead to multiple candidate models. The final model with a set of predictive variables must be selected based upon the clinical knowledge and numerical results.

Results: The predictive model which performed best on the discrimination, calibration, Brier score and decision curve analysis must be selected to develop machine learning algorithms. Logistic regression should be compared with the LASSO model. Usually for the big databases, the predictive model selected through logistic regression gives higher Area Under the Curve (AUC) than those with LASSO model. The predictive probability derived from the best model could be uploaded to an open access web application which is easily deployed by the patients and surgeons to make a risk assessment world-wide.

Conclusions: Machine learning algorithms provide promising results for the prediction of outcomes following cranial and spinal surgery. These algorithms can provide useful factors for patient-counselling, assessing peri-operative risk factors, and predicting post-operative outcomes after neurosurgery.

Introduction

The treatment of neurosurgical diseases have been revolutionized due to greater stride observed in terms of systematic therapy, radiotherapy, and surgical techniques 1–3,4,5,6,78,9.  The multimodal management has resulted in achieving better clinical and functional outcomes 10–125,13,14. However, surgery for neurosurgical diseases such as spinal metastases remains the main stay of treatment for achieving durable local tumor control and spinal stabilization over other radiation and interventional techniques 15. Despite the technical expertise and surgical care, the incidence of complications in the postoperative period is not uncommon 10. As an example, the complication rates following surgery for metastatic spine disease ranges from 10-52% 16–19, 3-10% for lumbar degenerative spondylolisthesis 20, 4-11.1% for spinal osteotomies 21,22, and upto 70% for adult spinal deformity 23,24. These adverse events (AEs) include both medical and surgical issues, thus impairs the quality of life 25. Understanding of the factors that influence the postoperative outcomes is of utmost importance not only for surgical decision making, but for patient counseling as well since the benefits of improving ambulation and pain need to be weighed against the risks of postoperative AEs following surgery 10.

Furthermore, with an exponentially increasing costs associated with spinal surgery; there are several cost containment and quality improvement measures 26. The most common among all these is the introduction of the concept of  “capitalized bundle care” 27, which combines the hospital facility and professional charges for necessary care for a defined longitudinal period of care. Thus, a small percentage of unanticipated event can consume a large percentage of a service line’s capitated revenue 28. To achieve financial sustainability, it is of utmost importance to stratify the risks of unplanned events 27,28. This risk stratification can help in addressing the modifiable factors in the high-risk patients, which is ultimately important to negotiate for complex bundled payments.

The understanding of postoperative outcomes laid a foundation in numerous quality-reporting metrics. This is because of the fact that the perioperative events negatively impact the clinical and patient reported outcomes 29. There exist need to develop well-defined prognostic algorithms to stratify the risks in the patients undergoing surgery for brain and spinal diseases. Developing an appropriate criterion in an interdisciplinary setting will eventually lead to avoid complications and reduce the financial burden associated with it. Although studies have applied machine learning algorithm to predict postoperative outcomes using machine learning algorithms 30–33. However, to date, a few studies focused on using logistic regression for developing and validating the machine learning algorithm 34,35. Therefore, the purpose of this study was to understand, (i) how to develop a set of predictive factors for postoperative outcomes following spine surgery using machine learning algorithms and (ii) to create an open accessible user-interface to prognosticate postoperative outcomes which is easily accessible to individuals all over the world.

Methods

STUDY DESIGN AND DATA SOURCES:

Databases, preferably big data analytics like multi-institutional clinical registries 36,37,38 have been used for developing the predictive algorithms. The de-identified data from large national database usually get exemption from review by institutional review board.

GUIDELINES:

The Transparent Reporting of Multivariable Prediction Models for Individual Prognosis or Diagnosis (TRIPOD) and JMIR Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research 39,40 are followed. 

SPLIT-SAMPLE APPROACH: TRAINING VERSUS VALIDATION SET:

Based on the split proportions in the previous literature 41, the data is randomly divided into a training dataset (70%) and a validation dataset (30%). A set of predictive models for various types of postoperative outcomes like AEs, extended length of hospital stay etc. for the training dataset is made using, (i) generalized linear regression model with a logit link function (logistic regression model) , and (ii) least absolute shrinkage and selection operator (LASSO) regularization method. The performance of the prediction models developed using the training dataset is evaluated through the validation dataset.

PATIENT SELECTION AND COHORT FEATURES:

The variables for each eligible patient as a potential predictor of postoperative outcome following brain and spine surgery is extracted from the included dataset. The missing data could be imputed using multiple imputation with chained equations from large administrative databases 32.

APPROACHES TO BUILD THE PREDICTIVE MODEL:

The first approach in building the predictive model is the logistic regression. Based upon the Akaike Information Criterion 41, a forward and backward stepwise selection procedure is conducted. A natural cubic spline method is used to determine the non-linearity of the continuous variables 42. The second approach is based on the penalized regression model to obtain shrinkage estimators for the regression coefficients using LASSO method 43. Using LASSO, there is shrinkage of regression coefficients for some variables to zero since it uses a regularization method and shrinkage estimator to impose a constraint on the model parameters. Furthermore, a 10-fold cross validation is used to find a tuning parameter for each predictive model 34. An absolute value of the z-statistic for each model is used to evaluate the importance of included variable.

PERFORMANCE EVALUATION OF THE PREDICTIVE MODEL:

The discrimination of the predictive model is assessed using the receiver operator characteristic (ROC) area under the curve (AUC) on both the training and validation dataset. Furthermore, the calibration is assessed by plotting the observed incidence of each postoperative outcome against the incidence of the model-predicted probability. When the predicted effect for the model is equivalent, we expect the predictions to be closer to a 45βΈ° diagonal line. Overall model performance is further assessed using the Brier score, which is the mean squared error between the predicted probability and the observed outcome of each model. The Brier score ranges between 0 and 1. A Brier score value of 0 shows a perfect fit.

Furthermore, a simulation study to evaluate the influence of sample size on the performance of prediction models for the assessment of postoperative outcomes is also important. Therefore, a random subset of data is selected from a varying sample size and repeated the model fitting procedure for calculating the predictive ability of overall complications using the logistic regression to calculate the AUC. Furthermore, decision curve analysis is performed to determine the best model for clinical management using net benefit over a range of probability thresholds.

Results

PREDICTING POSTOPERATIVE NEUROSURGICAL OUTCOMES:

The receiver operating characteristics of postoperative outcomes based on the training data set are shown in Figure 1, with an AUC value corresponding to >0.70. The ROC curve for predicting postoperative neurosurgical outcomes is better with logistic regression (seen in Figure 1) compared to AUC for predicting outcomes with LASSO (Figure 2, AUC <0.70). This demonstrates that the performance of the predictive model based upon LASSO is a little bit lower than that using logistic regression. Furthermore, a calibration plot showing predicted probability and observed outcome in good agreement are illustrated in Figure 3 (Brier score: ~0.00).

VALIDATION AND SENSITIVITY ANALYSIS:

The remaining dataset (30%) is used for validation of the performance of this prediction model. The AUCs of the validation set are comparable to those of the training data (AUC >0.70).

Furthermore, a simulation study is performed by increasing the sample size to determine the influence of increasing the sample size on AUCs. However, it does not change beyond a particular sample size, thus shows there is no improvement in the accuracy as the sample size is increased.

DECISION CURVE ANALYSIS:

The decision curve was further plotted as illustrated in Figure 5, which shows that our prediction model provided a higher net benefit in the management of patients compared to changing management for no patients or for all patients undergoing brain or spine surgery over all thresholds. The overall predictive probability of the model along with an estimated risk is calculated from the decision curve analysis.

Furthermore, the variable importance for the prediction model for the postoperative outcome is also plotted as shown in Figure 6.

PREDICTIVE PROBABILITY OF POSTOPERATIVE OUTCOMES:

The predictive probability is calculated from the coefficients in the logistic regression. This can be further uploaded to an open access web application for risk stratification for the physicians and the patients.

Discussion

This study evaluates the utility of machine learning algorithms using logistic regression to identify a set of predictive variables for prognosticating postoperative outcomes following spine surgery. The predictive model with the highest performance across the discrimination, calibration and decision analysis is selected. Furthermore, an open access web application could be created using the coefficients in the logistic regression. The predictive probability of postoperative outcome is one of the most important aspects of clinical decision making. Although various surgical decision-making algorithms such as the NOMS criteria exist 3, the ability to predict other postoperative outcomes in this patient population still requires development and further study. Instruments such as this can serve as helpful decision-making tools when faced with the question of whether to operate. While surgeons may recognize neurologic impairment as a result of cranial and spinal surgery, the most challenging aspect of deciding upon surgery is often the determining whether the patient will survive long enough to benefit from surgery along with postoperative clinical outcomes.

There are several existing studies that have explored predicting factors of postoperative outcomes using National Surgical Quality Improvement Program (NSQIP) and claims database 29,34,35,44.. Since the purpose of this study is to create a clinical decision tool, therefore it is of utmost importance to calibrate the model. Therefore, our study assessed the calibration slope and intercept both numerically and graphically. Unlike previous studies 32, we used logistic regression for accurate prediction as it explains the independent effect of individual risk factor on the postoperative outcomes.

There are several limitations of this study, which include, (i) Large administrative datasets such as NSQIP, provide patient- and surgical characteristics that likely vary for the patients for which this predictive model will ultimately be generated. As such, the clinicians should be cautious while interpreting the predictive probabilities derived from this analysis using multi-institutional clinical registries, (ii) Sometimes the AUC could not be >0.80, this is most likely due to large numbers of missing variables from the large multi-institutional datasets. Alternative machine-learning algorithms based upon other analyses like tree-base, Bayes-point machine, neural network or deep-learning methods may enhance the predictive accuracy, (iii) The LASSO regression does not show better performance than logistic regression. This is most likely due to the fact that the penalized regression approach does better when the number of observations are relatively smaller compared to the number of features, which is different from our study 34,35.

Despite these limitations, our study achieved the primary objective of developing a machine learning algorithm for predicting postoperative outcomes following cranial and spinal surgery. The model performed well across discrimination, calibration and decision-analysis. Furthermore, we created a predictive probability and developed an open access web application for risk stratification. With further development, instruments like this could foreseeably be integrated into electronic medical record systems to provide real-time risk assessments to aid clinical decision making at the bedside.

Conclusions

Machine learning algorithms provide a novel computational model that predicts the postoperative outcomes following cranial and spinal surgeries. The implementation of these algorithms as an accessible tool may improve the assessment of postoperative clinical outcomes, thus can enhance the risk modelling strategies, quality assessment, patient counselling and bed management.

Declarations

Competing interests: The authors declare no competing interests.

References

  1. Laufer I, Bilsky MH. Advances in the treatment of metastatic spine tumors: The future is not what it used to be. J Neurosurg Spine. 2019. doi:10.3171/2018.11.SPINE18709
  2. Barzilai O, Boriani S, Fisher CG, et al. Essential Concepts for the Management of Metastatic Spine Disease: What the Surgeon Should Know and Practice. Glob Spine J. 2019. doi:10.1177/2192568219830323
  3. Laufer I, Rubin DG, Lis E, et al. The NOMS Framework: Approach to the Treatment of Spinal Metastatic Tumors. Oncologist. 2013. doi:10.1634/theoncologist.2012-0293
  4. Wang YXJ, Káplár Z, Deng M, Leung JCS. Lumbar degenerative spondylolisthesis epidemiology: A systematic review with a focus on gender-specific and age-specific prevalence. J Orthop Transl. 2017. doi:10.1016/j.jot.2016.11.001
  5. Pugely AJ, Martin CT, Gao Y, Mendoza-Lattes S. Causes and risk factors for 30-day unplanned readmissions after lumbar Spine surgery. Spine (Phila Pa 1976). 2014. doi:10.1097/BRS.0000000000000270
  6. Howe CR, Agel J, Lee MJ, et al. The morbidity and mortality of fusions from the thoracic spine to the pelvis in the adult population. Spine (Phila Pa 1976). 2011. doi:10.1097/BRS.0b013e3181f453e2
  7. Auerbach JD, Lenke LG, Bridwell KH, et al. Major complications and comparison between 3-column osteotomy techniques in 105 consecutive spinal deformity procedures. Spine (Phila Pa 1976). 2012.
  8. Fatima N, Meola A, Pollom E, Chang SD SS. Stereotactic Radiosurgery for Large Benign Intracranial Tumors. World Neurosurg. 2019;(Epub ahead of print):S1878-8750(19)32620-8. doi:10.1016/j.wneu.2019.10.005
  9. Fatima N, Meola A, Pollom EL, Soltys SG, Chang SD. Stereotactic radiosurgery versus stereotactic radiotherapy in the management of intracranial meningiomas: a systematic review and meta-analysis. Neurosurg Focus. 2019;46(6). doi:10.3171/2019.3.FOCUS1970
  10. Choi D, Fox Z, Albert T, et al. Prediction of quality of life and survival after surgery for symptomatic spinal metastases: A multicenter cohort study to determine suitability for surgical treatment. Neurosurgery. 2015. doi:10.1227/NEU.0000000000000907
  11. Lieberson RE, Veeravagu A, Eckermann JM, et al. Intramedullary spinal cord metastasis from prostate carcinoma: A case report. J Med Case Rep. 2012. doi:10.1186/1752-1947-6-139
  12. Patchell RA, Tibbs PA, Regine WF, et al. Direct decompressive surgical resection in the treatment of spinal cord compression caused by metastatic cancer: A randomised trial. Lancet. 2005. doi:10.1016/S0140-6736(05)66954-1
  13. Gupta R, Moore JM, Amorin A, et al. Long-term follow up data on difficult to treat intracranial arteriovenous malformations treated with the CyberKnife. J Clin Neurosci. 2019. doi:10.1016/j.jocn.2018.10.109
  14. Rolston JD, Han SJ, Lau CY, Berger MS, Parsa AT. Frequency and predictors of complications in neurological surgery: National trends from 2006 to 2011: Clinical article. J Neurosurg. 2014. doi:10.3171/2013.10.JNS122419
  15. Patchell RA, Tibbs PA, Regine WF, et al. Direct decompressive surgical resection in the treatment of spinal cord compression caused by metastatic cancer: A randomised trial. Lancet. 2005. doi:10.1016/S0140-6736(05)66954-1
  16. Walsh GL, Gokaslan ZL, McCutcheon IE, et al. Anterior approaches to the thoracic spine in patients with cancer: Indications and results. Ann Thorac Surg. 1997. doi:10.1016/S0003-4975(97)01034-5
  17. Gokaslan ZL, York JE, Walsh GL, et al. Transthoracic vertebrectomy for metastatic spinal tumors. J Neurosurg. 1998. doi:10.3171/jns.1998.89.4.0599
  18. Cooper PR, Errico TJ, Martin R, Crawford B, Dibartolo T. A systematic approach to spinal reconstruction after anterior decompression for neoplastic disease of the thoracic and lumbar spine. Neurosurgery. 1993. doi:10.1227/00006123-199301000-00001
  19. DeWald RL, Bridwell KH, Prodromas C, Rodts MF. Reconstructive spinal surgery as palliation for metastatic malignancies of the spine. Spine (Phila Pa 1976). 1985. doi:10.1097/00007632-198501000-00004
  20. Steiger F, Becker HJ, Standaert CJ, et al. Surgery in lumbar degenerative spondylolisthesis: Indications, outcomes and complications. A systematic review. Eur Spine J. 2014. doi:10.1007/s00586-013-3144-3
  21. Daubs MD, Lenke LG, Cheh G, Stobbs G, Bridwell KH. Adult spinal deformity surgery: Complications and outcomes in patients over age 60. Spine (Phila Pa 1976). 2007. doi:10.1097/BRS.0b013e31814cf24a
  22. Bridwell KH, Lewis SJ, Edwards C, et al. Complications and outcomes of pedicle subtraction osteotomies for fixed sagittal imbalance. In: Spine. ; 2003. doi:10.1097/01.BRS.0000090891.60232.70
  23. Smith JS, Shaffrey CI, Glassman SD, et al. Risk-benefit assessment of surgery for adult scoliosis: An analysis based on patient age. Spine (Phila Pa 1976). 2011. doi:10.1097/BRS.0b013e3181e21783
  24. Scheer JK, Smith JS, Schwab F, et al. Development of a preoperative predictive model for major complications following adult spinal deformity surgery. J Neurosurg Spine. 2017. doi:10.3171/2016.10.SPINE16197
  25. Choi D, Bilsky M, Fehlings M, Fisher C, Gokaslan Z. Spine oncology - Metastatic spine tumors. Clin Neurosurg. 2017. doi:10.1093/neuros/nyw084
  26. Lee MJ, Cizik AM, Hamilton D, Chapman JR. Predicting medical complications after spine surgery: A validated model using a prospective surgical registry. Spine J. 2014. doi:10.1016/j.spinee.2013.10.043
  27. McGirt MJ, Parker SL, Chotai S, et al. Predictors of extended length of stay, discharge to inpatient rehab, and hospital readmission following elective lumbar spine surgery: Introduction of the Carolina-Semmes Grading Scale. J Neurosurg Spine. 2017. doi:10.3171/2016.12.SPINE16928
  28. McGirt MJ, Sivaganesan A, Asher AL, Devin CJ. Prediction model for outcome after low-back surgery: Individualized likelihood of complication, hospital readmission, return to work, and 12-month improvement in functional disability. Neurosurg Focus. 2015. doi:10.3171/2015.8.FOCUS15338
  29. Veeravagu A, Li A, Swinney C, et al. Predicting complication risk in spine surgery: A prospective analysis of a novel risk assessment tool. J Neurosurg Spine. 2017. doi:10.3171/2016.12.SPINE16969
  30. Emblem KE, Nedregaard B, Hald JK, Nome T, Due-Tonnessen P, Bjornerud A. Automatic glioma characterization from dynamic susceptibility contrast imaging: Brain tumor segmentation using knowledge-based fuzzy clustering. J Magn Reson Imaging. 2009. doi:10.1002/jmri.21815
  31. Obermeyer Z, Emanuel EJ. Predicting the future-big data, machine learning, and clinical medicine. N Engl J Med. 2016. doi:10.1056/NEJMp1606181
  32. Karhade A V., Thio QCBS, Ogink PT, et al. Development of Machine Learning Algorithms for Prediction of 30-Day Mortality After Surgery for Spinal Metastasis. Neurosurgery. 2019. doi:10.1093/neuros/nyy469
  33. Karhade A V., Ogink P, Thio Q, et al. Development of machine learning algorithms for prediction of discharge disposition after elective inpatient surgery for lumbar degenerative disc disorders. Neurosurg Focus. 2018. doi:10.3171/2018.8.FOCUS18340
  34. Han SS, Azad TD, Suarez PA, Ratliff JK. A machine learning approach for predictive models of adverse events following spine surgery. Spine J. 2019. doi:10.1016/j.spinee.2019.06.018
  35. Ratliff JK, Balise R, Veeravagu A, et al. Predicting occurrence of spine surgery complications using big data modeling of an administrative claims database. J Bone Jt Surg - Am Vol. 2016. doi:10.2106/JBJS.15.00301
  36. DePasse JM, Durand W, Daniels AH. Predictors of Nonneurologic Complications and Increased Length of Stay After Cervical Spine Osteotomy. World Neurosurg. 2018;118:e727-e730. doi:10.1016/j.wneu.2018.07.029
  37. Karhade A V., Larsen AMG, Cote DJ, Dubois HM, Smith TR. National databases for neurosurgical outcomes research: Options, strengths, and limitations. Clin Neurosurg. 2018. doi:10.1093/neuros/nyx408
  38. Hall MJ, DeFrances CJ, Williams SN, Golosinskiy A, Schwartzman A. National Hospital Discharge Survey: 2007 summary. Natl Health Stat Report. 2010.
  39. Luo W, Phung D, Tran T, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: A multidisciplinary view. J Med Internet Res. 2016. doi:10.2196/jmir.5870
  40. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. Eur Urol. 2015. doi:10.1016/j.eururo.2014.11.025
  41. Bertrand P V., Sakamoto Y, Ishiguro M, Kitagawa G. Akaike Information Criterion Statistics. J R Stat Soc Ser A (Statistics Soc. 1988. doi:10.2307/2983028
  42. Chiang AY. Generalized Additive Models: An Introduction With R. Technometrics. 2007. doi:10.1198/tech.2007.s505
  43. Gauraha N. Introduction to the LASSO. Resonance. 2018. doi:10.1007/s12045-018-0635-x
  44. Campbell PG, Yadla S, Nasser R, Malone J, Maltenfort MG, Ratliff JK. Patient comorbidity score predicting the incidence of perioperative complications: Assessing the impact of comorbidities on complications in spine surgery: Clinical article. J Neurosurg Spine. 2012. doi:10.3171/2011.9.SPINE11283