1.1 Selection of the subjects
A total of 8389 RA patients, with 9.04% prevalence of stroke (758 RA with stroke patients) were filtered from the inpatient department of rheumatology and immunology of the First Affiliated Hospital of China Medical University during January 2011 to December 2018 in this study. According to the inclusion and exclusion criteria, 313 RA with stroke patients and1827 RA patients were included into studies (shown as figure s1, see in additional file 1), all aged 18 and older, from EMR of a third-senior hospital in Liaoning province. EMRs were classified and coded by using the International Classification of Diseases Tenth Revision of the Beijing clinical version (RA: M05.x~06.x; stroke (ischemic and hemorrhagic): I60 I60.1-I60.0 I61 I61.0-I61.9 I69.0 I69.1 I63 I63.0-I63.9 I69.3). The study conformed to the principles outlined in the Declaration of Helsinki and was conducted under the guidelines of the Institutional Review Board approved by the ethics committee of the medical science research institute of the First Affiliated Hospital, China Medical University (approval number: AF-SOP-07-1.0-01). All subjects gave written informed consent for the use of their data.
This study criteria included the American College of Rheumatology 1987/201018 for RA and the CVD criteria adopted at the Fourth Academic Conference by the Chinese Neuroscience Society in 1995 19 for stroke. RA with stroke cohort, which patients should meet the follow inclusion criteria: Ⅰ) the patients conform to the above stroke and RA diagnostic criteria; Ⅱ) Based on the first record time of RA and stroke in the EMRs, if the record of stroke was later than that of RA, we believed that the patient has developed stroke after being diagnosed with RA and included in the RA with stroke cohort. Ⅲ) the patients have been detected at least one time laboratory assessments when they were in hospital at the first time (i.e, serum inflammatory-, antibody-, complement-, lipid-assays); Ⅳ) over 18 years old. RA cohort (without stroke), which patients should met the follow inclusion criteria: Ⅰ) conform to the above RA diagnostic criteria; Ⅱ) over 18 years old. RA with stroke cohort and RA cohort were excluded if they met the following criteria: Ⅰ) patients who still suffered from other connective tissue diseases, including systemic lupus erythematosus, scleroderma, dry syndrome and vasculitis; Ⅱ) RA patients with coexisting ankylosing spondylitis and gout arthritis. Finally, we selected 70% of the RA with stroke and RA patients as the develop cohort randomly, with the rest comprising the validation cohort20.
1.2 Data collection
All of data were screened from EMR, mainly including personal information, such as age, gender, height and weight, metabolic indices including serum TC, TG, LDL, HDL and fasting blood-glucose (FBG), and serologic profiles including CRP, ESR, rheumatoid factor (RF), complement3 (C3), complement4 (C4) and anticyclic citrullinated peptide (anti-CCP) antibodies, and coronary heart disease (CHD), atrial fibrillation (AF), left ventricular hypertrophy (LVH), cardiovascular disease (CVD) history records. The medication history in record was also included, that is, hypotensive medicine (hy-med), biologic disease modifying anti-rheumatic drugs (Bio-med). All laboratory tests were carried out using overnight fasting venous blood samples and conducted with clinical standard operating procedures for inspection items. In addition, when the results of multiple laboratory tests at different time points were assessed during the initial data filtering, the first laboratory test results were selected at first admission due to stroke among RA patients (RA with stroke cohort) and selected at first admission of RA patients without stroke (RA cohort).
1.3 Statistical analysis.
All reported statistical significance levels were set at 0.05 with two-sided. The categorical data were expressed as percentages by cohort, and some continuous predictors (ie., age, SBP and CRP) were categorized after assessed using consensus approaches or guidelines and previously published articles. In addition, the absence of some features in clinical medical records was inevitable (which accounted for less 20% in this study), and we used multiple imputation, based on 5 replications and a chained equation approach method (predictive mean matching, PMM for quantitative data, and linear regression for categorical data), to account for missing data in SPSS 23.0 software. Adopting Chi-Square test or Fisher exact test to compare the difference in participant characteristics between develop cohort and validation cohort. The univariable association between RA with stroke group and RA group with serum biochemical marker levels was assessed in the develop cohort using univariate LR, based on variables associated with stroke which were assessed by clinical importance, scientific knowledge, and predictors identified in previously published articles to develop the risk model of RA patients with stroke21, then validated in the validation cohort.
1.3.1. Developing model between the RA with stroke group and RA group in the develop cohort
The model was built by binary logistic regression (LR) as a simple model with unadjusted, and a complex model adjusted by sex and age, considering the disparity between male and female morbidity in RA patients and the obvious aging of stroke patients. Further, to provide the clinician with a quantitative tool to predict the individual probability of stroke, we built the nomogram for the prediction of stroke in RA patients based on multivariable LR analysis in the develop cohort. All the analysis was conducted with R software version 3.6.2 (packages mainly include rms, Hmisc, dca, PredictABEL. R packages. http: // www. Rproject. org) .
1.3.2. Comparing several machine learning models between the RA with stroke group and RA group in the develop and validation cohorts
Machine learning models are conducted based on scikit-learn which is an open source machine learning library, using Bayesian optimization method to implement algorithm optimization, and cross-validation method, N-folds=5, to complete algorithm evaluation during the optimization process. We used 6 kinds of machine algorithms running three 30-minute sessions, including LR, Support Vector Machine (SVM), Random Forest (RF), xgboost (XGB), gradient boosting decision tree (GBDT), k-Nearest Neighbors (KNN), to compare algorithms and evaluate the simple model and complex model in develop and validation cohorts respectively, which were compared by evaluation metrics as follows: accuracy, precision, recall, f1-score, balance error (ber).
1.3.3. Comparing the performance of developed models with Framingham Risk model and validation of which in the validation cohort
The performance of the nomogram of the model was assessed by discrimination and calibration. Calibration curves, the accuracy of point estimates of the LR function, accompanied with the Hosmer-Lemeshow test to assess if the model calibrated perfectly or not. The discrimination of the nomograms was evaluated using the Harrell’s concordance index (C-index), the predictive accuracy for individual outcomes (discriminating ability), is equivalent to area under the curve (AUC), and compared among the Framingham Risk Score in predicting stroke and our prediction models. In addition, the calibration was calculated via a bootstrap method with 1000 resamples in the develop cohort. Internal validation was performed using the validation cohort. The LR formula formed in the develop cohort was applied to all patients of the validation cohort. The net reclassification index (NRI) indicates the proportion of patients correctly reclassified by a new model compared with an existing or standard model, while the integrated discrimination improvement (IDI) indicates the change in difference in average predicted probabilities between those who combined with stroke and those who did not in a new and existing model22. Furtherly, NRI and IDI between the Framingham Risk Score in predicting stroke and our prediction models were assessed based on low risk (0~20%), medium risk (20%~59%), high risk (60%~100%).
1.3.4. Clinical Use
In the develop cohort, DCA was conducted to evaluate the clinical usefulness of the nomogram by quantifying the net benefits at different threshold probabilities and was used to identify the predictive models with the best discriminative abilities23. In addition, net benefit was defined as the proportion of true positives minus the proportion of false positives, weighted by the relative harm of false-positive and false-negative results 24. Simple and complex models were used to predict risk stratification of 1000 people with a bootstrap resample by the clinical impact curve.