Using The Random Forest Algorithm To Detect The Activity of Thyroid-Associated Ophthalmopathy

Objective: The aim of this study is to establish a random forest model to detect active and quiescent phases of patients with Thyroid-associated ophthalmopathy (TAO) and to evaluate its diagnostic performance. Methods(cid:0)A total of 146 patients (292 eyes) who were diagnosed with TAO and were treated in the Ophthalmology Outpatient Clinic of Beijing TongRen hospital were retrospectively included in the study. We took the clinical activity score of TAO as the target; took gender, age, smoking status, I-131 treatment history, thyroid nodules, thyromegaly, thyroid hormone and TSH-receptor antibodies (TRAb) as predictive characteristic variables to establish a random forest model. The proportion of the training group to the testing group was 7:3. We analyzed the model’s accuracy, precision, sensitivity, speci�city, positive predictive value (PPV), negative predictive value (PPV), F1 score and out-of-bag (OOB) error, with the accuracy, the brier loss and the area under the receiver operating characteristic curve compared with logistic regression model. Results(cid:0)Our


Introduction
Thyroid-associated ophthalmopathy (TAO) is an autoimmune disorder affecting the orbital and periorbital tissues, which can cause dysthyroid optic neuropathy, exposure keratitis, even corneal ulcer in severe cases.Therefore, timely interventions and treatments are particular crucial to the prognosis of patients with TAO.The natural course of TAO is presented as a rundle curve, which can be divided into two stages: the active phase and the quiescent phase [1,2].Treatments during these two phases are very different [3].As a result, detecting the clinical activity phase of patients with TAO is critical to the choice of treatment [4,5].
Nowadays, the Clinical Activity Score (CAS) is the most common method to evaluate the phase of the disease [6].However, such evaluation standards require professional ophthalmologists to conduct a comprehensive examination of the patient's eye manifestation, which may cause the delay in monitoring the patient's progression.Also, due to the subjectivity of CAS, the score evaluated by different doctors are not exactly the same.Therefore, the CAS has limitations, and it is necessary to nd a more objective method to predict the activity of patients.
The random forest algorithm is superior to traditional statistical methods.It is based on a large number of decision tree classi ers, which helps to reduce deviations, tolerant outliers, and avoid over tting [7].
Although this algorithm has been widely applied in many elds, no one has used it to study the activity of TAO yet.
It has been well-documented that the patient's gender, age, smoking status, I-131 treatment history, thyroid function status, and thyroid hormone receptor antibody (TRAb) are high-risk factors for TAO which are closely related to the development of the disease [8,9].In this study, we integrate the above indicators and plan to use the random forest algorithm to detect the activity of TAO patients.The purpose of the study is to investigate whether we can predict the activity of patients with TAO through the random forest algorithm, a machine learning method without professional eye examinations.

Patients:
The study was approved by the Ethics Committee of Beijing Tongren Hospital, Capital Medical University and carried out strictly in accordance with the ethical guidelines of the declaration of Helsinki (Approval No.TRECKY2018-056).From April 2019 to January 2021, 146 patients (292 eyes in total) who visited Ophthalmology Outpatient Clinic of Beijing TongRen hospital were examined.The inclusion criteria were as follows: (1) age more than 18 years, (2) meeting the internationally recognized diagnostic criteria for TAO, and (3) integral clinical demographic data.The enrolled patients were registered with the European Group on Graves' orbitopathy (EUGOGO) rst diagnosis form to collect the patient's gender, age, smoking status, I-131 treatment history, thyroid nodules, thyromegaly, serum thyroid hormone, TRAb, ocular symptoms and signs.The exclusion criteria were as follows: (1) underage patients; (2) eye in ammation caused by other eye diseases; (3) patients with other autoimmune diseases.

Clinical thyroid associated ophthalmopathy evaluation
Based on the EUGOGO guidelines, patients were divided into two groups according to their clinical activity of TAO [10].Professional ophthalmologists from Beijing Tongren Hospital conducted eye examinations on the patients.Their clinical activity was evaluated based on the clinical activity score (CAS), which consists of 7 orbital signs and symptom variables (including spontaneous retrobulbar pain, pain on attempted upward or downward gaze, swelling of the eyelids, redness of the eyelids, redness of conjunctiva, chemosis, swelling of caruncle or plica), each variable is worth 1 point [10].The sum of the scores of the 7 variables can re ect the degree of disease activity, and CAS ≥ 3 points is used as a sign of activity phase [11].(Fig. 1)

Statistical analysis
The t-test was used for continuous variables, and Fisher's exact test for categorical variables.The IBM SPSS Statistics software (Version 20.0.IBM Corporation, NY, USA) was used for analyses.

Random forest modeling
We used the sklearn package in python3.9 to build a random forest model, including the patient's gender, age, I-131 treating history, smoking status, thyromegaly, thyroid nodules, FT3, FT4, TT3, TT4, TSH, TRAb, and clinical activity, altogether 13 characteristic variables.All variables were uniformly quanti ed and coded (such as table 1).We took the CAS of TAO as the target, and other 12 factors as predictive characteristic variables.We randomly selected 30% of the sample from the total data as the test set, and the remaining 70% as the training set.
In order to address the class-imbalanced dataset, we applied the Synthetic Minority Oversampling Technique (SMOTE) to oversimple the minority class.SMOTE considered a sample from the minority class and its k-nearest neighbors, taking vectors between the sample and neighbors in the feature space.New records were then generated by picking points randomly on these vectors.
In this model, the data was randomly sampled from the training set repeatedly and replaced by bootstrapping/bagging.A subset was generated for each decision tree, and then the decision tree model was trained using this subset [7,12].This method can reduce the variance of random forest model, that is, reduce the impact of individual abnormal samples on the overall prediction performance of the model.Speci cally, in order to minimize the classi cation error of the model, the random forest model randomly selects a subset of feature variables (the number is mtry).We took the recommended value of mtry=3, which is the square root of the total number of features [13].
We chose out-of-bag (OOB) error that is generally used to estimate the generalization error of the model and use the number of decision trees (ntree) in the random forest model as a variable to repeatedly train the model and calculate the OOB error until it becomes stable [14].We chose the model with the lowest OOB error and took the corresponding ntree as the optimal value.The input variables in the optimized model were ranked by relative importance in predicting clinical activity based on the mean decrease accuracy (MDA) and mean decrease gini (MDG) [13].
On the basis of this model, we calculated the model's accuracy, sensitivity, speci city, positive predictive value, negative predictive value, and F1 score of the test set.F1 score is an index to measure the accuracy of binary classi cation model, which conveys the balance between precision and sensitivity.Finally, we integrated the variables into the logistics regression model, comparing the accuracy, the area under the receiver operating characteristic curve and the brier loss of two models [15].

Results
Baseline characteristic 146 patients with 292 eyes were analyzed in this study, 116 (39.7%) of whom were diagnosed with active phase of TAO (Table 1 Combined with the patient's gender, age, I-131 treating history, smoking status, thyromegaly, thyroid nodules, FT3, FT4, TT3, TT4, TSH, TRAb, we successfully constructed a random forest model and tested it to predict the clinical activity of TAO.When mtry= 3, the out-of-bag error decreases as ntree increases (Fig. 1) and remains stable until about ntree=350.Our nal random forest model is built with 350 trees (ntree=350), and its out-of-bag error is as low as 0.12 (Fig. 2).
Based on the MDA and MDG of the predictive characteristic variables in the model, we ranked the relative importance of the variables respectively (Fig. 3).It can be seen that the TRAb level is the most important feature for the model, and its importance is signi cantly higher than other features.Five thyroid hormones all rank high and show similar feature importance, no matter the model was based on the MDA and MDG.The accuracy of this random forest model to predict the clinical activity of TAO is 0.93, the sensitivity is 0.88, the speci city is 0.96, the PPV is 0.94, the NPV is 0.93, F1 Score of 0.91, and the OOB error is 0.12 (Table 2).The accuracy of the random forest model is higher than that of the logistic regression model, which are 0.93 and 0.79, respectively; the brier losses are 0.06 and 0.20, respectively; the AUC are 0.95 and 0.86, respectively (Fig. 4).
Table2 Prediction of Clinical activity based on a random forest model.

Discussion
Although CAS is gradually being adopted by ophthalmologists around the world, it is often the endocrinologist or the primary care physician who rst evaluates patients with TAO.Thus, accurate CAS evaluation can be time-consuming and laborious.In this study, we constructed a random forest model containing 146 patients with a total of 292 eyes.Based on the patient's gender, age, smoking, I-131 treating history, thyroid hormones, TRAb, thyromegaly and thyroid nodules, we predicted the clinical activity of patients with TAO.
The most important clinical signi cance of the CAS score is to guide the treatment of diseases.We analyzed the CAS score as a continuous variable instead of binarizing, because the binary classi cation is su cient for our purpose.For patients in the active phase, immunomodulatory therapy can suppress in ammation and eye diseases caused by autoimmune reactions.Once the in ammation is static, immunomodulatory treatments is no longer effective, and we should consider reconstructive surgery to improve the appearance of the eyes and eliminate symptoms.
Unlike classic statistical models, random forest models can provide unbiased importance rankings of input variables.In our model, TRAb is the most important predictor of clinical activity which is consistent with the opinion of previous study that serum TRAb is a laboratory test marker for judging the clinical activity of TAO [4].The recommendation of the American Association of Clinical Endocrinologists and EUGOGO agrees that TRAb is the central pathogenic event of TAO, and can be used to distinguish patients with active and quiescent phase [16][17][18][19][20].
In our model we also found that the thyroid hormone ranks high both based on the mean decrease in accuracy and mean decrease in the Gini coe cient, representing the importance of maintaining normal thyroid function among all factors except for TRAb.It has been reported that the thyroid hormone, like autoimmune in ammation, is a signi cant contributor to oxidative stress in patients with TAO, which manifests as upregulated lipid peroxidation and antioxidant system activation [21,22].It is found that euthyroid state restoration leads to a relative reduction in activity and levels of most antioxidant parameters [23,24].Thus, the autoimmune in ammation of the orbital tissue and thyroid hormone status seem to be mutually independent modi ers of TAO activity which may explain the fact that CAS score is related to hormone levels.
Our random forest model predicts the clinical activity of TAO with an accuracy of 0.93, a sensitivity of 0.88, a speci city of 0.96, a PPV of 0.94, a NPV of 0.93, an F1 Score of 0.91, and an OOB error as low as 0.12, which is better than the traditional multi-logistic regression and shows good prediction ability.These ndings indicate that the random forest model can provide us with a satisfactory alternative of CAS to evaluate the clinical activity of TAO.In particular, its high speci city may make our method useful for detecting.
In this study, we chose the random forest instead of other algorithms, since the random forest algorithm does not require strict assumptions of the data, and it is less sensitive to outliers than classic methods (such as decision trees).Random forest models can handle a large number of input variables that are common in clinical situations, and they are not prone to over tting bias [25,26].In the training process, not only random samplings, but also different features can be randomly selected by bootstrapping/bagging methods to generate training data sets, which helps to reduce the correlation between decision trees in the nal random forest model.
Besides, we used out-of-bag (OOB) error to estimate the generalization error of the model [15].OOB error provides an unbiased estimation of error and predictive power, which means that random forest algorithms do not require cross-validation, and this has proven to be unbiased in many tests [7].It is also observed that our sample size is not very large and the number of quiescent phase and active phase of the target in the data is quite different (class-imbalanced).If you train directly, it may cause the trained model to be biased.A common solution to this problem is SMOTE.After SMOTE, the number of records of two types of data will be the same, so that the accuracy of the trained model will be more convincing.
These results indicate that for patients with TAO who cannot meet professional ophthalmologists in time, we can use the random forest model to distinguish patients who may need treatments from patients in quiescent phase, arranging ophthalmology consultation for them.In this way, we could successfully alleviate the overburden of the health system in some countries due to the long waiting time for expert consultation.
There are some limitations in our research.First of all, the sample size was quite small, and patients included in this study are all Chinese, thus the application of the results is restricted by ethnicity.Besides, this study was a cross-sectional study without long-term follow-up of patients, while we hope that the model can be used to predict the response to treatment in subsequent studies.We will continuously collect more patients' information to further improve the accuracy of the model.In future research, we hope we can add these factors to our model and expand our samples to observe whether it can be improved.

Conclusion
This research lays the foundation for the application of the random forest model in TAO.Random forest algorithm can generate a model that combines age, gender, smoking status, I-131 treatment history, thyroid function and TRAb to predict the clinical activity of TAO, having a good diagnostic performance.We suggest that the random forest model can be used as an auxiliary method to judge the clinical activity of patients with TAO, so as to distinguish patients who need urgent eye consultations from patients who only need regular follow-ups.

Figure 1 Patients
Figure 1

Figure 2 The
Figure 2