Performance Comparison of Improved Machine Learning Algorithms Based on Bayesian Optimization in High-dimensional and Unbalanced COPD Data

doi:10.21203/rs.3.rs-3239086/v1

Download PDF

Research Article

Performance Comparison of Improved Machine Learning Algorithms Based on Bayesian Optimization in High-dimensional and Unbalanced COPD Data

https://doi.org/10.21203/rs.3.rs-3239086/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background and objective: Early identification of individuals at high risk of chronic obstructive pulmonary disease (COPD) is crucial for reducing related mortality rates and economic burden. However, conventional machine learning (ML) models have limitations when making predictions using COPD data that exhibit high-dimensional and unbalanced characteristics. Therefore, to address this issue, this study developed a well-performing Bayesian optimization (BO)-ML hybrid model combined with variable screening and resampling technology to construct a COPD risk prediction model.

Methods: We collected a sample of 4,747 COPD cases with no missing data from the 2019 COPD Surveillance project in Shanxi Province, and extracted 34 potentially relevant variables from the dataset. Firstly, we used the Smoothly Clipped Absolute Deviation (SCAD) method to select variables associated with COPD. Secondly, we oversampling the unbalanced data using Synthetic Minority Over-sampling Technique (SMOTE) algorithm. Thirdly, we construct risk prediction models in the training set using four BO-improved ML models, including BO-Decision Tree (DT), BO-Naive Bayes (NB), BO-Support Vector Machine (SVM) and BO-K-nearest neighbor (KNN). Finally, the predictive performance of the combined models is tested and evaluated.

Result: The SCAD method was used to select 14 variables specifically associated with COPD from a dataset of 34 features. After applying the SMOTE resampling method, the ratio of COPD patients to non-COPD patients in the dataset of this study was balanced at 1:1. In the construction process of the four ML models, this study utilized BO algorithm to identify their optimal hyperparameters. Furthermore, in the comparison of model performance, this study found that combining BO-ML hybrid models with data balancing techniques can improve their performance. Specifically, the combination of SMOTE and BO-NB demonstrated stable performance and attained high scores in the comprehensive evaluation index, with AUC and G-means values of 0.770 and 0.696 respectively.

Conclusion: Despite the challenges posed by high dimensionality, redundancy, and class imbalance in data set, the BO-NB model, when integrated with SCAD and SMOTE, has exhibited excellent performance in accurately identifying individuals at a high risk of COPD. It provides early warnings to clinical doctors, helping them take timely preventive measures.

chronic obstructive pulmonary disease

Smoothly Clipped Absolute Deviation

Synthetic Minority Over-sampling Technique

Bayesian Optimization

Machine learning

Chronic Obstructive Pulmonary Disease (COPD) is a chronic inflammatory disease that is characterized by the progressive limitation of airflow. Its primary symptoms encompass persistent cough, excessive production of sputum, and a gradual deterioration of respiratory function^[1]. As of 2017, COPD had become the third leading cause of death worldwide^[2] and was ranked eighth in terms of reducing global life expectancy in 2019 ^[3]. Undoubtedly, COPD has become a significant public health issue. However, there remains a notable gap in the availability and quality of COPD screening, diagnosis, and treatment. Hence, the establishment of a robust COPD risk prediction model holds immense clinical significance as it can assist healthcare professionals in early detection and timely implementation of preventive measures.

A research survey conducted in China found that there are approximately 100 million COPD patients worldwide, with a prevalence rate of 13.7% among individuals aged 40 and above^[4]. Currently, our dataset shows a lower incidence rate of COPD at 9.3% in individuals aged 40 and above, indicating an issue of class imbalance. This issue significantly affects the classification accuracy of predictive models ^[5], particularly for the prediction of minority class samples, which are often the focus of research objectives. To address this challenge, the Synthetic Minority Over-sampling Technique (SMOTE) ^[6] will be employed in this study. Previously, SMOTE has been widely recognized as an effective solution for addressing class imbalance and has been applied in various domains, including computer vision, medical diagnosis, fraud detection, and more, to handle imbalanced data ^[7–9].

In addition, COPD is influenced by multiple factors, leading to high dimensionality and redundant information. Traditional risk prediction models often use a single classification algorithm without considering variable redundancy in the data, resulting in compromised performance. Considering this issue, Smoothly Clipped Absolute Deviation (SCAD) ^[10], a feature selection method, is employed in this study. SCAD is an improvement over the Lasso penalty method and selectively compresses coefficients of insignificant variables while preserving the significant ones. Therefore, this study employs SCAD for the selection of relevant variables associated with COPD.

With the rise of computer data mining techniques, machine learning (ML) algorithms are frequently utilized to aid experts and physicians in conducting clinical diagnostic research for COPD. For instance, Bodduluri S et al. employed a linear forward feature selection method to identify the most optimal feature set and utilized the K-nearest neighbor (KNN) learning algorithm to classify and recognize COPD ^[11]. Yu H et al. diagnosed the severity of COPD by utilizing Support Vector Machines (SVM) from sparse-channel lung sounds ^[12]. Wang C et al. developed a recognition model for Acute Exacerbations of Chronic Obstructive Pulmonary Disease using five ML algorithms ^[13]. However, each of the aforementioned ML algorithms encompasses a set of hyperparameters that significantly influence the algorithm's performance. In practice, determining the appropriate model hyperparameter configuration often relies on human expertise or iterative experimentation^[14]. As models become more complex with a growing number of hyperparameters, manual selection becomes increasingly challenging. Recently, Bayesian Optimization (BO) algorithm has gained widespread application as a tool for hyperparameter optimization, as it efficiently explores the hyperparameter space ^[15–17]. Given the advantages offered by the BO algorithm, this study aims to utilize the BO algorithm to find optimal hyperparameters for ML models and explore the feasibility of a hybrid BO-ML model for early identification of COPD.

In summary, this study explores approaches to address issues such as high-dimensional feature space, feature redundancy, and class imbalance in the investigation of COPD survey data. It employs SCAD for feature selection, utilizes SMOTE resampling to tackle class imbalance, applies BO to optimize ML methods for constructing the classification model, and investigates the impact of combining SMOTE with BO-ML models on model performance. The research aims to provide a more robust and effective approach for COPD risk prediction, contributing to early diagnosis and prevention of COPD.

2.1. Study participants

The data for this study was sourced from the 2019 COPD monitoring data of residents in Shanxi Province, China. After excluding missing data, a total of 4747 valid cases were retained. The survey employed a multi-stage stratified random sampling method to investigate residents aged 40 and above in 11 cities of Shanxi Province. Prior to the survey, the study obtained approval from the Sino-Japanese Friendship Hospital, and all participants or their representatives signed informed consent forms indicating their understanding and agreement. The survey included questionnaire surveys, anthropometric measurements, and pulmonary function tests. For specific details regarding the survey content, as well as the sampling methods and procedures, please refer to Appendix 1. Additionally, this study selected 34 variables from participants' demographic information, respiratory symptoms, smoking habits, living environments, and other related indicators. For specific variables and their distributions, please refer to Table S2 in Appendix 1.

The participants of the survey were Chinese citizens aged 40 years or older who had been residing at the surveillance sites for a minimum of 6 months prior to the survey. The exclusion criteria were shown below: (1) Residents in functional areas (e.g., workshops, military facilities, student dormitories, nursing homes, etc.). (2) Residents with mental or cognitive impairments (e.g., cognitive disorders, dementia, deafness, etc.). (3) Newly diagnosed or receiving treatment for tumors. (4) Residents with severe paralysis. (5) Pregnant or nursing women.

2.2. Definitions

According to pulmonary function test results, the diagnosis of COPD is based on a post-bronchodilation forced expiratory volume in 1 second to forced vital capacity ratio (FEV1/FVC) of less than 0.7. Body weight is classified as: low (BMI < 18.5 kg/m2), normal (BMI: 18.5 kg/m2 to 24 kg/m2), overweight (BMI: 24 kg/m2 to 28 kg/m2), or obese (BMI ≥ 28 kg/m2). Central obesity is defined as a waist circumference of ≥ 80 cm in women and ≥ 85 cm in men. Participants who reported smoking during the survey are categorized as current smokers, including both current and former smokers. Household air pollution is defined as the use of wood, animal dung, or coal for cooking or heating within the past six months or longer. Occupational exposure refers to exposure to dust or toxic gases in the workplace, including agricultural work. A family history of respiratory diseases is defined as the occurrence of respiratory conditions (such as asthma, chronic bronchitis, or emphysema) in either one or both parents.

2.3. SCAD

In the initial stage of model development ^[18], we considered incorporating numerous factors that could potentially influence the model to minimize bias. However, not all factors have significant effects. Introducing certain variables not only complicates calculations but also increases the risk of collinearity ^[19]. Therefore, this study utilizes SCAD for variable selection. SCAD, proposed by Fan and Li (2001) ^[10], incorporates different penalty terms based on different situations. Compared to Lasso, it allows for more precise feature variable selection and alleviates the issue of excessive compression. The specific parameter estimation form is as follows:

$$\widehat{\beta }=\text{arg}\text{min}\left\{-\text{ln}\left(\beta \right)\right.+n\sum _{j=1}^{k}{p}_{\lambda }\left(\left|{\beta }_{j}\right|\right)\}$$

The specific form of the penalty term is as follows:

$${p}_{\lambda }\left(\left|{\beta }_{j}\right|\right)=\left\{\begin{array}{c}\lambda \left|{\beta }_{j}\right|， \left|{\beta }_{j}\right|\le \lambda \\ \frac{-{\left|{\beta }_{j}\right|}^{2}-2a\lambda \left|{\beta }_{j}\right|+{\lambda }^{2}}{2a-2}，\lambda ＜\left|{\beta }_{j}\right|\le a\lambda \\ \frac{\left(a+1\right){\lambda }^{2}}{2}， \left|{\beta }_{j}\right|＞a\lambda \end{array}\right.$$

In the above equation, both λ ≥ 0 and a > 2 are adjustable parameters. Fan and Li ^[10] suggest setting the parameter a to 3.7 in their literature.

2.4. SMOTE

In our dataset, the proportion of non-COPD patients is nearly ten times higher than that of COPD patients, resulting in a significant class imbalance (See Supplementary Table S3.). This imbalance presents a challenge in developing accurate models. To address this issue, SMOTE was introduced by Chawla et al. ^[6] in 2002. SMOTE tackles the problem of class imbalance by generating synthetic samples for the minority class. These synthetic samples are created to have attribute characteristics similar to the minority class while also introducing some variations. By doing so, SMOTE helps balance the dataset and alleviate the skewness issue. Given the substantial class imbalance in our dataset, we will utilize SMOTE to address this problem.

2.5. Hyperparameter Optimization with Bayesian Optimization

Hyperparameters are crucial in achieving accurate predictions in machine learning. They are adjustable variables within the model or its training algorithm that need to be manually set before training. However, selecting the optimal hyperparameters can be challenging as they often have loosely constrained ranges. In the past, this task was accomplished using trial and error or relying on expert knowledge, but these methods are time-consuming and prone to bias. To overcome these limitations, the application of robust optimization techniques is necessary. Common methods for optimizing hyperparameters include grid search ^[20] (GS),, random search ^[21] (RS), and BO. Both GS and RS do not consider information from previous parameter evaluations, leading to slow search speeds and potential failure to find the global optimum. In contrast, the BO algorithm, introduced by Pelikan et al. ^[22], enables the optimization of complex objective functions with a limited number of function samples, even with fewer evaluations^{[17, 23]}. Moreover, the algorithmic framework of BO is sequential, allowing it to effectively utilize information from known data points^[24]. This makes it an important method for hyperparameter estimation.

2.6. Classification machine learning techniques based on Bayesian optimization.

Classification techniques are typically divided into two categories: supervised ML and unsupervised ML. The primary distinction between these algorithms is whether the training samples are labeled or not. In supervised ML algorithms, the training set contains class labels, while unsupervised ML methods are applied to unlabeled samples ^[25]. In this study, we conducted tests and comparisons of four popular supervised ML algorithms: Decision Trees (DT) ^[26], Naive Bayes (NB) ^[27], SVM ^[28], and KNN ^[29]. In medical binary classification problems, these models are commonly utilized. Each model is given identical input variables for consistency in the evaluation process. Furthermore, the BO algorithm was utilized to fine-tune the hyperparameter values of the aforementioned four ML models. Table 1 summarizes the hyperparameters and their search space used in this study.

Table 1

Hyperparameters and their search space of the proposed models
Classification Algorithm	Hyperparameters	Search Range
DT	Maximum number of splits	[1, (n*-1)]
DT	Split Criterion	Gini's diversity index, Twoing rule, and Maximum deviance reduction
NB	Distribution names	Gaussian / Kernel.
NB	Kernel type	Gaussian, Box, Epanechnikov, and Triangle.
SVM	Kernel Function	Gaussian, Linear, Quadratic, and Cubic.
	Kernel Scale	[0.001–1000]
	Box Constraint level	[0.001–1000]
	Standardize data	True/False
	Multiclass method	One-vs-One / One-vs-All
KNN	Number of neighbors	[1, (n*-1)]
	Distance metric	Euclidean, City block, Chebyshev, Minkowski (cubic), Mahalanobis, Cosine, Correlation, Spearman, Hamming and Jaccard
	Distance weight	Equal, Inverse, and Squared inverse.
	Standardize	True/False
(Note: n is the number of observations.)

2.7. Evaluation parameters

After completing the training and construction process of the models, evaluating and comparing the performance of the predictive models becomes an essential step. In this study, a variety of standard performance metrics, such as the area under the receiver operating characteristic curve (AUC), accuracy (ACC), specificity, sensitivity, and G-mean, were employed to evaluate the classifiers' performance. Some of these metrics can be calculated based on the confusion matrix (refer to Table 2).

Table 2

Confusion matrix
True label	Predicted label
True label	Positive	Negative
Positive	TP	FN
Negative	FP	TN
(Note: TP: COPD patients were correctly classified as COPD; TN: healthy participants were correctly classified as healthy; FP: healthy participants were incorrectly classified as COPD; FN: COPD patients were wrongly classified as healthy.)

$$Accuracy = \frac{\left(TN+TP\right)}{\left(TP+TN+FP+FN\right)}\times 100\%$$

$$Specificity =\frac{TN}{\left(TN+FP\right)}\times 100\%$$

$$Sensitivity = \frac{TP}{\left(TP+FN\right)}\times 100\%$$

$$G-mean=\sqrt{\frac{TP}{TP+FN}\times \frac{TN}{TN+FP}}\times 100\%$$

2.8. Statistic analysis

Statistical descriptive analysis of factors affecting COPD was conducted using IBM SPSS Version 24. SCAD feature selection was performed using the SCAD program in the SIS package of R software. SMOTE resampling was conducted using the imbalance-learning library in Python (version 3.10). All classification models were implemented on the MATLAB 2022a platform. The graphs in this paper were generated using Excel.

3.1. Experimental setup

To determine if the combination of SCAD, SMOTE resampling, and BO-ML models can improve classification performance, the following stages need to be completed:

1.Importing COPD Monitoring Data.

2. Using SCAD to select the most relevant features.

3. Splitting the original training set into a training set (70%) and a test set (30%). (Please see Supplementary Table S3.)

4.Balancing COPD Dataset with SMOTE.

5.Building BO-ML Models in both Balanced and Unbalanced Datasets.

6.Comparing and Evaluating Model Performance.

The flowchart can be found in Figure 1. Throughout this process, it will be observed whether the combination of these methods enhances or reduces the overall efficiency of the models. Furthermore, to ensure the models' ability to generalize, the training set underwent SMOTE resampling exclusively, while the test set retained the same feature variables without any additional processing.

3.2. Baseline characteristics

Among the initial 6,648 study participants, 1,901 individuals with incomplete data were excluded, resulting in a final analysis sample of 4,747 participants. Among them, 443 individuals (9.3%) were confirmed as COPD patients. The gender distribution showed that 48.9% were male and 51.1% were female. The age distribution of the participants was as follows: 26.9% were between 40 and 49 years old, 36.6% were between 50 and 59, 28.6% were between 60 and 69, and 7.9% were over 70 years old. More detailed information can be found in Supplementary Table S2. In Figure 2, it is evident that the higher prevalence of COPD among males compared to females; the prevalence of COPD decreases with increasing literacy; the prevalence of COPD increases with advancing age; Individuals with lower BMI values have a higher prevalence of COPD, particularly among those with low body weight, reaching 27.1%.

3.3. Using SCAD to screen COPD related factors

The SCAD model was utilized to incorporate 34 potential risk factors associated with COPD, and the "tune.method" option in the SIS package was used to specify the methods for selecting the optimal tuning parameter λ include AIC, BIC, eBIC, and CV. After debugging, to ensure the retention of sufficient information, this study employs AIC as the tune.method. Eventually, the SCAD method identifies 14 variables that exhibit strong correlations with COPD. As shown in Table 3.

Table 3 the selected variables and regression coefficients for the SCAD method.

Variables	AIC	BIC	eBIC	CV
Cough frequently at age 14 and before(X₂)	-0.30505211	-0.07537729	-	-
Hospitalization for pneumonia or bronchitis between the ages of 15 and 17(X₄)	0.50023039	-	-	-
Respiratory disease(X₅)	0.87865125	0.57747375	0.41777030	0.41777030
Gastroesophageal reflux(X₁₂)	-0.20792135	-	-	-
family history(X₁₄)	0.28988007	0.08573527	-	-
Current smoking(X₁₆)	0.44375813	0.00910660	0.07505639	0.07505639
Polluting fuel for household heating(X₁₈)	0.17471077	-	-	-
Age(X₂₂)	0.58876871	0.53305883	0.40153387	0.40153387
Marital status(X₂₄)	-0.13293717	-	-	-
Region(X₂₅)	0.02864334	-	-	-
Gender(X₂₆)	-1.03309069	-1.25037895	-0.85786855	-0.85786855
BMI(X₂₇)	-0.10398143	-	-	-
Kyphosis(X₃₁)	0.05966857	-	-	-
funnel chest(X₃₃)	1.89946875	-	-	-

In addition, to examine whether the variables selected by the SCAD method exhibit collinearity, we conducted a test for multicollinearity using the variance inflation factor (VIF). A VIF value below 5 indicates weak multicollinearity. Table 4 demonstrates that the selected variables have VIF values close to 1, indicating a weak collinearity among them. This suggests that by carefully selecting variables, we can effectively mitigate the adverse effects of feature collinearity on the classification performance of the model.

Table 4 VIF test value.

Variables	VIF value	Variables	VIF value
X₂	1.053	X₂₂	1.051
X₄	1.016	X₂₄	1.036
X₅	1.069	X₂₅	1.320
X₁₂	1.021	X₂₆	1.737
X₁₄	1.055	X₂₇	1.026
X₁₆	1.747	X₃₁	1.022
X₁₈	1.341	X₃₃	1.012

3.4. The results of SMOTE resampling.

The original training dataset exhibits a class imbalance, with a lower number of COPD patients (n=314) and a higher number of non-COPD patients (n=3008). After applying SMOTE resampling, a balanced distribution was achieved between the two categories, with a 1:1 ratio of COPD patients to non-patients. As shown in Table 5.

Table 5 Class distribution before and after SMOTE resampling.

Dataset	COPD patients	Non-COPD patients
The original training set	304	3008
After SMOTE resampling	3008	3008

3.5. Model establishment and evaluation

In the BO algorithm, the maximum number of evaluations for the objective function is set to "100" as a termination criterion. The optimization is run 8 times to find the optimal parameter values for each classifier in its specific search space. Table 6 summarizes the best configurations for all classifiers. The internal validation results for each classification model in the training dataset are summarized in Table 7.As shown in Table 6, before balancing the data, all classification models achieved a high specificity （1.000）but had extremely low sensitivity values （ranging from 0.000 to 0.021). This suggests that the classification models did not achieve satisfactory performance in accurately detecting COPD patients within the imbalanced dataset. In contrast, after applying SMOTE resampling to balance the data, there were significant improvements in the comprehensive evaluation metrics, namely AUC and G-mean, for all models. This outcome demonstrates the effectiveness of the data balancing process in enhancing the classification models' recognition performance for minority class samples.

When comparing different models, the BO-KNN model stood out for its relatively strong performance on the imbalanced dataset. During internal validation using the holdout method, the model achieved notable evaluation metrics: AUC (0.680), ACC (0.908), specificity (1.000), sensitivity (0.021), and G-mean (0.146). Nevertheless, after applying data balancing techniques, BO-DT performs excellently with the following metric values: AUC (0.920), ACC (0.860), Specificity (0.854), Sensitivity (0.867), G-mean (0.860). However, after applying data balancing techniques, the BO-DT model exhibited exceptional performance with improved metric values: AUC (0.920), ACC (0.860), specificity (0.854), sensitivity (0.867), and G-mean (0.860). These results highlight the superior performance of the SMOTE and BO-DT combination compared to other models in mitigating the effects of data imbalance.

Table 6 The optimal selection of hyperparameter values for different Models.

Models	Optimized Hyperparameters	Strategy
Models	Optimized Hyperparameters	Imbalance	SMOTE resampling
BO-DT	Maximum number of splits	2	420
BO-DT	Split Criterion	Maximum deviance reduction	Maximum deviance reduction

BO-NB	Distribution names	Kernel	Kernel
BO-NB	Kernel type	Gaussian	Box

BO-SVM	Kernel Function	Gaussian	Gaussian
	Kernel Scale	0.001	0.0043
	Box Constraint level	0.017	6.4026
	Standardize data	TRUE	FALSE
	Multiclass method	One-vs-One	One-vs-All

BO-KNN	Number of neighbors	8	2985
	Distance metric	Euclidean	Spearman
	Distance weight	Equal	Squared inverse
	Standardize	TRUE	FALSE

(Note: BO-DT: Bayesian optimization algorithm improved Decision Trees; BO-NB: Bayesian optimization algorithm improved Naive Bayes; BO-SVM: Bayesian optimization algorithm improved Support Vector Machines; BO-KNN: Bayesian optimization algorithm improved K-nearest neighbors.)

Table 7 summarizes the performance of the model on the internal validation data.

Models	AUC	ACC	Specificity	Sensitivity	G-mean
BO-DT	0.500	0.906	1.000	0.000	0.000
BO-NB	0.760	0.906	1.000	0.000	0.000
BO-SVM	0.530	0.906	1.000	0.000	0.000
BO-KNN	0.680	0.908	1.000	0.021	0.146
SMOTE+BO-DT	0.920	0.860	0.854	0.867	0.860
SMOTE+BO-NB	0.750	0.695	0.667	0.723	0.695
SMOTE+BO-SVM	0.820	0.836	0.729	0.942	0.829
SMOTE+BO-KNN	0.920	0.843	0.812	0.875	0.843

To ensure the models' ability to generalize, this study proceeded with external validation by employing a test set for each model. The results obtained from the external validation (refer to Table 8) align with the results from internal validation, indicating that the classification models improved their ability to identify COPD patients when using resampling techniques to handle the imbalanced dataset. During the external validation process of the imbalanced dataset, the BO-KNN model demonstrated performance that was consistent with the previous results from internal validation, and the performance was satisfactory. However, after balancing the data using SMOTE resampling, the BO-NB model demonstrates remarkable stability in generalization performance during external validation. At the same time, the BO-NB model combined with SMOTE outperforms other models in terms of evaluation metrics, especially with noticeably higher scores in comprehensive metrics such as AUC (0.770) and G-mean (0.696). This suggests that BO-NB exhibits a higher recognition rate for both positive and negative samples, as well as excellent overall predictive performance.

Table 8 summarizes the performance of the model on the external validation data

Models	AUC	ACC	Specificity	Sensitivity	G-mean
BO-DT	0.500	0.909	1.000	0.000	0.000
BO-NB	0.750	0.909	1.000	0.000	0.000
BO-SVM	0.540	0.909	1.000	0.000	0.000
BO-KNN	0.680	0.907	0.995	0.023	0.152
SMOTE+BO-DT	0.670	0.816	0.863	0.349	0.549
SMOTE+BO-NB	0.770	0.671	0.665	0.729	0.696
SMOTE+BO-SVM	0.610	0.738	0.765	0.465	0.597
SMOTE+BO-KNN	0.650	0.789	0.825	0.426	0.593

This study aimed to construct machine learning models using high-dimensional, highly redundant, and imbalanced medical data to accurately predict high-risk individuals for COPD. Traditional statistical methods were limited in this context, so SCAD and SMOTE techniques were applied to overcome class imbalance and feature redundancy. ML models optimized by BO were then constructed to improve COPD prediction accuracy. The results of four supervised classifiers based on these two data preprocessing methods were discussed. Performance evaluation metrics such as AUC, ACC, specificity, sensitivity, and G-mean were used to assess the models. The predictive models developed in this study aimed to assist clinical practitioners in identifying individuals at high risk for COPD. If an individual's predicted value is 1, they are classified as part of the high-risk population, enabling appropriate screening and early-stage diagnosis to improve patient care and outcomes.

During feature engineering, this study utilized the SCAD method to reduce the dimensionality of a dataset comprised of 34 features. This method effectively mitigated the impact of high dimensionality, redundant information, and collinearity among variables on the model's performance, while also reducing the complexity in subsequent model construction. Finally, SCAD identified 14 features that are closely related to COPD. Previous research has confirmed that the variables identified in this study, such as frequent coughing before the age of 14, hospitalization for pneumonia or bronchitis between the ages of 15 and 17, respiratory diseases, gastroesophageal reflux, family history, current smoking, and the remaining eight characteristics, are all significant risk factors for COPD^[30–34]. Given these factors, the selection of variables in this study is justified as reasonable.

Secondly, regarding data imbalance, four classification models showed high accuracy (> 0.9) before applying SMOTE resampling. Nonetheless, these models had low sensitivity in detecting individuals at high risk for COPD and a high rate of false negatives. This highlights the limitations of relying solely on accuracy when dealing with imbalanced data. After implementing SMOTE resampling, the ratio of COPD patients to non-COPD patients was balanced (at a ratio of 1:1). Moreover, using balanced data significantly improved the ability of all classification models to identify high-risk individuals for COPD. The BO-NB model, in particular, showed the greatest improvement, with sensitivity increasing from 0 to 0.729. This underscores the importance of employing data balancing techniques to address the impact of data imbalance.

Thirdly, in model construction, this study utilizes the BO algorithm to select the optimal hyperparameters for eight supervised classifiers. Previously, BO has been successfully utilized by many researchers for hyperparameter optimization of ML models. For example, eynep CEYLA et al. determined the optimal hyperparameter set for different ML models using BO to predict high-risk groups of breast cancer patients, and the results showed that BO effectively improves the performance of ML models ^[15]. In another study, Ding Kexin, after optimizing XG-Boost parameters using BO, found that BO outperformed the RS optimization method in terms of model parameter selection and optimization efficiency^[35]. These studies highlight the importance of using BO to optimize hyperparameters in ML model construction. Therefore, it is justified to employ BO algorithm for hyperparameter optimization in this study.

Lastly, in the comparative analysis of model performance, we found that although BO- improved KNN exhibited better classification performance in imbalanced data, it still had a lower recognition rate for minority class samples. While the performance of the BO- improved NB model showed relative superiority after implementing SMOTE to address the issue of data imbalance. Previous studies have demonstrated the strong predictive ability of NB in various research domains. For instance, in a study classifying mild cognitive impairments, NB outperformed random forest, logistic regression, and KNN ^[36]. In another study predicting prostate cancer staging, NB performed comparably to more complex classifiers such as SVM and artificial neural networks ^[37]. Furthermore, NB has shown promising performance in various research domains, such as neonatal jaundice diagnosis and brain tumor classification ^{[38, 39]}.

We acknowledge the limitations of our study. Firstly, the predictive factors included in this study only encompass questionnaire information and basic physical measurements obtained from COPD monitoring data, without incorporating data from lung function monitoring. Consequently, the identification rate of COPD is relatively low. Furthermore, further exploration is required to ensure the robustness of the model we constructed in diverse data application scenarios.

In summary, considering the challenges of high dimensionality, redundancy, and class imbalance in COPD risk prediction factors, this study proposes the BO-NB model combining SCAD feature selection and SMOTE resampling techniques to identify individuals at high risk of COPD. The utilization of this model can greatly assist clinical practitioners in providing early warnings for COPD and implementing timely and effective preventive measures.

COPD: chronic obstructive pulmonary disease; SCAD: Smoothly Clipped Absolute Deviation; SMOTE: Synthetic Minority Over-sampling Technique; ML: machine learning; BO: Bayesian optimization; DT: Decision Trees; NB: Naive Bayes; SVM: Support Vector Machines; KNN: K-nearest neighbors; BO-ML: Bayesian optimization algorithm improved machine learning; BO-DT: Bayesian optimization algorithm improved Decision Trees; BO-NB: Bayesian optimization algorithm improved Naive Bayes; BO-SVM: Bayesian optimization algorithm improved Support Vector Machines; BO-KNN: Bayesian optimization algorithm improved K-nearest neighbors.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

This study was approved by the China-Japan Friendship Hospital. Informed consent was signed by all study participants or their agents. All experiments and methods were performed under the relevant guidelines and regulations.

Consent for publication

Not applicable.

Author Contributions Statement

YTL, XCW, and LXQ participated in research design; XCW, YCQ, JHR, HR, YC, JL, and RQZ conducted the survey and collected data; YTL analyzed and interpreted the data; XCW, YCQ and JHR were responsible for preprocessing the data and checking the results; QLX gave constructive suggestions for the manuscript. All authors read and approved the final manuscript.

Funding:

This work was funded by the National Natural Science Foundation of China (Grant No: 81973155).

Conflicts of Interest:

The authors declare no conflict of interest.

Acknowledgements

The authors thank all the interviewees who participated in the survey data collection.

Singh D, Agusti A, Anzueto A, et al. Global strategy for the diagnosis, management, and prevention of chronic obstructive lung disease: the GOLD science committee report 2019. Eur Respir J. 2019;53(5):1900164.
Soriano JB, Kendrick PJ, Paulson KR, et al. Prevalence and attributable health burden of chronic respiratory diseases, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017. The Lancet Respiratory Medicine. 2020;8(6):585–96.
WHO Department of Data and Analytics. Global health estimates 2019: disease burden by cause, age, sex, by country and by region, 2000–2019. Geneva: World Health Organization; 2020.
Wang C, Xu J, Yang L, et al. Prevalence and risk factors of chronic obstructive pulmonary disease in China (the China Pulmonary Health [CPH] study): a national cross-sectional study. The Lancet. 2018;391(10131):1706–17.
Wang Xiaomeng S, Desheng Z. Application of Boosting algorithm combined with SMOTE technique in predicting HIV infection in young men who have sex with men. Chin J Health Stat. 2012;39(01):31–5.
Chawla NV, Bowyer KW, Hall LO, et al. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321–57.
Van Horn G, Mac Aodha O, Song Y, et al. The inaturalist challenge 2017 dataset. arXiv preprint arXiv:1707 06642. 2017;1(2):4.
Min X, Li M, Dong D, et al. Multi-parametric MRI-based radiomics signature for discriminating between clinically significant and insignificant prostate cancer: Cross-validation of a machine learning method. Eur J Radiol. 2019;115:16–21.
He B, Ji T, Zhang H, et al. MRI-based radiomics signature for tumor grading of rectal carcinoma using random forest model. J Cell Physiol. 2019;234(11):20501–9.
Li FRVariable. Selection via Nonconcave Penalized Likelihood and its Oracle Properties. Publications of the American Statistical Association. 2001;96(456):1348–60.
Bodduluri S, Newell JD, Hoffman EA, et al. Registration-Based Lung Mechanical Analysis of Chronic Obstructive Pulmonary Disease (COPD) Using a Supervised Machine Learning Framework. Acad Radiol. 2013;20(5):527–36.
Yu H, Zhao J, Liu D, et al. Multi-channel lung sounds intelligent diagnosis of chronic obstructive pulmonary disease. BMC Pulm Med. 2021;21(1):1–13.
Wang C, Chen X, Du L, et al. Comparison of machine learning algorithms for the identification of acute exacerbations in chronic obstructive pulmonary disease. Comput Methods Programs Biomed. 2020;188:105267.
Snoek J, Larochelle H, Adams RP. Practical bayesian optimization of machine learning algorithms. Adv Neural Inf Process Syst. 2012; 25.
Ceylan Z. Diagnosis of breast cancer using improved machine learning algorithms based on bayesian optimization. Int J Intell Syst Appl Eng. 2020;8(3):121–30.
Wu J, Chen XY, Zhang H, et al. Hyperparameter optimization for machine learning models based on Bayesian optimization. J Electron Sci Technol. 2019;17(1):26–40.
Shahriari B, Swersky K, Wang Z et al. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE. 2015; 104(1): 148–175.
MichaelH.Kutner. ChristopherJ.Nachtsheim, JohnNeter.Applied linear regression models. Photocopy edition. Higher Education Press; 2005.
Wang Lu SJ. Application of Lasso regression method in feature variable selection. J Jilin Inst Eng Technol. 2021;37(12):109–12.
Bao Y, Liu Z. A fast grid search method in support vector regression forecasting time series[C]//Intelligent Data Engineering and Automated Learning–IDEAL 2006: 7th International Conference, Burgos, Spain, September 20–23, 2006. Proceedings 7. Springer Berlin Heidelberg, 2006: 504–511.
BERGSTRA J, BENGIO Y. Random Search for Hyper-Parameter Optimization. J Mach Learn Res. 2012;13(1):281–305.
Pelikan M, Goldberg DE, Cantú-Paz E. BOA: The Bayesian optimization algorithm[C]//Proceedings of the genetic and evolutionary computation conference GECCO-99. 1999, 1(1999).
Wu J, Chen XY, Zhang H, et al. Hyperparameter optimization for machine learning models based on Bayesian optimization. J Electron Sci Technol. 2019;17(1):26–40.
JONES D R, SCHONLAU M. Efficient Global Optimization of Expensive Black-Box Functions. J Global Optim. 1998;13(4):455–92.
BAO W, LIANJU N. Integration of Unsupervised and Supervised Machine Learning Algorithms for Credit Risk Assessment. Expert Syst Appl. 2019;128(AUG):301–15.
Steinberg D, Colla P. CART: classification and regression trees. The top ten algorithms in data mining. 2009; 9: 179.
CHESHIRE J. A First Course in Bayesian Statistical Methods. A First Course in Bayesian Statistical Methods; 2009.
CORTES C. Support-Vector Networks. Mach Learn. 1995;20(3):273–97.
Zeng Zhihao. Research on kNN classification algorithm and its application in poisoning diagnosis. Hunan University; 2005.
Quan D, Ren J, Ren H, et al. Exploring influencing factors of chronic obstructive pulmonary disease based on elastic net and Bayesian network. Sci Rep. 2022;12(1):7563.
Wang Jing S, Jian D, Aibing, et al. Analysis of the prevalence and influencing factors of chronic obstructive pulmonary disease in the ≥ 60-year-old health examination population in Cangzhou City. South China J Prev Med. 2021;47(06):781–3.
Wang X, Wright Z, Wang J, et al. Elucidating the Link: Chronic Obstructive Pulmonary Disease and the Complex Interplay of Gastroesophageal Reflux Disease and Reflux-Related Complications. Medicina. 2023;59(7):1270.
Muhammed A, Moiz JA, Singla D, et al. Postural abnormalities in phenotypes of chronic obstructive pulmonary disease. Braz J Phys Ther. 2020;24(4):325–32.
Wang X, Ren H, Ren J, et al. Machine learning-enabled risk prediction of chronic obstructive pulmonary disease with unbalanced data. Comput Methods Programs Biomed. 2023;230:107340.
Ding Kexin. Research on liver cancer survival prediction based on machine learning methods. Huazhong Agricultural University; 2022.
Jia Zhiying. Exploration and research on the dynamic optimization screening system for mild cognitive impairment based on machine learning. Shanghai Jiao Tong University; 2019.
Cosma G, Acampora G, Brown D, et al. Prediction of pathological stage in patients with prostate cancer: a neuro-fuzzy model. PLoS ONE. 2016;11(6):e0155856.
Ferreira D, Oliveira A, Freitas A. Applying data mining techniques to improve diagnosis in neonatal jaundice. BMC Med Inf Decis Mak. 2012;12(1):1–6.
Tsolaki E, Svolos P, Kousi E, et al. Fast spectroscopic multiple analysis (FASMA) for brain tumor classification: a clinical decision support system utilizing multi-parametric 3T MR data. Int J Comput Assist Radiol Surg. 2015;10:1149–66.

No competing interests reported.

Attachedfile.docx
Additional file 1: The first section provides specific details about the survey content, as well as the sampling methods and procedures. Supplementary Table S2 displays specific variables and their distributions. Supplementary Table S3 summarizes the sample situation.

Download PDF

Version 1

posted

You are reading this latest preprint version

Performance Comparison of Improved Machine Learning Algorithms Based on Bayesian Optimization in High-dimensional and Unbalanced COPD Data

Status:

Version 1

Abstract

Figures

1. Introduction

2. Methods

2.1. Study participants

2.2. Definitions

2.3. SCAD

2.4. SMOTE

2.5. Hyperparameter Optimization with Bayesian Optimization

2.6. Classification machine learning techniques based on Bayesian optimization.

2.7. Evaluation parameters

2.8. Statistic analysis

3. Results

4. Discussion

5. Limitations

6. Conclusion

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1