Assessing Machine Learning classifiers in COVID-19: The Role of Clinical, Laboratory, and Radiological Features in Predicting Oxygen Saturation

doi:10.21203/rs.3.rs-5031337/v1

Download PDF

Research Article

Assessing Machine Learning classifiers in COVID-19: The Role of Clinical, Laboratory, and Radiological Features in Predicting Oxygen Saturation

https://doi.org/10.21203/rs.3.rs-5031337/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Oxygen saturation is vital for evaluating COVID-19 severity in hospitalized patients, with levels below 90% indicating respiratory distress and a potential need for intensive care.

Objective

This study develops machine learning models that integrate CT-based features with clinical and laboratory data to predict binary oxygen saturation outcomes in COVID-19 patients.

Method

A retrospective study of 1008 COVID-19 patients admitted between October 2020 and May 2021, using 70% of data for training and 30% for testing. Classifiers used: Linear SVM, SVM with RBF kernels, Logistic Regression, Random Forests, Naïve Bayes, and XGBoost. Performance assessed by validation AUC and 10-fold cross-validation AUC range. Significant features identified by the top validation AUC classifier, prioritizing the top three with importance and stability scores over 0.7.

Results

Linear ML classifiers performed well in Clinical and Laboratory Models, while non-linear classifiers excelled in CT-Based and Integrated Models. Logistic Regression in the Clinical Model achieved an AUC of 0.82, with Age, Gender, and Fever as significant features. In the Laboratory Model, Linear SVM (0.82) identified White Blood Cell count as key. Random Forest in the CT-Based Model (0.87) highlighted Mean Lesion Volume. The Integrated Model's top classifier, SVM with RBF Kernel (0.89), found WBC and Mean NLLV critical.

Conclusion

Linear classifiers effectively predict oxygen saturation using clinical and laboratory data, while non-linear classifiers excel with CT-based and integrated models, highlighting the need for tailored machine learning approaches to different data types in COVID-19 patient care.

Artificial Intelligence and Machine Learning

Nuclear Medicine & Medical Imaging

Infectious Diseases

Radiomics

Coronavirus Disease 2019 (COVID-19)

Oxygen Saturation

Machine Learning

Computed Tomography

Predictive Modeling

The COVID-19 pandemic has profoundly impacted global healthcare systems, necessitating swift and precise evaluations of disease severity. Critical in this assessment is oxygen saturation, a vital indicator of respiratory function, where levels below 90% suggest severe respiratory compromise(1, 2). Computed tomography (CT) scans play a crucial role in gauging the severity of COVID-19 by offering prognostic insights not captured by standard methods. The extent of lung lesions visible on CT scans correlates significantly with the disease's severity, providing a measurable index of lung involvement. Notably, specific radiographic patterns, like the "crazy-paving pattern," indicate advancing severity towards substantial lung consolidation, highlighting the transition phases within the pulmonary structure affected by the infection(2–5).

This evolving understanding underscores the potential of advanced machine learning models that integrate CT data with clinical and laboratory assessments to enhance the prediction accuracy of critical outcomes like oxygen saturation. The intersection of machine learning and radiomics has transformed medical imaging analysis(4, 6–8). The development of sophisticated algorithms facilitates deep explorations into high-dimensional imaging data(9, 10), thus broadening the horizon for improved diagnostic precision and predictive capabilities in managing COVID-19 outcomes. However, the adoption of these advanced techniques in clinical practice is hampered by challenges such as biases in feature extraction and selection, which can undermine the reliability of predictive models(11–14). Furthermore, without careful management, feature selection processes might prioritize computational artifacts over clinically relevant data, necessitating a meticulous approach to ensure the utility of CT-integrated machine learning models in clinical settings

This study aims to develop machine learning models that incorporate CT-based interpretable features with clinical and laboratory data to predict binary oxygen saturation outcomes in COVID-19 patients. By evaluating both linear and non-linear classifiers, this research seeks to assess their effectiveness in forecasting oxygen saturation levels, considering the evolution of CT scan features from ground-glass opacities to complete lung consolidation. Incorporating domain knowledge to distinguish between clinically relevant features and computational artifacts is crucial, ensuring that the models remain applicable in real-world clinical settings, particularly regarding decisions on ICU admissions and mechanical ventilation requirements.

Study Design

A retrospective study design was chosen to analyze a cohort of patients diagnosed with COVID-19. The study included patients admitted to Baqiyatallah hospital. Patients were recruited from October 2020 to May 2021, during which all individuals admitted with a confirmed COVID-19 diagnosis were eligible for inclusion.

Patient data were obtained from hospital records, focusing on adults aged 18 and older. The primary inclusion criteria were as follows: a positive COVID-19 diagnosis confirmed by RT-PCR, admission to the hospital for COVID-19 treatment, and chest CT examination within one day of admission. Exclusion criteria were patients with incomplete clinical data due to transfer from other hospitals or other reasons. The final cohort consisted of 1008 out of 1744 after applying the inclusion and exclusion criteria. The study received ethical approval from the hospital's ethics committee (IR.BMSU.BAQ.REC.1400.079). All patient data were anonymized to ensure confidentiality and privacy. Included patients were divided into groups based on their oxygen saturation levels by the cutoff of 90%. Subsequently, the patient cohort was divided into training and testing groups for model development. Approximately 70% of the data was used for training, with the remaining 30% for testing.

Data collection

Clinical data were extracted from electronic health records (EHR) and hospital databases. This process involved identifying patients who met the study's inclusion criteria and retrieving their complete medical history, clinical assessments, and treatment records. Demographic information, including age and gender, was collected at admission. Contact history, documenting potential exposure to COVID-19, was obtained through patient interviews and travel history reports. Medical history, detailing pre-existing conditions like hypertension, diabetes, coronary heart disease, surgery, and hepatitis B, was also extracted from hospital records and patient self-reports. Clinical symptoms such as fever, cough, chills, dizziness, fatigue, and body ache were documented based on patient self-reporting and clinical observations by medical staff. Laboratory data were collected through routine blood tests and arterial blood gas tests conducted upon patient admission. Routine blood tests provided values for white blood cell counts, lymphocytes, eosinophils, neutrophils, and C-Reactive Protein (CRP). Additional biomarkers were measured to assess tissue and organ function, including D-Dimer, lactate dehydrogenase (LDH), creatine kinase isoenzyme, Alanine Aminotransferase (ALT), Aspartate Aminotransferase (AST), blood creatinine, blood urea nitrogen, and procalcitonin. After collection, data were integrated into a single dataset and cleaned to remove incomplete or redundant records.

CT Image Acquisition

The CT images were acquired using a multi-detector CT scanner, GE Revolution EVO 64-slice CT (GE Healthcare, Milwaukee, WI), within one day of hospital admission to ensure consistent imaging conditions and minimize external factors that could affect image quality. The scanning parameters were set to supine position with 120 kVp tube voltage, automatic tube current modulation, 0.725 mm collimation, and 1 mm and 5 mm reconstruction intervals. The scanning range spanned from the thoracic inlet to the upper abdomen, ensuring comprehensive lung coverage. The 12-bit CT images were mapped to 8-bit images with lung-specific window-level settings (window width: 1000 HU; window level: -500 HU).

Image Segmentation and Processing

A segmentation process was applied to the CT images. This process used two-dimensional U-Net models to segment lung and lesion regions. Each convolutional layer used a 3 × 3 kernel, followed by batch normalization and activation with a rectified linear unit (ReLU). The final segmentation layers determined the boundaries for lung and lesion regions. Data augmentation techniques were included random rotations of up to 20 degrees, random shear transformations, and zoom levels ranging from 0.9 to 1.1 times to improve the model's ability to generalize across variations in CT images. After automated segmentation, experienced radiologists manually reviewed the results. The manual review involved four radiologists with a minimum of two years of experience in chest imaging. The final confirmation and correction were made by two senior radiologists, with 6 and over 11 years of experience.

CT-based features

Following the segmentation process, a variety of CT variables were quantified to assess lung structure and disease progression. The lung volume, lesion volume, non-lesion lung volume (NLLV), and the fraction of NLLV (%NLLV) were calculated to determine the extent of disease involvement. Additionally, a comprehensive set of histogram texture features, including mean, standard deviation, skewness, kurtosis, energy, and entropy, was calculated for both lesions and NLLV. This quantification provided a robust dataset from which key CT variables were extracted for further analysis and modeling.

The CT variables were encoded to create a dataset suitable for machine learning modeling. Binary variables, representing the presence or absence of specific features, were encoded using binary encoding where a value of '1' indicated the feature's presence, and '0' indicated its absence. This approach was applied to a range of CT features, including Ground Glass Opacities (GGO), Consolidation, Crazy Paving Pattern, Halo Sign, Reverse Halo Sign, Peripheral Distribution, Lower Zone Predominance, Traction Bronchiectasis, Vascular Thickening, Subpleural Lines, Air Bronchograms, Pleural Effusion, Interstitial Thickening, Lymphadenopathy, Cavitation, and Fibrotic Bands.

Data integration

The one-hot encoding technique was used for categorical variables to convert each distinct category into a separate binary feature, thus capturing the inherent information without introducing bias. This technique was applied to variables such as nodules (for size, number, and location) and architectural distortion (various types). Continuous variables, with a range of potential values, were standardized using the z-score method, providing a uniform scale across the dataset. Variables such as lung volume, mosaic attenuation pattern, and white blood cell count required this approach to prevent disproportionate influence by features with higher numerical ranges.

Feature Selection

To train machine learning (ML) classifiers—including Linear Support Vector Machine (SVM), SVM with Radial Basis Function Kernels (SVM_RBF), Logistic Regression, Random Forests, Naïve Bayes, and Extreme Gradient Boosting (XGBoost)—an optimal feature count range of 8 to 22 was identified. This range varied based on each classifier's feature handling properties and aimed to optimize training efficacy and generalizability(15–21), given the training (n ≈ 700) and validation (n ≈ 300) sample sizes. The selection aimed to capture the central tendency, dispersion, complexity, and heterogeneity of the CT images of the lungs. Cross-validation (10-fold) was used to validate the feature selection process, ensuring consistency across different subsets of data.

Recursive Feature Elimination (RFE) was employed to select the most relevant features by iteratively training a model, ranking features by importance, and removing the least significant ones. This process continued until an optimal subset was obtained, particularly for SVM with RBF kernels and Linear SVM classifiers, which are sensitive to feature selection. This technique typically identified 7 to 13 optimal features, depending on the classifier.

For Random Forest and XGBoost classifiers, feature importance was calculated based on the frequency of feature use in decision tree splits and its impact on reducing impurity. This approach led to the selection of the most impactful features, generally between 13 and 22, based on their contribution to the overall model's accuracy and robustness. Logistic Regression and Naïve Bayes classifiers used a statistical approach for feature selection, examining the correlation between each feature and the target variable. Pearson's correlation was used for continuous variables, while chi-square tests were used for categorical variables. Features with statistically significant correlations (p < 0.05) were retained. Additionally, the Minimum Redundancy Maximum Relevancy (mRMR) method was applied to minimize feature redundancy, typically yielding a set of 10 to 15 significant features.

ML Classifiers

We implemented multiple imputation using chained equations instead of median imputation and designed two data preprocessing pipelines: one utilized StandardScaler for feature normalization, while the other retained the original feature scales. The multiple imputation process involved 20 iterations, following Rubin's rules. We chose the ML models based on their proven effectiveness with similar datasets. The selected models were Linear SVM (with a linear kernel and a tuned regularization parameter C), SVMRBF (optimized for C and gamma), Logistic Regression (utilizing C for regularization with both l1 and l2 techniques), Random Forests (fine-tuning max_depth, min_samples_split, and min_samples_leaf), Naïve Bayes (optimized var_smoothing), and XGBoost (optimized learning_rate, max_depth, n_estimators, min_child_weight).

Bayesian optimization was used for hyperparameter tuning, replacing grid search to improve efficiency. This approach used Gaussian processes as priors to explore the hyperparameter space with 4-fold cross-validation to determine optimal settings based on accuracy. To address class imbalances, we conducted exploratory data analysis to identify and quantify imbalances. We used Adaptive Synthetic Sampling (ADASYN) and Synthetic Minority Over-sampling Technique (SMOTE) for resampling. Additionally, cost-sensitive learning adjusted misclassification costs to favor correct predictions of the minority class. Balanced accuracy and the F1 score were chosen as performance metrics due to their sensitivity to class imbalances.

Validation Strategy

Our validation strategy employed 10-fold cross-validation with stratification to maintain class distribution across folds. We assessed classifier performance using adjusted sensitivity, specificity, and precision metrics, focusing on accurate classification of the minority class. Model performance was analyzed using the area under the curve (AUC), considering the AUC range across folds and the overall AUC for the validation dataset. This approach provided insights into model stability and generalizability.

Feature Importance and Stability

To assess feature importance, we used normalized coefficients in linear models like Linear SVM and Logistic Regression. Ensemble methods, such as Random Forests and XGBoost, utilized built-in feature importance metrics, complemented by Permutation Importance for a more comprehensive evaluation. RFE was applied for SVMRBF to dynamically evaluate feature contributions. Naïve Bayes determined feature significance through variance ranking. SHapley Additive explanations (SHAP) values were used across all models to offer interpretable, model-agnostic insights into feature importance. Stability selection techniques utilized subsampling and aggregation to calculate stability percentages for each feature across various data subsets.

Unsupervised Learning

For unsupervised learning, Principal Component Analysis (PCA) was conducted to achieve two components from each model, aiding SVM_RBF decision boundary plotting. We retained Principal Components based on eigenvalue > 1 and scree plot analysis, ensuring significant variance capture while avoiding overfitting.

Configurations of the Generative Pre-trained Transformer (GPT)

This study utilized specialized configurations of the GPT-4 model, meticulously designed through advanced prompt engineering and schema modifications to meet specific research requirements. The range of tasks included optimizing machine learning classifier parameters, conducting comprehensive literature reviews, and enhancing manuscript language quality.

The characteristics of COVID-19 patients with oxygen saturation levels below and above 90% were examined in two distinct cohorts, each with training and validation groups. The first cohort includes patients with oxygen saturation below 90%, comprising 224 in the training group and 96 in the validation group. The second cohort involves patients with oxygen saturation equal to or above 90%, with 482 in the training group and 206 in the validation group. A detailed breakdown of clinical characteristics, biological measures, symptoms, and CT features is provided in Table 1. Figure 1 depicts the typical progression of lung damage in COVID-19, from ground-glass opacity to consolidation, reflecting the increasing severity of the disease.

Figure 1 illustrates the radiographic progression of lung involvement in COVID-19 pneumonia, beginning with ground-glass opacity (GGO), an early radiographic finding representing alveolar damage and fluid accumulation (a, b). As the disease advances, GGO may increase in distribution, exhibiting peripheral predominance (c). The crazy paving pattern, characterized by thickened interlobular septa superimposed on ground-glass opacity, indicates deeper lung parenchyma involvement (c, d). Further progression leads to consolidation (e, f), where dense opacities obscure the underlying vasculature, signaling severe alveolar damage. This figure helps radiologists assess disease severity and predict clinical outcomes based on imaging patterns.

Table 1
	O2 < 90; Training (N = 224)	O2 < 90; Validation (N = 96)	O2 ≥ 90; Training (N = 482)	O2 ≥ 90; Validation (N = 206)		O2 < 90; Training (N = 224)	O2 < 90; Validation (N = 96)	O2 ≥ 90; Training (N = 482)	O2 ≥ 90; Validation (N = 206)
Clinical characteristics					Biological Measures
Age (years)	64.8 ± 12.3	66.2 ± 12.8	55.4 ± 10.9	54.6 ± 11.2	White Blood Cell Count (µL)	10,530 ± 2,450	10,620 ± 2,550	9,400 ± 2,200	9,450 ± 2,100
Gender (Female)	45.1 (101)	46.9 (45)	48.3 (233)	47.6 (98)	Lymphocyte Count (µL)	796 ± 152	820 ± 160	1,050 ± 230	1,060 ± 240
BMI (kg/m²)	28.47 ± 4.21	28.6 ± 4.5	26.7 ± 3.9	26.2 ± 3.7	Eosinophil Count (µL)	41.2 ± 19.8	43.7 ± 22.3	55.6 ± 25.1	56.4 ± 26.1
Contact History	33.0 (74)	35.4 (34)	28.4 (137)	27.7 (57)	Neutrophil Count (µL)	7,987 ± 1,989	8,080 ± 2,030	6,400 ± 1,800	6,460 ± 1,790
Clinical Examination					C-Reactive Protein (CRP, mg/L)	60.3 ± 20.7	61.2 ± 21.1	38.4 ± 15.6	37.7 ± 16.0
Oxygen Saturation (%)	85.3 ± 3.9	85.5 ± 4.1	95.2 ± 3.2	94.7 ± 3.1	Platelet Count (µL)	180,000 ± 29,700	181,500 ± 31,200	200,500 ± 25,600	201,000 ± 25,100
Diastolic Pressure (mmHg)	75.1 ± 10.4	76.3 ± 10.7	73.5 ± 9.4	72.4 ± 8.8	D-Dimer (µg/mL)	2.48 ± 1.03	2.65 ± 1.10	1.80 ± 0.80	1.75 ± 0.78
Respiratory Rate (breaths/min)	24.8 ± 5.7	25.4 ± 5.9	18.7 ± 4.6	18.4 ± 4.2	Lactate Dehydrogenase (LDH, U/L)	298 ± 98	306 ± 103	265 ± 79	262 ± 77
Systolic Pressure (mmHg)	129.7 ± 15.3	131.2 ± 15.7	120.8 ± 14.2	118.9 ± 14.0	Alanine Aminotransferase (ALT, U/L)	40.1 ± 14.8	41.3 ± 15.2	32.7 ± 11.5	33.1 ± 12.4
Body Temperature (°C)	38.52 ± 1.19	38.4 ± 1.3	37.3 ± 0.9	37.2 ± 1.0	Aspartate Aminotransferase (AST, U/L)	50.3 ± 20.2	52.1 ± 21.7	40.8 ± 14.3	41.2 ± 13.7
Comorbidities and Smoking					Procalcitonin (µg/mL)	0.20 ± 0.11	0.22 ± 0.13	0.10 ± 0.05	0.12 ± 0.06
Hypertension	55.0 (123)	57.3 (55)	45.2 (218)	44.7 (92)	CT Features
Diabetes	20.1 (45)	21.9 (21)	15.6 (75)	16.5 (34)	Lesion Volume (mL)	502 ± 101	508 ± 105	250 ± 75	248 ± 72
Cardiovascular Disease	17.9 (40)	18.8 (18)	12.2 (59)	11.7 (24)	Non-Lesion Lung Volume (NLLV, mL)	2,005 ± 505	2,010 ± 520	2,700 ± 600	2,710 ± 610
COPD	12.1 (27)	11.5 (11)	8.92 (43)	9.71 (20)	GGO	75.0 (168)	78.1 (75)	45.0 (217)	43.2 (89)
Chronic Liver Disease	8.04 (18)	9.38 (9)	7.26 (35)	6.80 (14)	Consolidation	59.8 (134)	62.5 (60)	25.6 (123)	26.2 (54)
Asthma	9.82 (22)	10.4 (10)	8.09 (39)	8.25 (17)	Crazy Paving	50.0 (112)	52.1 (50)	30.1 (145)	29.1 (60)
Emphysema	5.80 (13)	6.25 (6)	5.18 (25)	5.34 (11)	Halo Sign	40.2 (90)	42.7 (41)	20.8 (101)	18.9 (39)
Cancer	7.14 (16)	6.25 (6)	4.56 (22)	5.83 (12)	Reversed Halo Sign	35.3 (79)	36.5 (35)	17.4 (84)	16.5 (34)
Symptoms					Peripheral Topography	79.9 (179)	81.3 (78)	50.8 (245)	49.5 (102)
Fever	84.8 (190)	82.3 (79)	67.6 (326)	69.4 (143)	Lower Zone Predominance	70.1 (157)	71.9 (69)	45.6 (220)	44.7 (92)
Cough	80.4 (179)	78.1 (75)	72.2 (348)	73.3 (151)	Vascular Thickening	50.0 (112)	52.1 (50)	30.1 (145)	29.1 (60)
Chills	45.1 (101)	46.9 (45)	28.4 (137)	26.7 (55)	Subpleural Lines	55.0 (123)	56.3 (54)	36.7 (177)	37.9 (78)
Fatigue	70.1 (157)	72.9 (70)	52.5 (253)	51.9 (107)
Body Aches	65.2 (146)	68.8 (66)	51.7 (249)	50.5 (104)
Dizziness	29.9 (67)	31.3 (30)	22.0 (106)	21.4 (44)
Loss of Taste/Smell	40.2 (90)	41.7 (40)	34.4 (166)	35.9 (74)

Clinical, Biological, and CT Features of COVID-19 Patients by Oxygen Saturation Levels This table presents the clinical characteristics, comorbidity profiles, symptomatology, and CT imaging features of COVID-19 patients, stratified by oxygen saturation levels (below 90% and at or above 90%) and further categorized into training and validation cohorts. Continuous variables are expressed as mean ± standard deviation, while categorical variables are presented as percentages with the number of patients affected.

Outperformance of Linear ML Classifiers in Clinical and Laboratory Models

The performance of the ML classifiers in predicting oxygen saturation outcomes (below or above 90%) in COVID-19 patients was assessed, with the validation AUC values and training folds range detailed in Table 2. The Clinical Model's top performer was Logistic Regression, achieving a validation AUC of 0.80, with a training folds range of 0.80–0.85. This tight range indicates that Logistic Regression consistently performed well during training, demonstrating strong linear relationships with the clinical data. Linear SVM showed a validation AUC of 0.80 in the Clinical Model, with a training range of 0.77–0.83. This result suggests that Linear SVM is slightly less consistent than Logistic Regression but still a strong performer among linear classifiers. In the Laboratory Model, Linear SVM outperformed other classifiers with a validation AUC of 0.82 and a training range of 0.80–0.84, suggesting a strong linear association with laboratory-based features. Logistic Regression had a validation AUC of 0.81 and a training range of 0.78–0.94, indicating slightly lower consistency compared to Linear SVM.

Table 2
Classifier	Clinical Model	Laboratory Model	CT-Based Model	Integrated Model
Linear Classifiers
Logistic Regression	0.82 (0.80–0.85)	0.81 (0.78–0.94)	0.76 (0.78–0.85)	0.84 (0.83–0.87)
Linear SVM	0.80 (0.77–0.83)	0.82 (0.80–0.84)	0.71 (0.63–0.88)	0.78 (0.76–0.90)
Naive Bayes	0.76 (0.72–0.78)	0.74 (0.71–0.76)	0.79 (0.66–0.82)	0.81 (0.78–0.83)
Non-Linear Classifiers
SVM (RBF Kernel)	0.75 (0.72–0.78)	0.76 (0.74–0.79)	0.85 (0.86–0.91)	0.89 (0.92–0.97)
Random Forest	0.78 (0.74–0.81)	0.79 (0.76–0.82)	0.87 (0.78–0.93)	0.86 (0.81–0.96)
XGBoost	0.77 (0.63–0.80)	0.78 (0.71–0.81)	0.81(0.77–0.92)	0.85 (0.80–0.95)

Classifier Performance for Oxygen Saturation Prediction: Validation AUC and Training Folds Range The table contains validation AUC values and the range of AUC from 10-fold cross-validation for each model type: Clinical, Laboratory, CT-Based, and Integrated. Classifiers are grouped into linear and non-linear categories. Tighter cross-validation ranges indicate greater consistency across training folds, suggesting a more reliable and generalizable model.

Non-Linear Classifiers excelled in CT-Based and Integrated Models

In the CT-Based Model, Random Forest demonstrated the highest validation AUC of 0.87, with a training range of 0.78–0.93. This result indicates that Random Forest is highly effective in handling complex CT-based data with varying patterns. SVM with RBF Kernel showed a validation AUC of 0.85 with a training range of 0.86–0.91, suggesting a slightly lower consistency compared to Random Forest but still robust in capturing non-linear relationships in CT data. In the Integrated Model, SVM with RBF Kernel achieved the highest validation AUC of 0.89 and a training range of 0.92–0.97, demonstrating its ability to manage complex interactions among clinical, laboratory, and CT-based features. Random Forest showed a validation AUC of 0.86 with a training range of 0.81–0.96, indicating robust performance in integrated scenarios. XGBoost, with a validation AUC of 0.85 and a training range of 0.80–0.95, was also effective but slightly less consistent than SVM with RBF Kernel. Figure 2 illustrates the SVM-RBF decision boundaries in a 2D space, derived from the first two principal components, revealing distinct patterns of separability in the clinical, laboratory, CT-based, and integrated models.

Key Features for Oxygen Saturation Prediction

The Clinical Model's Logistic Regression classifier achieved an AUC of 0.82, with Age emerging as the most important predictor of oxygen saturation in COVID-19 patients (Table 3). It had a feature importance of 0.51 and a stability of 0.89. Gender followed with an importance of 0.33 and a stability of 0.81. Fever, with an importance of 0.31 and stability of 0.73, also contributed significantly, highlighting the role of clinical symptoms in oxygen saturation prediction. In the Laboratory Model, Linear SVM (AUC 0.82) identified White Blood Cell (WBC) count as the most significant predictor, with an importance of 0.53 and stability of 0.88. The Lymphocyte count, with an importance of 0.35 and stability of 0.83. Platelet Count, with an importance of 0.32 and stability of 0.80, indicates the potential link between coagulation and respiratory outcomes in COVID-19.

For the CT-Based Model, Random Forest achieved an AUC of 0.87, with Mean Lesion Volume showing a high feature importance of 0.24 and stability of 0.90. Lower Zone Predominance achieved an importance of 0.20 and stability of 0.85, and Non-Lesion Lung Volume (NLLV) Skewness, an importance of 0.16 and stability of 0.80. In the Integrated Model, the SVM with RBF Kernel (AUC 0.89) led the way with WBC as the most significant predictor, having an importance of 0.31 and stability of 0.88. The Mean NLLV followed with an importance of 0.30 and stability of 0.85, reinforcing the importance of CT-based lung volume metrics. Crazy Paving, with an importance of 0.22 and stability of 0.72, highlights the role of specific CT patterns in the model's predictive accuracy.

Table 3
Model Type	Best Classifier (AUC)	Top Feature 1 (Importance, Stability)	Top Feature 2 (Importance, Stability)	Top Feature 3 (Importance, Stability)
Clinical Model	Logistic Regression (0.82)	Age (0.51, 0.89)	Gender (0.33, 0.81)	Fever (0.31, 0.73)
Laboratory Model	Linear SVM (0.82)	WBC (0.53, 0.88)	Lymphocyte (0.35, 0.83)	Platelet Count (0.32, 0.80)
CT-Based Model	Random Forest (0.87)	Mean LV (0.24, 0.90)	Lower Zone Predominance (0.20, 0.85)	NLLV skewness (0.16, 0.80)
Integrated Model	SVM_RBF (0.89)	WBC (0.31, 0.88)	Mean NLLV (0.30, 0.85)	Crazy paving (0.22, 0.72)

Top Features for Predicting Oxygen Saturation in COVID-19 Patients This table displays the top features for predicting oxygen saturation in each model type, based on the classifier with the highest AUC in the validation dataset. The feature importance values are normalized, reflecting the relative significance of each feature within its respective model, while feature stability measures the consistency of importance across subsampling runs.

Our study explored the comparative performance of machine learning classifiers across four model types, focusing on the top-performing classifiers and their key features for predicting binary oxygen saturation outcomes in COVID-19 patients, to guide resource allocation in healthcare settings, such as deciding when to admit patients to intensive care or administer high-flow oxygen therapy. The models incorporated a diverse set of features, including clinical, laboratory, and CT-Based, and Integrated data to offer a comprehensive understanding of the outcomes. The best-performing classifiers for each model align with the underlying patterns of the data, reflecting the linearity or non-linearity of the feature sets. The feature importance values and stability metrics provide insights into the robustness and reliability of each model.

The ability to predict oxygen saturation levels in COVID-19 patients is crucial for assessing disease severity and guiding clinical decisions. In the Clinical Model, where the Logistic Regression classifier achieved an AUC of 0.82, age emerged as the most significant predictor, with a feature importance of 0.51 and a stability of 0.89. This finding underscores the well-documented correlation between advanced age and severe respiratory distress in COVID-19. Gender, with a feature importance of 0.33 and stability of 0.81, indicates possible gender-related differences in disease progression. Fever, a common symptom of COVID-19, also contributed significantly to the model, suggesting that clinical symptoms play a vital role in predicting oxygen saturation (2, 3).

The Laboratory Model, with an AUC of 0.82 for Linear SVM, identified WBC count as the primary predictor. This strong importance points to the role of the immune response in the progression of COVID-19. The Lymphocyte count, with an importance of 0.35 and stability of 0.83, further supports the idea that immune system markers are critical in understanding disease severity. Platelet count, with an importance of 0.32 and stability of 0.80, suggests that coagulation factors may also have a role in predicting oxygen saturation outcomes, emphasizing the broader systemic impact of COVID-19. The CT-Based Model, where the Random Forest classifier achieved an AUC of 0.87, brought attention to the radiological features of COVID-19. Mean Lesion Volume was the top predictor, highlighting the significance of lung lesion volume in assessing disease severity. Lower Zone Predominance and NLLV Skewness suggest that spatial distribution and volume consistency of lung tissue are essential factors in determining oxygen saturation(22).

Finally, the Integrated Model, which combined clinical, laboratory, and CT-based features, demonstrated a broader range of significant predictors. The SVM with RBF Kernel achieved an AUC of 0.89, with WBC count and Mean NLLV as the leading predictors, suggesting that combining immune response markers with radiological data provides a more comprehensive view of disease severity. Crazy Paving, a specific CT pattern, further contributes to the predictive power of the integrated approach. The integration of these diverse features emphasizes the critical role of radiology and underscores the need for ongoing research to improve predictive accuracy and clinical outcomes(2, 3, 22).

A limitation of this study is its retrospective design, which may inherently carry biases due to reliance on existing hospital records. The inclusion of only admitted patients with confirmed COVID-19 could lead to selection bias, potentially excluding milder cases not requiring hospitalization. The study's cohort focused on a single hospital, which may not represent broader demographic or regional variations. Data standardization techniques like z-score and one-hot encoding may also introduce inconsistencies in the processed data, affecting model robustness. Finally, the CT-based features, while comprehensive, may not capture all relevant variables contributing to disease progression. The reliance on specific ML classifiers, though effective, could be restricted by their inherent assumptions and limitations, impacting the broader applicability of the findings.

Our analytical framework highlights the strengths and limitations of various classifiers across different models, emphasizing the underlying linearity or non-linearity in their feature sets. The study contributes to the field of COVID-19 research by demonstrating the importance of CT scans in assessing disease severity and predicting patient outcomes. The findings are expected to guide clinical decision-making, such as ICU admissions and the need for high-flow oxygen therapy. Additionally, the study highlights the potential of machine learning models to integrate various data types, leading to more accurate severity assessments and enhanced patient care.

These insights provide a detailed comparative analysis that guides the selection of the most appropriate classifiers for predicting oxygen saturation outcomes in COVID-19 patients. The intertwined and multi-level approach to the discussion underscores the importance of understanding the unique characteristics of each model type and the complex interactions among various features in determining the best-performing classifiers. The results may inform future research directions, focusing on developing quantitative analysis tools for CT scans and integrating them with clinical algorithms for improved predictive accuracy and reproducibility.

Ethics approval and consent to participate

This study received formal approval from the Institutional Review Board at Baqiyatallah University of Medical Sciences, Tehran, Iran (IR.BMSU.BAQ.REC.1400.079).

Competing interests

The authors declare that they have no competing interests

Funding

none

Authors' contributions

M.Shahidzade conceptualized the study and played a pivotal role in image analysis, data interpretation, and figure preparation. M.Shahidzade and N.J.Jafari were responsible for drafting the manuscript. F.Salmanizadegan contributed to figure preparation and provided critical edits to the manuscript. M.Sabouri conducted image and data analysis. M.Yargholi, O.Teymouri and Z.Mollaahmadipour collected and interpreted patient data. M.Sabouri provided technical support and offered conceptual guidance. All authors critically reviewed, revised and approved the final manuscript for publication.

Acknowledgements

none

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Kang J, Kang J, Seo WJ, Park SH, Kang HK, Park HK et al (2023) Prediction models for respiratory outcomes in patients with COVID-19: integration of quantitative computed tomography parameters, demographics, and laboratory features. J Thorac Dis 15(3):1506–1516
Salahshour F, Mehrabinejad MM, Nassiri Toosi M, Gity M, Ghanaati H, Shakiba M et al (2021) Clinical and chest CT features as a predictive tool for COVID-19 clinical progress: introducing a novel semi-quantitative scoring system. Eur Radiol 31(7):5178–5188
Metwally M, Basha M, Zaitoun MMA, Abdalla H, Nofal H, Hendawy H et al (2021) Clinical and radiological imaging as prognostic predictors in COVID-19 patients. Egypt J Radiol Nuclear Med. ;52
Prakash J, Kumar N, Saran K, Yadav AK, Kumar A, Bhattacharya PK et al (2023) Computed tomography severity score as a predictor of disease severity and mortality in COVID-19 patients: A systematic review and meta-analysis. J Med Imaging Radiat Sci 54(2):364–375
Yanamandra U, Shobhit S, Paul D, Aggarwal B, Kaur P, Duhan G et al (2022) Relationship of Computed Tomography Severity Score With Patient Characteristics and Survival in Hypoxemic COVID-19 Patients. Cureus 14(3):e22847
Varghese BA, Shin H, Desai B, Gholamrezanezhad A, Lei X, Perkins M et al (2021) Predicting clinical outcomes in COVID-19 using radiomics on chest radiographs. Br J Radiol 94(1126):20210221
Hu Z, Yang Z, Lafata KJ, Yin FF, Wang C (2022) A radiomics-boosted deep-learning model for COVID-19 and non-COVID-19 pneumonia classification using chest x-ray images. Med Phys 49(5):3213–3222
Sun Y, Salerno S, He X, Pan Z, Yang E, Sujimongkol C et al (2023) Use of machine learning to assess the prognostic utility of radiomic features for in-hospital COVID-19 mortality. Sci Rep 13(1):7318
Zhang Y, Li G, Bian W, Bai Y, He S, Liu Y et al (2022) Value of genomics- and radiomics-based machine learning models in the identification of breast cancer molecular subtypes: a systematic review and meta-analysis. Ann Transl Med 10(24):1394
Arika RN, Mindila A, Cheruiyo W (2022) Machine Learning Algorithms for Breast Cancer Diagnosis: Challenges, Prospects and Future Research Directions. J Oncol Res. ;5(1)
Kumar V, Gu Y, Basu S, Berglund A, Eschrich SA, Schabath MB et al (2012) Radiomics: the process and the challenges. Magn Reson Imaging 30(9):1234–1248
Hawkins DM (2004) The problem of overfitting. J Chem Inf Comput Sci 44(1):1–12
Guyon I, Elisseeff A (2003) An Introduction of Variable and Feature Selection. J Mach Learn Res Special Issue Variable Feature Selection 3:1157–1182
Saeys Y, Inza I, Larrañaga P (2007) A review of feature selection techniques in bioinformatics. Bioinformatics 23(19):2507–2517
Breiman L (2001) Random Forests. Mach Learn 45(1):5–32
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297
Freund Y, Schapire RE (1997) A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. J Comput Syst Sci 55(1):119–139
Hastie T, Tibshirani R, Friedman J (2009) Kernel Smoothing Methods. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer New York, New York, NY, pp 191–218
Hastie T, Tibshirani R, Friedman J (2009) Overview of Supervised Learning. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer New York, New York, NY, pp 9–41
Hastie T, Tibshirani R, Friedman J (2009) High-Dimensional Problems: p N. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer New York, New York, NY, pp 649–698
Rish I (2001) An Empirical Study of the Naïve Bayes Classifier. IJCAI 2001 Work Empir Methods Artif Intell. ;3
Li D, Zhang Q, Tan Y, Feng X, Yue Y, Bai Y et al (2020) Prediction of COVID-19 Severity Using Chest Computed Tomography and Laboratory Measurements: Evaluation Using a Machine Learning Approach. JMIR Med Inf 8(11):e21604

The authors declare no competing interests.

Download PDF

Version 1

posted

You are reading this latest preprint version

Assessing Machine Learning classifiers in COVID-19: The Role of Clinical, Laboratory, and Radiological Features in Predicting Oxygen Saturation

Status:

Version 1

Abstract

Background

Objective

Method

Results

Conclusion

Figures

Introduction

Methods

Study Design

Data collection

CT Image Acquisition

Image Segmentation and Processing

CT-based features

Data integration

Feature Selection

ML Classifiers

Validation Strategy

Feature Importance and Stability

Unsupervised Learning

Configurations of the Generative Pre-trained Transformer (GPT)

Results

Outperformance of Linear ML Classifiers in Clinical and Laboratory Models

Non-Linear Classifiers excelled in CT-Based and Integrated Models

Key Features for Oxygen Saturation Prediction

Discussion

Conclusion

Declarations

Ethics approval and consent to participate

Competing interests

Funding

Authors' contributions

Acknowledgements

Availability of data and materials

References

Additional Declarations

Status:

Version 1