An Intelligent System for Prediction of Severity of SARS-Cov-2 Infection and Progression to Critical Illness: Using Machine Learning Models

Introduction: The rapid worldwide outbreak of coronavirus disease 2019 (COVID-19) has posed serious and extraordinary challenges to healthcare industries in predicting disease behavior, and outcomes. Aim: This study aimed to develop a Clinical Decision Support System (CDSS) for predicting the severity of SARS-CoV-2 infection and progression to critical illness in a patient with COVID-19 using several machine learning algorithms. Material and Methods: Using a two-center registry, the data of 2482 COVID-19 patients from February 9, 2020, to December 20, 2020, were reviewed. The Relief Feature Selection (RFS) algorithm was used for optimizing the input variables. Then, selected variables feed into ML models including XGBoost, HistGradient Boosting (HGB), Random Forest (RF), and Naïve Bayesian (NB) to construct prediction models. Afterwards, the performance of each combination was compared using some evaluation metrics. Eventually using the best ML model performance, a Clinical Decision Support System (CDSS) was implemented with C# programming language. Results: of the 63 included variables, 15 features were identi�ed as the most important predictors. The experimental results indicated that the HGB classi�er with an average classi�cation accuracy of 94.2%, mean speci�city of 92.4%, mean sensitivity of 91%, mean F-score of 87.2 %, and �nally mean AUC of 87.3 % was selected as the most appropriate machine learning model for predicting the Severity of SARS-CoV-2. Conclusion: The results of this study showed that the hybrid ML algorithms and in particular the RFS-HGB (by optimizing input variables and customizing the structure of the algorithms (can help the frontline clinicians to predict the severity of COVID-19 progression.


Introduction
The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an ongoing global pandemic with over 208,795,446 cases and more than 4,386,181 deaths worldwide as of August 17, 2021 (1,2).The coronavirus disease 2019 (COVID-19) has created signi cantly impact and tension on global healthcare systems and continues to be a main worldwide concern as the number of patients with COVID-19 and death rates are still quickly rising (3,4).The clinical manifestation spectrum of COVID-19 patients ranges from asymptomatic to mild disease or severe viral pneumonia with acute respiratory failure, multi-organ failure, shock and even death (5)(6)(7).A large proportion of people infected with the virus, around 40-45%, experienced little or no symptoms(8), and severe illness or ICU admission occurs between ve and approximately 30% of hospitalized patients (9).Older people and those with underlying medical problems such as cardiovascular disease, diabetes, chronic respiratory disease, and cancer are more likely to develop serious illnesses (6, 9, 10).
The high mortality rate due to COVID-19, insu cient drugs and hospital beds, low speed of vaccination, and overtiredness of medical staffs have enforced the world to consider COVID-19 pandemic management as a critical priority.These challenges are more prominent in the developing and lowincome countries in which there are not enough hospital facilities (i.e., ICU beds, oxygen generators, medical providers and etc.) for e cient management of the disease.
In this scenario, recognition of the key predictive variables of COVID-19 severity and prognosis the patient's status into one of the four classes (need to home quarantine, ward admission, ICU admission, or expired) is highly important.It helps health care providers to have more e cient decision making, better resource allocation, and also better patient triage and treatment planning.Di culties for the identi cation of key predictor variables of COVID-19 outcomes and patient infection severity categorization are consequences of COVID-19 complexity.Machine learning(ML) models can be helpful by analyzing large sets of data to discover patterns rapidly, identifying the important predictors through feature selection, and providing techniques that can help predict the severity category of patients involved with SARS-CoV-2 infection (10,11).
To date, there have been several efforts to apply ML algorithms for f forecasting the prognosis of patients infected with SARS-CoV-2, and also for automatic diagnosis and classifying SARS-CoV-2 infected patients (5,10,12).However, most of these studies included a small size of data sets with a low quantity of cases in each outcome class that enforced them to use binary classi ers instead of having multiple classes.For instance, because of limited number of patients with mild and critical types of SARS-CoV-2, several studies used only two strict classes (non-severe and severe), instead of four classes types (i.e., mild, common, severe, and critical) (13).
Therefore, because of the importance of exact classifying of a patient in the multiple classes for assignment of health care resources, utilization of ICU facilities and early and active treatment of highrisk cases with a high risk of involving with severe covid-19, it is essential to develop an intelligence system to address these issues.
In the present study, we aimed to use a modi ed several ML model on large data set to identify the most important predictor factors of Severity of SARS-CoV-2 Infection and also, to develop an intelligent clinical decision support system for prediction of the severity of SARS-CoV-2 infection into one of the four different categories, including home quarantine, ward admission, ICU admission and expired cases status by using ML models.

Study roadmap, and experiment environment
This is a retrospective and two-center study that was conducted in 2021 for proposing a CDSS to predict the severity of COVID-19 and progression to critical illness in a patient with Covid-19.The results for several ML models for covid-19 severity prediction based on the selected variables were tested.A Feature selection algorithm was used to choose the most important variables, and on these selected variables, the performance of the prediction algorithms was tested.The four ML Algorithms including XGBoost, HistGradient Boosting (HGB), Random Forest (RF) ,and Naïve Bayesian (NB) to construct prediction models were used in the system.The algorithms validation and performance assessment criteria were calculated.The road map of the developed system consists of four stages 1-data preprocessing, 2feature selection and cross-validation 3-prediction models, 4-Performance assessment metrics.The road map of the proposed system for Prediction of Severity of SARS-CoV-2 Infection is displayed in Figure 1.
All of the experiments exerted on the algorithms (preprocessing, training, and performance evaluation) were conducted using Python programming language (version 3.7.7).And, the software User Interface was implemented using the C# programming language.

Data set description
The dataset used to conduct this research is obtained from the database registry in Imam Khomeini hospital, a liated to Ilam University of Medical Sciences )which is the main center for delivery COVID-19 specialized care and treatments in West of Iran( and Baqiyatallah teaching hospital in north of Tehran.The Baqiyatallah University of Medical Science ethics board approved the study design (Ethics code: IR.MEDILAM.REC.1399.294).The time frame of the data collection was from February 9, 2020, to December 20, 2020.A total of 14392 suspected people with SARS-CoV-2 have been referred to these two centers.Of those, according to RT-PCR results, 7214 cases were introduced as positive for SARS-CoV-2.
Only con rmed RT-PCR patients with COVID-19 were involved in the current study (see Figure1).63 features in ve demographical, clinical manifestation, high-risk groups and laboratory ndings classes are correlated in the feature selection phase (input variables).Besides the dependent variable (output class) is the severity of COVID-19 patients in four home quarantine, general hospitalization, ICU hospitalization ,and expired subclasses (see Table1).

Data Pre-processing
The preprocessing stage is essential for the effective use of data and predicting algorithms which should be trained and tested in an e cient method.Several Preprocessing methods such as treat of missing values, standard scale ,and minimum or maximum scalar were used to effective use of data to predict the disease severity in patients with COVID-19.The standard scalar guarantees that every characteristic has the mean zero and variance has the mean one, bringing all features to the same Coe cient.Likewise, in Minimum and Maximum Scalar transfers the values such that all attributes are between 0 and 1, and also rows with missing values (greater than 70%) were removed.Noisy and abnormal values, errors, duplicates, and meaningless data were checked by two authors (MO and MSH) in collaboration with two infectious disease specialists.

Patient selection criteria
The data on 3750 eligible patients were extracted from the Imam Khomeini hospital registry database and baghiyatollah teaching hospital.Then, 1268 incomplete case records which had a lot of missing data (more than 70%) were excluded from the analysis.Also, the missing values were imputed with the mean or mode of each variable.After applying exclusion criteria, ultimately the 2482 records were remained (Figure 2).

Feature Selection
Feature selection is necessary before applying the machine learning algorithms because usually irrelevant features affect the prediction models' performance.Feature selection improves the prediction accuracy and reduces the models' run time, and its aim is to choose the signi cant variables and thereby dropping the model's computational intricacy.For feature selection in our clinical decision support system, we used a Relief Feature Selection Algorithm to select the most important features in the prediction of severity of covid-19.Relief is a feature selection algorithm, which assigns weights to all the features in the dataset and these weights can be updated with the passage of time.

Model development
To predict the severity of patients with covid-19, several ML prediction techniques including HistGradient Boosting Classi er, XGBClassi er, Naïve Bayesian Classi er, Random Forest algorithm were used in this study.

Model evaluation
In this study, to compare the results of several ML models in the prediction of severity of covid 19, we applied 10-fold cross-validation and six performance evaluation metrics.Accuracy, speci city, sensitivity, KAPA, error rate, and area under receiver operator characteristics (AUC-ROC) curve was measured for comparing the performance of the classi ers (Equations 1 to 4).In addition, to have a better comparison in terms of the algorithms' performance, we assessed the effectiveness of selected ML algorithms in terms of time to build the model, Kappa statistic, Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), Relative Absolute Error (RAE), and Root Relative Squared Error (RRSE).

Characteristics of patients
After applying the exclusion criteria, nally the number of 2482 patients met eligibilities.Of these patients, 1312 (52.86%) were male and 1170 (47.14%) were women and the median age of the participants was 57.25 (interquartile 18-100

K-Fold Cross-Validation
Selected features by the Relief Feature Selection algorithm were tested by six ML models with 10 foldcross-validation.In a 10-fold cross validation method 90% of covid 19 dataset was used for training ML models and 10% was used for testing algorithms.To better evaluation of the actual performance of ML models with a 95% con dence interval, the average for included metrics of 10-fold cross validation was measured.Table 5 and Table 6 show the 10-fold cross-validation results of six prediction algorithms based on selected feature for prediction of the severity in a patient with covid.
For better presentation of the results, the confusion matrix of the best ML model (in terms of the highest assessment metrics and the highest AUC rate) is displayed in Figure3.We also measured the error rate of classi er in terms of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Root Relative Squared Error (RRSE) %( Table 7).Time to build a model (s) and Kappa Statistic (KS) metrics, also have been obtained for all six classi ers (Table7).Generally, according to the obtained results we choosed the HGB classi er as the best algorithm and used it for developing our Clinical Decision Support System (CDSS) interface for prediction of severity of SARS-CoV-2 and progression to critical illness.The confusion matrix of the HGB classi er as the bestdeveloped ML model (in terms of the highest assessment metrics) in this study is displayed in Figure 3.

System implementation
Using the implemented HGB classi er, we designed and implemented a CDSS interface during June and July 2021.The CDSS consisted of two types of implementation codes namely as: codes for the user interface implementation, and codes for the logic layer implementation (programming rule of CDSS according to HGB algorithm).The user interface in our study comprised of two pages: Welcome page (sign up and log in page), and CDSS modules.Two screenshots of implemented system are shown in gures 4 and 5.

Discussion
Because of the Importance of early recognition and treatment of cases with a high risk of involving with severe covid-19, there is substantial interest in examining the associated risk factors among researchers.
For instance, a literature review done by Gallo Marin et.al.(2021) showed that variables such as pre-existing comorbidities, computed tomography ndings, hypoxia, some laboratory test results, and endorgan dysfunction have an important role in predicting the severity of Covid-19 (14).Boot et all in a systematic review and meta-analysis study demonstrated that factors such as some of the patient characteristics, comorbidities, in addition to vital measurements and some real-time symptoms are valuable to predict severe illness and mortality due to Covid-19 (15).
The present study has some limitations.Although our early ndings suggest that this system could be effective in early prediction and classi cation of patients into the four de ned classes, the system performance should also be evaluated based on external data (Other organizations' data).Because of di culties in data sharing between health care organizations, the system is not validated with an external data set.In addition, we had a selection bias; we needed blood test results for data modeling, therefore only patients already admitted to the hospital were included in the present study.

Conclusion
The present study indicates that arti cial intelligence-based systems are powerful tools to prognoses the severity of Covid-19 based on the predictor variables.We have identi ed some demographic data, signs and symptoms, and routine laboratory variables which can contribute to having a sensitive prognosis for patients with COVID-19.The proposed prediction model (Hist Gradient Boosting Classi er) can predict the severity rate for each case with an accuracy of 94%.As a result of such data modeling studies, better disease prognosis, better resource allocation, and improving patients' outcomes would be more accessible especially in developing countries, where resources such as ICU beds are limited.
The road map of the proposed system for Prediction of Severity of SARS-CoV-2 Infection Error rate of classi ers on the given dataset  The welcome page of the developed CDSS interface

Figure 2 Flowchart describing patient selection Figure 3
Figure 2

Figure 4 Multi
Figure 4

Table 1
Primary list of variables of the selected dataset associated with COVID-19 severity prediction

Table 2 .
The descriptive statistics of qualitative variables of the study

Table 3 .
The descriptive statistics of quantitative variables of the study The most important predictors and their rank for prediction of severity in a patient with COVID-19 are shown in Table4.Based on the results, the most important variables for the prediction of severity in a patient with COVID-19 are age, dyspnea, admission saturation, C-reactive protein respectively.

Table 5
Average evaluation metrics obtained from 10 runs of ML Models Based on results in Table3 and 4, the HGB classi er obtained the best performance on the test dataset compared with the other three models with a mean classi cation accuracy of 94.2%, mean speci city of 92.4%, mean sensitivity of 91%, mean F-score of 87.2%, and mean ROC of 87.3%. the result obtained for classi ers regarding the time to run the ml algorithms, Kappa Statistic (KS), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Root Relative Squared Error (RRSE) are demonstrated in Table7and gure 3.

Table 7
The required time to build, training and simulation error of selected classi ers It is apparent from Table5that the XGB algorithm takes about 38 (s) to build its model as the fastest model, whereas the HGB Classi er takes about 63 (s) that was the second fastest classi er to build the prediction model.Additionally, the RF model, takes about 83 (s) to build and the NB algorithm takes about 117 (s).The error rate of classi ers on the given dataset is shown in gure3.Based on the results obtained for KAPPA metrics and error rate of classi er, XGB and HGB have the highest rates of KAPPA metrics (91% and 92.30% respectively) in comparison to two the other developed classi ers.The XGB algorithm and HGB classi er, also have the lowest rates of Mean Absolute Error (0.03).The best model in our study (HGB classi er) has 12% Root Mean Square Error and 23% Root Relative Squared Error.
(15)17)sent study, we used a Relief Feature Selection Algorithm to identify each predictor variable's role Being able to predict the severity of Covid-19 based on easily accessible data (demographic data, signs and symptoms, and routine laboratory tests) is one of the advantages of the present study.Some other studies approved the predictive values of some of the variables identi ed in our study, for instance: Dyspnea (16), demographical variables like high age(15-  17)and gender(15,17), comorbidities including diabetes and hypertension (16), and Oxygen saturation(15).In the present study, the Hist Gradient Boosting (HGB) has the best performance among the four evaluated algorithms.The classi er can classify every new case to one of the four classes with an accuracy of 94% that is very considerable in the full of uncertainty covid-19 world.Therefore, the software has been developed based on the HGB classi er.In addition to mining laboratory data, some other researchers have been successful to predict Covid-19 severity based on the patients' medical images (18-20).Because of the diversity in Covid-19 data sets, comparing the proposed model in the present study with other researchers' studies is not possible logically.But, one considerable point is that we used a relatively large data set compared to many existing COVID-19 studies.Besides, good distribution of the data in different classes helped us to classify our patients into four different categories, including Home quarantine, Ward Admission, ICU Admission and Expired cases.While, some other researchers have mentioned that small data sets with a low quantity of cases in each outcome class enforced them to use binary classi ers instead of having multiple classes; For instance, because of limited numbers of patients with mild and critical types of COVID-19, Tang et al. (2021) used two classes (non-severe and severe), instead of four classes (i.e., mild, common, severe and critical)