Automated Systems for Detection of COVID-19 Using Chest X-ray Images and Lightweight Convolutional Neural Networks

Since December 2019, the appearance of an outbreak of a novel coronavirus disease namely COVID-19 and which is previously known as 2019-nCoV. COVID-19 is a type of coronavirus that leads to the general destruction of respiratory systems and a severe respiratory symptom which are associated with highly Intensive Care Unit (ICU) admissions and death. Like any disease, the early diagnosis of coronavirus leads to limit its wide-spreading and increases the recovery rates of patients. The gold standard of COVID-19 detection is the real-time reverse transcription-polymerase chain reaction (RT-PCR) which has been used by the clinician to discover the presence or absence of this type of virus. The clinicians report that this technique has a low positive rate in the early stage of this disease. Based on this, the clinicians were forced to use another way to help in the early diagnosis of COVID-2019. So, the clinician's attention moved towards the medical imaging modalities especially the computed Tomography (CT) and X-ray chest images. Both modalities show that there is a change in the lungs in the case of COVID-19 that is different from any other type of pneumonic disease. Therefore, this research targeted toward employing different Artificial Intelligence (AI) techniques to propose a system for early detection of COVID-19 using chest X-ray images. These images are classified using different AI algorithms and a combination of them, then their performance was evaluated to recognize the best of them. These algorithms include a convolutional neural network (CNN), Softmax, support vector machine (SVM), Random Forest, and K nearest neighbor (KNN). Here CNN is into two scenarios, the first one to classify the X-ray images using a softmax classifier, and the second one to extract automated features from the images and pass these features to other classifiers (SVM, RFF, and KNN). According to the results, the performance of all classifiers is good and most of them record accuracy, sensitivity, specificity, and precision of more than 98%.


Introduction
COVID 2019 is the most recent type of coronavirus that emerges in Wuhan, Hubei Province. It was found in cases of pneumonia. This form of viral pneumonia spreads both inside and outside of Wuhan.
This sharp rise in the number of infected cases worldwide leads the World Health Organization called it 2019-nCoV [1]. The name of this novel coronavirus that caused extreme acute respiratory syndrome is SARS-CoV-2 [2]. Feng Pan et al. used CT in the chest to determine the extent of COVID-19 pneumonic lung involvement, finding that COVID-19 CT-chest recovery patients displayed extent after 10 days after initial symptoms [3]. This new form of the crown virus has been discovered and identified in the cases of viral pneumonia that occurred in Wuhan, called COVID-19 by the World Health Organization (WHO). It spreads rapidly inside and outside the province of Hubei and even other countries, leading to sharp rises and widespread panic among people. In their paper, Rabi, Firas A., et al [4] presented a description of all the existing issues surrounding the novel coronavirus and the disease it causes as the medical point of view.
The most effective approaches to monitor the spread of coronavirus are quadrantile and adequate care for suspected cases. The pathogenic laboratory test is used as a diagnostic device for suspected cases of coronavirus, but it is time-consuming with the false positives and low supply. In the early stage detection of COVID-19, some patients may have positive pulmonary imaging manifestations, but may they have no sputum findings and they have negative test results in swabs of RT-PCR. Such cases like those are not diagnosed as suspected or confirmed [5,6]. Although  in about 82% of cases causes milder symptoms, the others are serious or critical [7]. The total number of coronavirus cases is approximately 335,403, of which 14,611 died, and 97,636 were recovered. The number of patients infected is 223.156. Although the disease marginally survives 95% of the number of infected patients, 5% of the remainder have a severe or critical illness [8]. Infection signs include symptoms of coughing, nausea, cough, and dyspnea. The infection can cause pneumonia, extreme acute respiratory syndrome, septic shock, multi-organ failure and death in more serious cases [7]. It was determined that men are more infected than women and that children between 0-9 years of age do not die. Respiratory levels of COVID-19 pneumonia cases are higher compared with healthy individuals [9].
It has been reported that to prevent the spread of COVID-19 disease, successful patient screening and prompt medical response for the infected patients is a crying requirement. The Reverse Transcription Polymerase Chain Reaction (RT-PCR) test on respiratory specimens is the gold standard screening tool used to test the COVID-19 patients [10]. This technique is the most widely used test method for COVID-19 detection, but it is a manual, complex, laborious and time-consuming procedure with a positive result of just 63% [10]. There is a severe shortage of its stock leading to delay in efforts to prevent disease [11]. Several countries in the world suffer from inaccurate detection or count of the number of COVID-19 patients, which is due not to all patients being tested, but also to delays in the test results [12]. These delays may cause infected patients to communicate with and infect healthy patients in the process. It is reported that the RT-PCR kit costs approximately USD 120-USD 130 and also requires a specialized biosafety laboratory to house the PCR system, each of which can cost USD 15,000 to USD 90,000 [13]. These costly and delayed test results allow the disease to spread and make the situation much worse. Not only is this an issue for low-income nations, but some developed countries are still struggling to tackle it [14]. COVID-19's other diagnostic methods may be clinical symptom analysis, epidemiological history and positive radiographic images (CT / Chest radiograph (CXR)), as well as positive pathogenic monitoring. The clinical features of extreme COVID-19 infection are that of bronchopneumonia that induces fever, cough, dyspnea, and acute respiratory distress syndrome (ARDS) failure [16][17][18][19].
The big testing method for COVID-19 is readily available, including radiological imaging. Most COVID-19 cases have common characteristics on radiographic images including early-stage bilateral, multifocal, ground-glass opacities with a peripheral or posterior distribution, primarily in the lower lobes, and late-stage pulmonary consolidation [19][20][21][22][23][24][25]. Although typical CXR images that help to screen suspected cases early, the images of various viral types of pneumonia are similar and overlap with other infectious and inflammatory diseases of the lungs. Radiologists also have trouble distinguishing COVID-19 from other viral pneumonia. The signs of COVID-19 being close to viral pneumonia can also lead to an erroneous diagnosis in the present scenario, where hospitals are crowded and run round the clock. Consequently, an incorrect diagnosis may result in Non-COVID viral pneumonia being falsely labeled as highly suspicious of COVID-19 and thus delaying care with consequent exposure costs, effort, and risk to positive COVID-19 patients. Many biomedical problems (e.g., brain tumor detection, breast cancer detection, etc.) currently use approaches based on Artificial Intelligence (AI) especially Deep Learning (DL) can help in the detection of COVID-19 and extract automated features from these images [26][27][28][29].
Deep learning techniques can reveal image features, which are not apparent in the original images.
Specifically, Convolutional Neural Network (CNN) has been proven extremely beneficial in feature extraction and learning and therefore widely adopted by the research community. CNN was used to enhance image quality in low-light images from a high-speed video endoscopy [30] and was also applied to identify the nature of pulmonary nodules via CT images [31]. Also, CNN was used to electrocardiogram (ECG) classification [32], melanoma skin cancer detection [33], optical coherence tomography (OCT) images classification [34] and brain tumor classification [35]. Deep machine learning techniques on the X-Rays chest are becoming popular because they can be easily used with low-cost imaging techniques and there is plenty of data available to train various machine learning models.
Xu, Xiaowei, et. al. [5] used deep learning algorithms for 618 subjects in the classification of CT images, in three categories: COVID -19, influenza A and healthy people. In all CT cases, they obtained 86.7 % accuracy. Wang, Shuai, et. al. [6] have used profound learning methods to remove graphic features from CT images to provide pre-diagnosis to the clinician before the pathogenic examination.
They reached 89.5% accuracy and 87% sensitivity, but when applied to the external dataset, the accuracy of their algorithm was 79.3%.
Abdullah Ahmed [36] proposed a method for detecting coronavirus based on image processing techniques and on measuring the fringe shift concerning the background. He obtained higher contrast images using multiple-beam interference rather than just two beams interference. The coronavirus refractive index is inferred from a fringe shift. While Narin, Ali, Ceren Kaya, and Ziynet Pamuk [37] proposed a methodology for evaluating different CNN models in their ability to detect COVID-19 using X-ray images. These models are ResNet50, InceptionV3, and Inception-ResNetV2 which are analyzed using 5-fold cross-validation. Results show that the ResNet50 model achieved the best performance with 98 percent accuracy for InceptionV3 and 87 percent accuracy for Inception-ResNetV2 among other proposed models.
The researchers were directed by this hot topic to find an appropriate diagnostic method to avoid spreading it wide. The previous work has addressed this topic from a medical point of view and the others have addressed it as the presentation of CT images using profound thinking, Nonetheless, this paper deals with the use of artificial intelligence algorithms to automatically identify and extract graphical features of chest X-ray images for COVID-19, normal and pneumonia classes, and compares various types of classifiers to inform the most applicant of all.

Materials And Methods
In this section, the used dataset and the proposed methodology including three CNN architectures, three classifiers (SVM, RF, and KNN), in addition to automated extracted features will be discussed in detail.

Dataset Description
In this research, several publicly available datasets of COVID-19 and Non-COVID-19 chest X-ray images have been used [40]. COVID-19 images have collected from different sources; such as Cohen dataset on GitHub (https://github.com/ieee8023/covid-chestxray-dataset), Italian Society of Medical and Interventional Radiology (SIRM) website, Radiopaedia and Radiological Society of North America (RSNA) and a totally of 310 COVID-19 chest X-ray images have been collected [38].
Then a collection of normal and pneumonia images are taken from Kaggle publicly available dataset (https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia) were 310 images for normal chest X-ray images and 310 images for pneumonia for both viral and bacterial have been taken [39,40]. Also, all these images have been added to the dataset to be augmented during the training, the augmentation has been done to prevent CNN from overfitting and memorizing the exact details of the training images. The augmentation here includes flipping the images, rotating the image, translate the image, scale the image and make the total number of images increased which makes the dataset suitable for deep learning [40]. The augmented dataset is publicly available by Alqudah and Qazan using and published online at (https://data.mendeley.com/datasets/2fxz4px6d8). Fig.1 A,

B, and C
show an x-ray image for the Normal, COVID-19 and Pneumonia subjects respectively.

Convolutional Neural Network (CNN)
In general, Convolution Neural Networks (CNNs) is one of the recent advancements of the traditional artificial neural network (ANN), that is composed of self-learning neurons with their weights and biases. For each pixel from an input image entered for every neuron, it will be followed by a dot product calculation in addition to elective non-linearity capture [41]. The input layer represents a layer that will enhance the image with enhancement such as mean subtraction and feature-scaling.
While the feature that distinguished between CNN's and ANNs is a large number of hidden layers, so it is called deep learning. CNN's are consisting of convolution layers that are convolved the image with learnable filters, and each one will produce a features map in the output [42].
Then the spatial coordinate of the generated feature map will be reduced using max-pooling layers.
The process is performed by applying a window on the image with a predefined stride value for each step and finally voting the maximum value of the pixel and putting it in the new image. Meanwhile, another method can be applied in the pooling layer based on the average value, not the maximum [43]. ReLU layer which introduces the nonlinearity for the network its function is [10]. The last layer is a fully connected layer which represents the output layer that has the classification percentages which is serving the classification result as a single vector of probabilities these probabilities passed to the softmax classifier to select the class with the highest probability [44].

AOCTNet
The AOCTNet [34] proposed by Alqudah and initially used to classify optical coherence tomography (OCT) images, and in this research paper to classify x-ray images of the chest. It starts with the first convolutional layer, which uses 32 filters with size 33 and one padding zeros, while the remaining convolutional layers devoted 16 filters of the same size, except for the third, which uses only 8 filters.
The batch normalization layer between the convolution layer and the nonlinearity layer (RELU) is introduced for time-consuming, accelerating training stage purposes, and decreasing the sensitivity of network initialization. Max pooling layer is used with window size 22 in AOCTNet and also increases 2 pixels [35]. The output layer is a fully-connected layer with output size 2, softmax layer, and a layer of classification. Figure 2 illustrates the AOCTNet graphic representation.

ShuffleNet
The ShuffleNet [45] is a very computational and lightweight CNN architecture that is commonly used in mobile devices and embedded systems that use a small computing capacity. The ShuffleNet offers better performance on ImageNet classification tasks as compared with MobileNet. The architecture of ShuffleNet is composed essentially of stacking ShuffleNet units grouped into three levels. In each stage, the first building block is added with a step equal to 2. In each stage, the output channels are similar but multiplied for the next one. ShuffleNet architecture uses two different operations to reduce the cost of computation while preserving accuracy [45]. These operations are group convolution and shuffle of the channels pointwise. The channel shuffle operation allows the channels to be divided into several subgroups in each group, then feed each group to different subgroups in the next layer [45].

MobileNet
Generally, MobileNet is the most commonly used and suitable CNN for embedded device applications such as smartphone and robot vision with a lightweight CNN architecture [46]. The MobileNet consists of 25 layered MobileNet architecture built for the current study that employs four layers of Conv2D, seven layers of batch normalization, seven layers of ReLU, three layers of Zero Padding 2D, and single layers of DepthwiseConv2D, Global Average Pooling, Dropout, and Dense [47]. Like other CNN's, MobileNet is pretrained on ImageNet [48]. Figure 4 Displays MobileNet architecture.

Transfer Learning
Due to the uncommon availability of a huge medical image dataset, training the entire convolutional neural network (CNN) by medical images to achieve a certain classification accuracy is unlikely often.
Therefore, random initialization of weights is replaced using exist CNN that has been already trained on a large dataset usually ImageNet, and this process is called transfer learning. There are two ways to learn how to transfer; the first is to remove fixed features, whereby the data from the intermediate layer is used to train the entire classifier. The second is fine-tuning, which is focused on replacing the last fully connected layer with the new classes and retraining the entire CNN using new modification. This modification will update the weights in the deeper layers [23,33].

X-ray Images Preprocessing
Since this research focused on using lightweight CNNs and any reduction in the size of the input images will result in reducing the time required for both training and classification of new images.
Based on that the input image layer for all architectures was modified to be the size of 128 128 3.
While the size used for RGB images and the chest x-ray represent only 2D images, a 3D concatenation was done for the images to make it RGB.

Features Extraction
Automatic feature extraction using CNN is usually extracted from the fully connected (FC) layer of any architecture. Fully connected layer extract only representative features that able to distinguish between the input class and in this research to distinguish Normal, COVID-19 and Pneumonia cases.
In this paper, the features are extracted from the FC layer of three architectures (AOCT-Net, MobileNet, and ShuffleNet) which is a common method since the FC layer proceeds the Softmax classifier. Based on the selected layer only three features from each class will be extracted and these features will a fine selected and representative [49,50]. The scatter distribution for the extracted features using three models is represented in Figure 5.
Furthermore, these features are used to build hybrid systems another classifier than Softmax classifier using Support Vector Machine (SVM) and Random Forest (RF) classifiers. All classifier's details are discussed in detail in the following sub-sections from 2.4 to 2.7.

Softmax Classifier
CNN's default classifier is the Softmax classifier, and it is a very powerful and commonly used form of discriminant classifier. The softmax discriminant function (SDF) assigns a new input of the test sample to the output class, using the nonlinear transformation of the distance between the test sample and training samples. In this way, the learning rule in a Softmax classifier for the binary units is similar to the regular binary unit law. The only difference is that the Softmax function model is the generalization of the logistic sigmoid function, which can handle classification problems with more than two possible values [49,50].

Support Vector Machine (SVM) Classifier
Support Vector Machine (SVM) is one of the leading and commonly used supervised algorithms in machine learning used to classify all data into two main groups. The SVM uses training data to create a model that distinguishes the data entered and that can be used to predict the new data class. The main objective of the SVM is to find the best hyperplane separating the entire dataset and optimizing the distance between the nearest data point and the hyperplane separating [51,52,53]. In this research, we have been used radial basis function (RBF).

K-Nearest Neighbor (KNN) Classifier
K-nearest neighbor (KNN) algorithm is a non-parametric machine learning method that is fast, lazy, widely used, and instantaneous. In general, KNN's input vector includes the feature space and the target that represents the class member that is graded based on the classes of its neighbor's majority voting technique. The majority vote is applied to the weights that indicate the distance between each characteristic point and the vector mass center [54,55]. In this research, we have been used a hamming distance function with three neighbors using the exhaustive search method.

Random Forest (RF) Classifier
In 2001 Breiman invented the Random Forest algorithm. It is composed of a large number of collective decision trees that function together. Each distinct tree in this form of classifier spits out a class prediction, and the class with the most choices to be the prediction of our model. Simplicity and powerfulness are critical impressions behind random wood. In data science-speaking, several fairly uncorrelated models are the explanation that works so well because of the random forest model [55,56]. In this research, we have been used RF with 100 bags used for bootstrapping.

Performance Evaluation
The confusion matrix is the most commonly used tool for evaluating the efficiency of the artificial algorithm used. It compares device output to reference data. The matrix of confusion shows the most common metrics, such as accuracy, specificity, sensitivity, and accuracy. To test each, the four

Results
The proposed methodology, using different CNN architectures for automated feature extraction to build a hybrid system for the detection of COVID-19 using chest X-ray images. The following sections discuss the results of the hybrid systems built using three different CNN Architecture. All results for training and evaluation below are obtained using tenfold-cross-validation for all classifiers. Also, all used CNN was trained using Adaptive Moment Learning Rate (ADAM) solver with an initial learning rate value of 0.001 and a maximum number of epochs value of 200.

AOCTNet Results
AOCTNet CNN has been used to extract features then to feed these features to four different classifiers (Softmax, SVM, KNN, and RF). The average training accuracies and loss are shown in Figure   6 and Figure 7 shows the activation maps for three cases using AOCTNet. While the testing confusion matrices for all classifiers that have been employed are shown in Figure 8. Finally, Table 1 shows a comparison between all performance evaluations values for the used classifiers.

MobileNet Results
MobileNet CNN has been used to extract features then to feed these features to four different classifiers (Softmax, SVM, KNN, and RF). The average training accuracies and loss are shown in Figure  9 and Figure 10 shows the activation maps for three cases using MobileNet. While the testing confusion matrices for all classifiers that have been employed are shown in Figure 11. Finally, Table 2 shows a comparison between all performance evaluations values for the used classifiers.

ShuffleNet Results
ShuffleNet CNN has been used to extract features then to feed these features to four different classifiers (Softmax, SVM, KNN, and RF). The training and validation accuracies and loss are shown in Figure 12 and Figure 13 shows the activation maps for two cases using ShuffleNet. While the testing confusion matrices for all classifiers that have been employed are shown in Figure 14. Finally, Table 3 shows a comparison between all performance evaluations values for the used classifiers.  Tables 1, 2 and 3 and for automated features extraction using Figure 5.
However, classifiers must be tested for time consumption for classification of an input X-ray image, as shown in Figure 15

Discussion
The present study aimed to investigate the impact of using lightweight deep learning CNN architectures for automated feature extraction from chest X-ray images for COVID-19 disease detection. The proposed method was applied to chest X-ray images dataset with three classes and to avoid overfitting during the training and testing stages of CNN the training dataset has been augmented and the system tested using tenfold cross-validation. Based on the results at first, all trained CNNs are utilized for two different scenarios, the first one is to detect the COVID-19, Normal and Pneumonia using default softmax classifier, while the second scenario is the CNN was employed to extract representative features for hybrid system implementation.
In future studies, these limitations of the present study can be overcome. A more in-depth analysis, in particular, requires much more patient data, particularly those suffering from COVID-19. A more promising approach to future research would concentrate on identifying patients with mild symptoms, rather than symptoms of pneumonia, although these symptoms might not be correctly visualized or visualized on X-rays at all. Also, it is important to establish models that can differentiate COVID -19 cases from other related viral cases, such as SARS, but also a greater variety of common pneumonia or even physiological X-rays. Besides, the automated diagnosis of cases was made using only a visual picture rather than a more nuanced approach to the patient, based on certain variables that could be given and behave as risk factors for disease onset.
The present research nonetheless leads to the possibility of using a lightweight CNN for low-cost, rapid and automatic COVID-19 disease diagnosis. But it's also about investigating whether the extracted features performed by the CNNs are accurate biomarkers that help detect COVID-19. Also, given the fact that the correct treatment is not determined solely by an X-ray image, an initial sampling of the cases will be helpful, not in the form of treatment but in the timely implementation of quarantine measures of the positive samples, before a more thorough evaluation and detailed treatment or follow-up protocol is followed. In addition to the benefit of automated COVID-19 identification from any medical picture lies in the sensitivity of nursing and medical personnel to the outbreak. Table 4 shows a comparison between the proposed method and other methods in the literature. Respiratory failure has caused deaths, leading to the loss of other organs. Since a significant number of outdoor or emergency patients attend, doctor's time is limited, and computer-aided diagnosis may save lives through early screening and prompt treatment. Also, due to the radiologist's knowledge differences, there is a significant degree of variability in sample images from the X-ray machines. All used CNN's exhibits an excellent success in the detection of COVID-19 by training itself effectively from a comparatively lower picture set. We assume this computer-aided diagnostic method will greatly improve the speed and accuracy of COVID-19 diagnosis cases. This will be particularly useful in this pandemic where the risk of the disease and the need for preventive measures are at odds with the resources available.  AOCT-Net graphical representation.

Figure 4
The Architecture of MobileNet [46,47,48].  The Average Training Accuracy and Loss for AOCTNet.   The Average Training and Loss for MobileNet.   The Average Training and Loss for MobileNet.   Time for Each Classifier with the Used Three CNN Architectures.