The aim of the proposed research is to build a model by utilizing appropriate feature selection techniques and classification algorithms to improve accuracy. The four phases of the approach used in this study include data collection, feature selection and pre-processing, data visualization, and classification for the student's class. The first phase deals with data collection and feature engineering techniques. Phase II is concerned with stratified-K fold cross validation to divide the dataset into training and testing. Phase III is concerned with classification algorithms to build a model for prediction, and the last phase is for evaluation purposes. Figures 1 and 4 shows the architecture of proposed methodology and the algorithm in detail.
3.1 Data Collection
In this research work we used a public dataset which was collected from several disjointed databases by the Polytechnic Institute of Portalegre, Portugal [16]. The dataset contains 4482 instances, which include various undergraduate courses enrolled in by the students during the academic years 2008-09 to 2018-19. The student's academic performance of the first two semesters is included in the dataset, along with the students' demographics, social-economic status, and academic trajectory that were available at the time of enrolment. The information is used to create classification models that forecast student success and dropout rates. At the conclusion of the typical course length, the issue is presented as a multiclass categorization exercise: graduate, dropout and enrolled. Table 1 shows a complete description of a dataset. For more details about the dataset can be found in [12, 16].
3.2 Data Preprocessing.
Missing values are a prevalent issue in many real-life concrete datasets. Missing values may bias the outcomes of machine learning models and/or lower the model's accuracy. Based on the dataset missing values can be handled by deleting row/column values or it can be replaced with some arbitrary values.
ML algorithms cannot use qualitative data if our features are qualitative or categorical data. Therefore, before inputting the characteristics into the prognosticative model, we must transform this qualitative data into a numerical form. In this way, category data may be transformed into numerical form using label encoding. Each unique nominal variable is given an integer number using this method. However, we solely used numerical data for this investigation.
Data scientists [14] examine and analyze data sets and epitomize their key properties using exploratory data analysis (EDA), which is regularly, employs data visualization techniques. It enables the researchers to extract the useful patterns, determine the anomalies, and verify the assumptions by identifying how to change the data assets for improved accuracy. EDA is typically used to study what type of data might be disclosed beyond the precise modeling or hypothesis testing assignment. It aids in a better exploration of the hidden patterns of the variables in the data collection and their interactions. It can also help you choose whether the statistical procedures you're considering for applying data analysis are applicable. Figure 2 shows the analysis of some of the features using visualization techniques.
Table 1
Complete Description of Dataset
S.No | Attribute | Description | Type |
1 | Marital status | Student's Marital Status | numeric (from 01 to 06) |
2 | Application mode | Mode of Application to Enroll the Course | numeric (from 01 to 18) |
3 | Application order | Students Application Order | Numeric/ordinal |
4 | Course | Course Specialization | numeric (from 01 to 17) |
5 | Daytime/evening attendance | Attendance Regime Values | binary (1-DayTime, 0-Evening) |
6 | Previous qualification | Student's Previous Qualification | numeric (from 01 to 17) |
7 | Nationality | Student's Nationality | numeric (from 01 to 21) |
8 | Mother's qualification | Mother's Education Qualification | numeric (from 01 to 34) |
9 | Father's qualification | Father's Education Qualification | numeric (from 01 to 34) |
10 | Mother's occupation | Mother’s occupation | numeric (from 01 to 46) |
11 | Father's occupation | Father’s occupation | numeric (from 01 to 46) |
12 | Displaced | Refugee/Displaced Candidate | binary (1-DayTime, 0-Evening) |
13 | Educational special needs | Req of Educational special needs | binary (1-Yes, 0-No) |
14 | Debtor | Debtor | binary (1-Yes, 0-No) |
15 | Tuition fees up to date | Student's Tuition Fees Status | binary (1-Yes, 0-No) |
16 | Gender | Student's Gender | binary (1-Yes, 0-No) |
17 | Scholarship holder | Students’ Scholarship status | binary (1-Yes, 0-No) |
18 | Age at enrollment | Student's Age at enrollment | Numeric/discrete |
19 | International | Student's Nationality Status | binary (1-Yes, 0-No) |
20 | Curricular units 1st semester (credited) | Student's Curricular units 1st semester (credited) | numeric (from 00 to 20) |
21 | Curricular units 1st semester (enrolled) | Student's Curricular units 1st semester (enrolled) | numeric (from 00 to 26) |
22 | Curricular units 1st semester (evaluations) | Student's Curricular units 1st semester (evaluations) | numeric (from 00 to 45) |
23 | Curricular units 1st semester (approved) | Student's Curricular units 1st semester (approved) | numeric (from 00 to 26) |
24 | Curricular units 1st semester (grade) | Student's Curricular units 1st semester (grade) | numeric (from 0.0 to 18.88) |
25 | Curricular units 1st semester (without evaluations) | Student's Curricular units 1st semester (without evaluations) | numeric (from 00 to 12) |
26 | Curricular units 2nd semester (credited) | Student's Curricular units 2nd semester (credited) | numeric (from 00 to 19) |
27 | Curricular units 2nd semester (enrolled) | Student's Curricular units 2nd semester (enrolled) | numeric (from 00 to 23) |
28 | Curricular units 2nd semester (evaluations) | Student's Curricular units 2nd semester (evaluations) | numeric (from 00 to 33) |
29 | Curricular units 2nd semester (approved) | Student's Curricular units 2nd semester (approved) | numeric (from 00 to 20) |
30 | Curricular units 2nd semester (grade) | Student's Curricular units 2nd semester (grade) | numeric (from 0.0 to 18.57) |
31 | Curricular units 2nd semester (without evaluations) | Student's Curricular units 2nd semester (without evaluations) | numeric (from 00 to 12) |
32 | Unemployment rate | Unemployment rate during academics | numeric (from 7.6 to 16.2) |
33 | Inflation rate | Inflation rate during academics | numeric (from − 0.8 to 3.7) |
34 | GDP | GDP during academics | numeric (from − 4.1 to 3.5) |
35 | Target | Target to Predict | Enrolled/Graduate/Dropout |
Machine learning usually makes use of the data preparation process known as "normalization." Normalization is the process of scaling all the attributes in a dataset to the same value. Machine learning does not require the normalization or standardization of every dataset. For example whether an outlier exists in a data collection depends on the data set. If so, use the Z-score normalization; if not, use the Min-Max normalization. It is only required when the feature ranges differ.
3.3 Stratified K-Fold Cross Validation
The basic principle behind the resampling approach known as cross-validation is to divide the dataset into two sets in which the first one is training and the second one test data. The model is trained using first set of training data, and predictions are made using second set of test data that hasn't been seen. It indicates that the model hasn't over fitted the training data and may be used for prediction if it performs well on the test data and provides excellent accuracy [13]. In our dataset we have highly imbalanced data so we used a stratified K-fold cross validation with the K value ‘5’ to make sure the all the categories proportions will be taken in approximate equal amount.
3.4 Imbalanced Dataset
Imbalanced data are datasets in which the dependent variable has an unequal distribution of target classes, i.e. one class label in particular class has a lot of observations, whereas the other class has little observations as shown in Fig. 3. The challenge was designed as a three-category classification assignment with a substantial bias toward one of the classes. Graduate accounts for 50% of total records, which means 2209 records out of 4424, Dropout accounts for 32% of total records, which means 1421 records out of 4424; and Enrolled accounts for 18%, which is almost 794 records out of 4424. This may result in a majority-driven increase in prediction accuracy at the price of a minority-driven decline in performance.
In this research work, the imbalanced dataset problem was handled by the sampling technique Synthetic Minority Over Sampling Technique (SMOTE). This resampling method is usually used before training and testing in a traditional machine learning approach. If that's the case, the test data may also have duplicate values, which makes the prediction wrong. To ensure better accuracy, SMOTE has been applied only to the training dataset. Figure 3 shows the variations before and after applying SMOTE.
3.5 ML Classification Models.
To predict student academic performance, the modeling stage employs nine different ML classification algorithms, namely K-Nearest Neighbors (KNN), Decision Tree Classifier, Random Forest, Support Vector Machine, Neural Networks, Naive Bayes, XG Boost, and AdaBoost used as Level-1 base classifier and Logistic Regression used as a Meta classifier. The main objective of this prediction process is to create an optimized hybrid model based on demographic, socioeconomic, macroeconomic, and academic data to forecast student performance.
3.6 Stacking Classifier
Stacking [14] [1] classifier is an ensemble learning method that incorporate multiple classification models to improve the veracity and robustness of predictions. It works by training several base classifiers on the same dataset, using different algorithms or hyper parameters. The predictions of these base classifiers are then combined by a meta-classifier, which makes the eventual prediction. The meta-classifier in a stacking classifier can be any classification algorithm i.e. Logistic Regression. The stacking classifier is trained using a 2-stage process. In the stage-I, the base classifiers are trained on the training set. In the stage-II, the meta-classifier is trained on the predicted probabilities of the base classifiers, using the same training set. Stacking classifier can handle complex datasets where non-linear relationships between the features and the target variable. It can also improve the robustness of predictions by combining the strengths of multiple base classifiers.
3.7 Evaluation Measures
Evaluation measures in machine learning are statistical metrics used to obtain the efficiency of a model. In this research, we used five different commonly used evaluation measures. Accuracy measures the proportion of correctly predictions made by the trained model. It is often used for classification problems, and it is the ratio of correctly predicted cases out of the total number of cases. Precision is a measure of a model's ability to identify positive cases correctly. It measures the proportion of accurately foreseen positive observations to all of the foreseen positive observations. Recall is a measure of a model's ability to forecast all the positive cases. It is the proportion of properly foreseen positive observations to the total number of true positive cases in the dataset. The harmonic mean of recall and accuracy is the F1 Score, and it is a recommended to use when there is an uneven class distribution. The ROC-AUC curve is well-used to represent the performance of a classifier graphically. The area under the curve (AUC) provides an overall indicator of performance for all potential thresholds since it displays the observed true positive rate out of the observed false positive rate for various thresholds values.
Accuracy = (True Positives (TP) + True Negative (TN)) / (Total Number of Predictions) (1)
Precision = (True Positives (TP)) / (True Positives (TP) + False Positives (FP)) (2)
Recall = (True Positives (TP)) / (True Positives (TP) + False Negatives (FN)) (3)
F1 Score = 2 * (Precision * Recall) / (Precision + Recall) (4)
Table-4 shows the result summary for the 8 different classification algorithms and given as a level-1 input to the Meta classifier which is implemented using Logistic Regression.
Table 4
Accuracy Comparison for the Level-1 Classification Algorithms.
Algorithm | Dataset with 35 Features |
Imbalanced Dataset | Balanced Dataset (SMOTE) | Balanced Dataset (SMOTE + SK-Fold) |
KNN | 70.04 | 60.87 | 70.12 |
DT | 68.58 | 67.18 | 74.17 |
RF | 79.87 | 78.72 | 80.55 |
SVM | 74.03 | 72.18 | 71.42 |
NN | 76.14 | 69.32 | 80.23 |
NB | 66.02 | 63.54 | 66.23 |
XGBoost | 72.84 | 69.46 | 79.05 |
AdaBoost | 70.54 | 68.02 | 75.74 |