Data Transformation and Wrapper Impact on Machine Learning Algorithms in Credit Evaluation

doi:10.21203/rs.3.rs-3332242/v1

Download PDF

Research Article

Data Transformation and Wrapper Impact on Machine Learning Algorithms in Credit Evaluation

https://doi.org/10.21203/rs.3.rs-3332242/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Banks utilize credit scoring as an important indicator for the financial strength and the eligibility for credit. Scoring models aim to assign statistical odds or probabilities for predicting if there is a risk of nonpayment in relation to many other factors involved. This paper aims to illustrate the beneficial use of the ten machine learning methods (Support Vector Machine, Gaussian Naïve Bayes, Decision Trees, Random Forest, XGBoost, K-Nearest Neighbors, Multi-layer Perceptron Neural Networks, CatBoost, Light Gradient Boosting Machine, and Logistic Regression) in finding the default risk as well as the features contributing to it. An extensive comparison is made in three aspects: (i) which ML models with and without its own wrapper feature selection performs the best; (ii) how appropriate data scaling influences the performance and computational costs; (iii) which of the most successful combination (machine learning method, feature selection and scaling) delivers the best validation indicators such as accuracy rate, sensitivity, and specificity ratio. An open-access credit scoring default risk data sets on German and Australian cases are taken into account for which we determine the best method, scaling and features contributing to default risk. We also illustrate the positive contribution of the feature selection method and scaling on the performance indicators, and a ranking system.

Credit Risk

Machine Learning

Wrapper Feature Selection

Data Scaling

Classification

Machine Learning (ML) methods offer a wide range of approaches to determine the characteristics, specifications, and subcategorizations of variables by taking into account the associations among each other [33]. Prediction accuracy of these methods is sensitive to the selection of the features, methods, size of the data and transformation properties. Additionally, such variability in ML gives many alternatives to the users to increase the prediction power. Certain statistical tests are used to depict the accuracy of these methods [19].

The implementation of ML becomes very attractive, especially after the development of high-speed computers which allows to process huge number of data sets [1]. As much as the other disciplines, financial sector benefits the use of ML methods as well as diversity in algorithms. A very important application area is the banking sector whose financial risk is highly uncertain and difficult to predict due to its complex structure with multi-dimensional factors involved. Banks are prone to high default risk, mainly due to credit risk arising, from market risk and economic recessions [28]. A recent history (subprime crisis in 2008) has shown that, as a result of the unrestrained growth of banking sector and financial globalization, the financial crisis arise from housing market fluctuations in the USA, and spread worldwide causing big financial disasters at every level in the world. Consequently, lack of risk management in banking sector before subprime crisis results in large losses. Hence, regulatory framework such as Basel II and then III are developed, and risk management becomes one of the main tools determining contestability of banks [17].

Besides, the current and future risk levels of the bank’s loan portfolio are crucial in terms of its competitiveness, resilience to financial crises and profitability. These requirements urge banks to quantify the amount of risk they are prone to by determining the credit risk at institutional and individual bases. Banks can make credit classification in two different ways. The first is based solely on the expert judgment of the credit analyst. Another is the credit scoring which is basically the classification criteria based on the current and historical financial strength of the client [12]. Credit assessment through the risk management system should be robust, reliable and modernized with respect to the changes in data science [2]. For this reason, the investigation of the most suitable and current ML classification algorithms fulfilling these requirements have gain great attention during the last decades.

In the implementation of ML methods, four aspects contribute to the success in prediction: (i) the ML method chosen, (ii) the feature selection, (iii) the data transformation, and (iv) the validation structure. It is apparent that the combination of these four aspects creates a huge number of trials and varying computation times to come up with the best predicting method. In literature, there exist vast amount of studies on measuring the credit risk, mostly taking into account, linear discriminant analysis, factor analysis, logistic regression, artificial intelligence such as expert systems and neural networks, Bayes classifier, nearest neighbor, and classification trees. It is found that neural networks and XGBoost (XGB) algorithms are more accurate in classification than other techniques, especially on loan assessment and predicting the default probability [25, 31, 34, 48, 50].

The recent and recognizable studies comparing the most suitable ML methods based on real data sets point out that random forest (RF), Gradient Boosting (GB), Decision Tree (DT), Neural Network (NN), Support vector Machine (SVM), K-Nearest Neighborhood (KNN) and some of their variations yield relatively good results compared to the traditional methods such as logistic regression [4, 9, 21, 28, 32].

In feature selection area on ML algorithms, variety of methods, such as wrapper, filter and hybrid algorithms, are applied by considering performance indicators like the highest median and the lowest standard deviation [10, 19, 35, 43, 49]. Especially, methods with Genetic Algorithms (GA) and Particle Swarm Optimization (PSO) and naïve Bayes (NB) are found to have the highest performance achievement against the widely used filter methods t-Test, linear discriminant analysis (LDA), logistic regression (LR), information gain, and rough set theory-based ones in terms of receivers operating curve (ROC) and accuracy ratio.

As the third aspect in the ML analyses, the effect of data transformation is not considered extensively in the literature. The most well-known transformation methods used in literature are Standard and Min-Max transformation methods. Besides, Median Norm, Sigmoid Norm, Statistical Column Normalization are also implemented to see the influence of the data transformation on the variables to point out the effectiveness of transformation on the accuracy [22, 44].

Finally, the ML algorithms require justification of goodness of fit through statistical tests (validation criteria) which are imposed on training data whose sensitivity changes with respect to its number of observations. The most common selection is 80 − 20% separation of train and test groups, which may not be effective for small samples [51, 52].

Having the motivation on figuring out the best ML algorithm together with the best combination of feature selection, data transformation and train-test separation in credit risk scoring, this paper aims to obtain more explanatory results in terms of dimensionality, and investigate capability of detection significant features when wrapper feature selection (WFS) method is used with ML algorithms. We also aim to analyze the contribution of hyper-parameter optimization of these parameters which are included in all ML algorithms and can be manually adjusted with Grid Search (GS) along with the four types of data transformation to these algorithms and methods. For illustration of our approach, the German and Australian credit data sets [15] are studied due to their online accessibility and availability as well as to make comparison of our findings with the literature [5, 30, 32, 37, 41, 46, 48, 51].

We expect that the findings of this paper guide the researchers and practitioners both in inducing the important characteristics in credit scoring as well as the correct implementation of the factors to have the maximum possible accuracy in ML applications.

The organization of the paper continues with Section 2 which shortly describes metrics, feature selection and parameter optimization. The preliminaries of ML methods are introduced in Section 3. The definition of validation criteria, experimental design approaches along with German and Australian data sets are presented in Section 4. Outcomes of the application of the ML algorithms are analyzed and illustrated in Section 5; we present the accuracy and efficiency indicators in terms of the significant characteristics in three aspects consisting of large variety of different approaches; the impact of data normalization techniques to the efficiency of the ML algorithm, the effect of wrapper methods on ML and the examination on how GS parameter optimization affects these algorithms are explained in detail. The final section concludes the paper with a summary and remarks.

In preprocessing phase of the data set, feature scaling is a vital part that could be easily ignored by the analysts. DT, RF and XGB are some of the tree-based algorithms that do not require feature scaling. These algorithms are insensible against to scaling, whereas the most of ML and optimization methods work more efficiently when attributes are on the same range [38]. For instance, different attributes are typically measured on various ranges, and if there is a use of Euclidean distance or Manhattan distance metric, it can cause some features completely outweigh the others that have smaller range of measurement [47]. Normalization and standardization are two of the most common methods for bringing distinct attributes to the same scale [39].

We consider natural log, Box-Cox, Min-Max and Standard scaling methods in our analyzes. Box-Cox and natural log are the members from the power transformation family. These two techniques are applied to make the data distribution bell shaped by shrinking the higher values and expanding it to the lower values. These are also useful for heavy tailed distributions [52]. Min-Max scaling provides a linear transformation of the initial data. It maintains the connection between the values of the initial data [19]. Min-max scaling also squeezes (or stretches) all feature values to be within the range between 0 and 1 by considering the maximum value and minimum of an attribute [52]. In standard scale feature columns are set in the midst at standard deviation one and mean zero. In contrast to Min-Max scaling, the standard scaling sustains useful outlier information and makes the algorithms more robust [38].

Feature selection is one of the most influential analysis for the performance of the model, since irrelevant or partially relevant features can cause negative effect. Besides, this analysis provides various benefits such as reduction in run time and obtaining more accurate results [42]. In our study, we use wrapper attribute selection approach (WFS) in which the learning algorithm is wrapped into variable subset selection procedure. It would be simple to make an independent evaluation of an attribute subset if there was a silver bullet way to determine when an attribute was related to the class. Yet, there is no widely accepted measure of significance, even though several varieties of measurements have been proposed [52]. The general concept of the wrapper method is to pick a subset of features using a learning algorithm as a part of the evaluation function that can be used with any machine learning algorithm [6]. To prevent overfitting, we apply this method with K-Fold Cross validation (KFCV). Also, application of this method is conducted by the Sequential Forward Selection (SFS). This begins with the induction of machine learning algorithms for \(n\) single-attribute subsets from which the best performer is selected after the first selection phase. Among the \(n-1\) attributes the most fitted one with the already selected feature is spotted, and this attribute is added to the subset. This process goes on until the number of subsets reaches to \(n\). In that way, whole attribute combinations are validated and the subset that has the highest score, such as accuracy or minimum weighted error, is detected.

In ML, there are basically two kinds of parameter estimates. The first one is learned from the training data, as the weights of neurons in MLP; the other is the hyper-parameters of a learning algorithm that are optimized separately, such as the number of trees in RF [38]. The search of the possible combinations of hyper-parameters, and where to look for an optimum is a cumbersome task. A hyper-parameter space may contain value combinations that perform better or worse. Even after detecting a good combination, there may be a better combination of the parameters due to being a local optima. A practical way to overcome this problem, verifying hyper-parameters for an algorithm applied to specific data, is to test them in large variety of KFCV trials and pick the best combination based on the highest accuracy rate.

The Grid Search (GS) makes this process easier by allowing the user by sampling the range of possible values to input into the algorithm and to spot when the general error rate is in minimum [33]. This is a brute-force approach in which a list of prespecified values belonging to different hyper-parameters is used, and the model performance is assessed based on the highest accuracy among the combination of those parameters.

To obtain more robust and general results, GS is commonly applied with a 10-fold cross validation [39] approach.

Ten commonly used mathematical and statistical algorithms investigated in this paper are briefly summarized below. Also, the hyper parameters of these algorithms are listed in Table 1.

Logistic Regression (LR) predicts dependent categorical variable with respect to independent ones. The class probabilities are derived through generated model yielding an estimated value between 0 and 1 [26]. It is known that LR as a ML method does not require diagnostic tests.

On the other hand, Gaussian Naïve Bayes (GNB) is a statistical classifier at which the ML process is transparent and clear understanding of how they behave [29]. The posterior probability of every sample is determined based on considering their particular classes [16]. These classes assign the most probable one to a prespecified instance defined by its attributes [40]. The conditional probability of an observation in class \(C\), whose distribution is Normal due to good handling of continuous variables.

The other distance-based method, Support Vector Machine (SVM), converts the initial training data into a higher dimension and searches for the linear optimal separating hyperplane. The decision boundary, separating the different class variables, is searched. The algorithm decides on the location and margins using support vectors in training set [19] at which the points above or below a dividing hyperplane constitute the classes.

One of the tree-based algorithms, Decision Tree (DT) having internal nodes, branches and leaf require the constructing model, representing a test on feature by each internal node, a class label symbolized by leaves, and an outcome defined by each branch [19]. DTs are structured general to specific information starting from the top node. A node changes to a leaf and then the pruning is completed [3]. The model is structured based on the measures such as Gini or Entropy.

Having similar structure, Random Forest (RF) allows a large number of classification trees (forest) based on Breiman’s CART tree [7]. Every tree in the forest holds a classification vote to determine an instance to an appropriate label. Bootstrap method is used in training process, to sample randomly the number of instances to build a new tree [8]. Once a bag of trees is constructed, the class of an observation is determined as an ultimate function of decision-making tree consensus.

As a modified version of Gradient Boosting algorithm based on CART, XGBoost (XGB) belongs also to the class of decision tree approach. The algorithm offers regularized objective and sparsity-awareness and handles the missing inputs, sparse values and binary variables with default direction approach [11]. Different than RF, at each step a new tree is constructed in accordance to the previous tree’s error.

Also, Light Gradient Boosting Machine (LGBM) which is an improved version of XGB, uses histogram-based method to reduce computation time in model construction and involves binning the continuous eigenvalues values. Along with saving memory use it provides leaf-wise growth with depth constraints without overfitting and reducing the prediction power [24].

Another decision tree algorithm based on Gradient Boosting, CatBoost (CB) aims to solve Gradient bias and reduce the prediction shift to prevent overfit [13]. It also combines categorical features into new ones and improves the extraction of the information from the data [20].

An extended version of feed forward neural networks, Multi-layer Perceptron Neural Networks (MLP) includes ordinarily an input layer, one or more hidden layers, and the output layer. The observations which are fed into input layer are transferred into hidden layer without any change. In hidden layer, every node receives the outputs from the previous layer, and transfers them into the next layer under some assumptions. At the last phase, the outcomes are generated which end up with a classification or regression in the output layer [18].

The last algorithm, which does not require any fitted model, is the K-Nearest Neighbor (KNN). It is a memory-based method at which the closest \(k\) points based on a distance is found in the sample to a testing point. Then, the majority vote assigns the instance to suitable class over all the \(k\) neighbors in terms of a metric such as Euclidean distance [45].

Table 1

The list of hyper-parameters and searching spaces
Algorithm	Hyper-parameter	Description	Searching space
CB	No. of iterations	The number of base predictors in the model	[5, 40]
	Maximum depth	Maximum depth level of the trees	3
LGBM	No. of estimators	The number of base predictors in the model	[5, 40]
	Maximum depth	Maximum depth level of trees	6
	Learning rate	It controls the step size in optimizations	0.3
RF	No. of estimators	The number of predictors in the model	[5, 40]
	Splitting Criterion	Function for measuring impurity ratio	{Gini, Entropy}
	Maximum depth	Maximum depth level of trees	6
XGB	No. of estimators	The number of predictors in the model	[5, 40]
	Maximum depth	Maximum depth level of the trees	6
	Subsampling rate	The subset of sample variables used for construction a single tree	1
	Learning rate	It controls the step size in optimizations	0.3
	Column sampling rate	The subset of sample variables used for construction a single tree	1
	Gamma	Decrease in minimum loss for the next split	1
DT	Complexity parameter	Division contribution model fitting ratio	[3.21e-4, 6.76e-3]
	Maximum depth	Maximum depth level of a trees	15
	Splitting Criterion	Function for measuring impurity ratio	{Gini, Entropy}
ANN	No. of hidden units	Structure of model	{(10,), (100,), (150,), (10,10), (100,100), (150,150)}
	Activation Function	Restrain the output value	{Identity, Logistic, Hyperbolic, Rectified}
	Decay	Penalty parameter	0.5
SVM	Kernel function	Function that increases the data dimensionality	{Radial, Linear, Sigmoid, Polynomial}
	Gamma	Controls the influence of an observation on hyperplane	1
kNN	k	Number of neighbors for voting	[2, 43]
	Distance Metric	Calculates the distance between initial samples	{Euclidean, Manhattan, Minkowski, Chebyshev}
GNB	Kernel	Consideration of conditional densities for class prediction	{TRUE, FALSE}
	Laplace Smoothing	Fixes the zero probability	{10 − 1,10 − 3,10 − 5,10 − 7,10 − 9}

Validation Methods and Credit Scoring Data

Validation criteria are the measures for assessing how good or how accurate a classifier is at predicting the class labels of instances. The accuracy ratio, Sensitivity and Specificity rates are used. Sklearn-learn libraries [36] are used to conduct the experiments for eight machine learning algorithms except for CB [14], and LGBM [23].

Ratio of the number of truly predicted observations and the total number of observations gives the accuracy. Sensitivity rate can be calculated by dividing the number of correctly classified observations that belong to that class by total number of observations in the class. Specificity is defined as the proportion of number of truly detected Class 0 to total number of cases in the same class [19].

On the other hand, in KFCV, the present samples are split at random into \(k\) disassociated subsets, \({D}_{1}, {D}_{2}, · · ·, {D}_{k}\), where each has roughly equal number of samples. In this validation process, testing and training is executed \(k\) times. In step \(i\), split part \({D}_{i}\) is kept as a validation set, and the rest of the data samples are used to train the model.

An important decision in the analyses requires the experiments to be separated into two sets as Train and Test parts. The proportion of the separation plays an important role. The most of the studies in literature considers as randomly divided subset of data set having 80% as the training and 20% as the test sets. This selection criterion is crucial depending on the size of the data and the frequency of the members in each class. Literature displays also different partition among train and test data. The study [51] shows partition of data set starting from 80% at each incremental percent to 90% has influence on the accuracy indicators.

In our analyses, we aim to show the classification ability of the ten ML algorithms, and impact of the four data transformation techniques on these algorithms. Besides, we employ three approaches in our analyses. The first one is the application of GS for parameter optimization, the second one is the implementation of Wrapper Feature Selection (WFS) method for feature selection. The last one is the jointly use of WFS and GS; 10-fold cross validation is applied to avoid over fitting in these approaches. Each of the experiments are validated on three randomly divided train and test subsets. The data sets are randomly divided into 90% as the train and 10% as the test sets by maintaining the distribution of classes same as the original data set. In this way, we can prevent the insufficiency in terms of default or non-default samples on train or test data sets for the credit datasets we study in this paper. Average of AUC, AR, Type I and Type II errors of these three cases is used to compare the effectiveness of the algorithms and methods.

Credit Datasets

Due to the regulations and laws on the protection of personal data in banking sector and institutional confidentiality reasons, it is very difficult to receive credit history from any financial institutions. For this reason, two real-world data sets, German and Australian, which are provided by the UCI machine learning data repository [15], are used to conduct our proposed study. Since these data sets are frequently used in literature, it also sets a good base to make comparison to other published works.

Table 2

The variables in German and Australian credit data sets with their features
German Dataset			Australian Dataset
#	Attribute	Type	#	Attribute	Type
1	Age (years)	Continuous	1	(A)	Categorical
2	Duration (month)	Continuous	2	(B)	Continuous
3	Credit history	Categorical	3	(C)	Continuous
4	Purpose	Categorical	4	(D)	Categorical
5	Credit amount	Continuous	5	(E)	Categorical
6	Savings account/bonds	Categorical	6	(F)	Categorical
7	Present employment since	Categorical	7	(G)	Continuous
8	Instalment rate	Continuous	8	(H)	Continuous
9	Personal status and sex	Categorical	9	(I)	Continuous
10	Other debtors/guarantors	Categorical	10	(J)	Continuous
11	Present residence since	Continuous	11	(K)	Continuous
12	Property	Categorical	12	(L)	Continuous
13	Status of existing checking account	Categorical	13	(M)	Continuous
14	Telephone	Categorical	14	(N)	Continuous
15	Other instalment plans	Categorical	15	Default State	Categorical
16	Housing	Continuous
17	Existing credits at this bank	Categorical
18	Foreign worker	Continuous
19	Job	Categorical
20	Number of dependants	Categorical
21	Default State	Categorical

German Dataset contains 24 attributes and 1000 observations 700 of which belong to Class 1 (No default) and 300 of the observations to Class 0 (Default). In our study we use the one adjusted to be suitable for working with algorithms that can work only numerical variables (not categorical ones). Seven of these attributes are numerical whereas the others are categorical variables. The definition of the attributes is given and details are listed in the left of Table 2. However, Australian Dataset consists of credit card applications of 690 observations, 307 to be classified as Class 0 and the rest as Class 1. The data source and the attributes contained within are labeled by letter codes. There are six numeric and 8 categorical attributes (right panel in Table 2) that are also appropriate for the algorithms. The details of the attributes are covered or transformed to protect personal information, remain only within the type of the variables.

Application

In this part, we investigate which of the ten machine learning methods is superior to others in terms of consumer credit classification. We also examine the effect of GS and WFS methods on the performance of these classification algorithms, and observe how four different data conversion techniques influence the effectiveness of these algorithms. WFS is applied to each algorithm with use of sequential forward selection (SFS) method. Experimental design is made on the separation of 90% for training and 10% for testing by maintaining the proportion of each class in both sets. In order to compare the results, data sets are divided according to certain random states. For each case, five different train and test sets are generated according to these random states, and average of these five tests results’ are considered for comparison in terms of accuracy, sensitivity and specificity rates. The tables in result section indicate the ML algorithms with different combinations of data transformations (no scaling, standardized, Min-Max, Box-Cox, and natural log), the ML algorithms (LGBM, CB, GNB, SVM, MLP, DT, RF, XGB, LR, and KNN) and methods (conventional ML, WSF, GS) in terms of their efficiency rates in percent. Illustrative comparisons of the findings are also plotted and presented in Appendix A (Figure A.1) for providing more details of the effects of feature scaling methods to the methods for six algorithms (SVM, LR, MLP, GNB, RF, and KNN). These figures depict the accuracy of each method used with respect to the data transformations, hyper parameter optimization and feature selections.

In this part, we examine the influence of four different data transformation methods, wrapper feature selection approach to the performance of ML algorithms, and their effect on the computational times. Additionally, we show the benchmark results of ten ML algorithms on two publicly available credit risk datasets. The average outcomes of each model for each credit risk dataset over five separate testing and training partitions are illustrated in Tables 3–7. In the tables, each evaluation criterion belonging to the top model is typed in bold to indicate its superiority. It should also be noted that the programs are run in an environment using Python 3.6.3 on an Intel 12700H CPU with 14 cores, 16 GB of RAM PC. The findings of the research are list in the following five titles.

Influence of transformations to prediction power

We consider Natural Logarithm, Cox-Box, Min-Max and Standard data transformation methods. The application of these methods is divided into 2 cases, the first, appliance to only the continuous variables, and, the second, to both continuous and categorical variables defined as (all).

Table 3

Rankings for Australian dataset on data transformations.
SVM	Rank	LR	Rank	KNN	Rank	MLP	Rank	RF	Rank	GNB	Rank
Min-Max	2.33	Standard	2.00	Min-Max (all)	2.33	Box-Cox	2.00	Box-Cox	1.33	Min-Max (all)	1.78
Standard (all)	2.56	Min-Max (all)	2.89	Standard (all)	2.67	Standard (all)	3.00	No Scale	1.33	Min-Max	1.89
Min-Max (all)	2.67	Natural logarithm	3.00	Min-Max	3.00	Min-Max (all)	3.11	Min-Max	1.67	Natural logarithm	1.89
Standard	3.78	Box-Cox	3.44	Standard	4.22	Natural logarithm	3.44	Standard	1.67	Box-Cox	1.89
Natural logarithm	4.22	Min-Max	3.44	Natural logarithm	4.67	Min-Max	3.67	Min-Max (all)	1.89	Standard (all)	2.00
Box-Cox	4.56	Standard (all)	3.78	No Scale	4.67	Standard	3.89	Standard (all)	1.89	Standard	2.22
No Scale	4.67	No Scale	3.89	Box-Cox	5.00	No Scale	5.78	Natural logarithm	2.00	No Scale	2.78

Table 4

Rankings for German dataset on data transformations.
SVM	Rank	LR	Rank	KNN	Rank	MLP	Rank	RF	Rank	GNB	Rank
Min-Max	2.22	Standard (all)	1.89	Min-Max	2.11	Standard	1.44	Natural logarithm	2.00	Standard	2.11
Standard (all)	2.89	Standard	2.11	Standard	2.56	No Scale	2.22	Box-Cox	2.22	No Scale	2.44
Standard	3.22	Box-Cox	3.00	Box-Cox	3.44	Standard (all)	3.56	No Scale	2.22	Standard (all)	2.56
Natural logarithm	3.44	No Scale	3.44	Standard (all)	4.11	Box-Cox	3.67	Min-Max	2.44	Min-Max (all)	2.67
No Scale	3.78	Min-Max (all)	3.44	Natural logarithm	4.11	Min-Max (all)	4.56	Standard	2.78	Min-Max	3.11
Min-Max (all)	4.22	Natural logarithm	3.44	No Scale	4.67	Min-Max	4.56	Standard (all)	2.78	Box-Cox	3.22
Box-Cox	4.44	Min-Max	4.67	Min-Max (all)	4.89	Natural logarithm	4.67	Min-Max (all)	3.44	Natural logarithm	3.44

In order to show the effect of transformations, we consider the approach that taking the average rankings of three methods default, GS, and WFS on each three criteria; accuracy, sensitivity, and specificity.

Table 3 indicates that any kind of scaling is profitable for SVM algorithm, especially, Min-Max delivers the best ranking, and similar results also apply for the German dataset. When we examine LR algorithm, the results suggest that Standard scaling is the most efficient approach for both datasets. Another ML algorithms that scaling is valuable is KNN. The Min-Max and Standard are the most convenient way of transformation for this algorithm. When we consider the MLP, it can be said that all transformation approaches utilize the performance on the Australian dataset, and best choice for scaling is the Box-Cox. However, we observe that transformation is ineffective and even has adverse effect on the other dataset (Table 4). A similar view applies to GNB algorithm, in which any scaling method enhances the prediction power on the Australian dataset, and oppositely there is no use in employing these techniques for the German dataset. The table illustrates that the best outcome is obtained with Min-Max scaling on Australian Credit dataset. It is observed that data scaling is ineffective for RF in literature, however the findings show that transformations influence the algorithm slightly.

Influence of transformations to the computational cost

In order to enrich the research, we also share the cost effectiveness of the applied analyses. We measure the computational time under two costly methods hyper-parameter optimization (GS) and WFS. In terms of SVM, the following, Tables 5 and 6, indicate a dramatical decrease in almost all cases, specifically Natural Logarithm and Min-Max scaling methods deliver consistent savings in both datasets. A similar view is seen for LR algorithm, and the cases that Min-Max and Standard scales are applied to continuous and ordinal variables show stable results. KNN also gain advantage from natural logarithm scaling in both datasets. The scaling provides a decrease most of the cases. Additionally, GNB benefits from standard and Min-Max transformation. However, for the MLP and RF algorithms, the undertaken methods are not convenient in terms of cost reduction.

Table 5

Impact of data scaling in terms of computation time on German dataset
Method	Scale	RF	GNB	SVM	MLP	KNN	LR
	Without	01:12	00:08	05:40:49	01:07	00:09	00:35
	Standard	01:14	00:01	09:38	01:11	00:02	00:23
Grid	Min-Max	01:23	00:01	07:09	01:11	00:02	00:23
Search	Box-Cox	01:13	00:01	08:24	01:10	00:02	00:38
	Natural log	01:12	00:01	06:45	01:11	00:02	00:27
	Min-Max (All At.)	01:13	00:01	08:04	01:17	00:02	00:11
	Standard (All At.)	01:13	00:01	11:23	01:03	00:02	00:07
	Without	02:36	00:28	08:37	25:33	00:42	03:04
	Standard	02:39	00:27	06:15	41:48	00:43	02:18
Wrapper	Min-Max	02:29	00:27	03:50	37:21	00:43	02:45
Feature	Box-Cox	02:42	00:29	06:42	34:06	00:43	03:02
Selection	Natural log	02:30	00:27	04:23	30:28	00:43	02:33
	Min-Max (All At.)	02:56	00:27	01:31	40:19	00:42	01:23
	Standard (All At.)	02:33	00:27	01:41	42:16	00:44	00:55

Table 6

Impact of data scaling in terms of computation time on Australian dataset
Method	Scale	RF	GNB	SVM	MLP	KNN	LR
	Without	00:38	00:01	01:26	00:11	00:43	00:06
	Standard	00:44	00:01	01:26	00:22	01:03	00:08
Grid	Min-Max	00:36	00:01	01:25	00:20	00:56	00:01
Search	Box-Cox	00:36	00:01	01:22	00:19	00:49	00:03
	Natural log	00:38	00:01	00:13	00:19	00:40	00:03
	Min-Max (All At.)	00:37	00:01	01:37	00:21	01:04	00:01
	Standard (All At.)	00:36	00:01	01:20	00:22	01:07	00:01
	Without	01:06	00:16	01:47	11:39	00:25	00:34
	Standard	01:09	00:13	01:37	10:53	00:30	00:22
Wrapper	Min-Max	01:10	00:15	01:49	16:59	00:29	00:29
Feature	Box-Cox	01:09	00:17	00:45	15:31	00:29	00:33
Selection	Natural log	01:01	00:18	01:38	12:08	00:25	00:35
	Min-Max (All At.)	01:12	00:15	00:58	13:37	00:29	00:21
	Standard (All At.)	01:07	00:16	01:41	13:35	00:27	00:19

Impact of wrapper feature selection method

To illustrate the effectiveness of wrapper approach, we compare the results with the untouched ML models in that no parameter optimization and no data transformation is applied. The predictive comparisons are made based on Table 7. As mentioned in section 2, we use ML algorithm as the undertaken algorithm for the wrapper approach. Also, we share which variables are selected during the analysis with two categories. The first category contains the variables that deliver the highest accuracy ratio, and the other one is given for the overall view of mostly chosen features in the trials. The second category is the list of features that appear more than twice in the selection stage. We cannot share the features resulting in the analysis of the Australian dataset since the feature names are not given in the original file.

The credit default risk detection performances are compared as the default of ML algorithm and WFS is applied version of the algorithms (Table 7). The findings suggest that almost all of the algorithms are influenced positively from WFS method. Except for LGBM on Australian dataset (88.9–88.35%), and XGB on German dataset (79.67–78.67%), we see an improvement on all the other cases in terms of prediction (accuracy). Specially, LR (78.1–86.67%), KNN (72.9–80.95%), MLP (74.3–84.76%), GNB (78.6–88.10%), and SVM (54.3–85.71%) gain a dramatical enhancement on Australian dataset. For the other dataset, DT (66.33–72.67%), LR (74.67–77%), and LGBM (72.40–76.40%) are the ones that gains notable default prediction utilization. In terms of the number of attributes, almost all algorithms gain an improvement with considerably less features. Specially, DT (five out of twenty), KNN (eight out of twenty), LGBM (nine out of twenty), and LR (eleven out of twenty) do better with much less attributes (Table 8).

Performance against minority and majority classes

We also investigate default risk capability of the algorithms on minority (default) and majority (non-default) class to show a deeper view. As mentioned in Section 4.1, German dataset has a 30–70% of class distribution which indicates less balanced ratio comparing to Australian dataset (45–55%). Regarding determining a more suitable algorithm on this subject, we consider the criterion balanced accuracy rate calculated as giving equal weights to sensitivity and specificity. In this way, we can see how good the algorithms are at paying attention to both classes.

For such a comparison we consider the cases that parameter optimization is applied to each algorithm. The outcomes of the analyses (Fig. 1) indicates that LGBM (90.46%), CB (89.39%), and XGB (88.95%) are the top three algorithms whereas SVM (73.94%), KNN (77.70%), and MLP (79.35%) are the least successful ones for equally pay attention to both classes on Australian dataset. However, MLP (73.09%) and SVM (70.95%) take their place in the top three with XGB (73.89%), and the bottom three performed algorithms are KNN (61.98%), DT (67.38%), and GNB (68.42%) on the other dataset. Overall, we can summarize the findings as XGB algorithm is the best choice for this credit risk detection task since it is placed in the top three on both datasets, and KNN is not a convenient algorithm for an unbalanced dataset because it is hierarchically placed in the bottom three on both datasets.

Benchmark validation

Now, we illustrate the ultimate performance validations that delivers the best result based on rankings. On Australian dataset, the top approach is the combination of parameter optimization and LGBM (90.60% of accuracy) algorithm, and it is followed by CB (89.50%) and XGB (89.10%) and with GS. The bottom three approaches are shared by the algorithms SVM (54.30% of accuracy), SVM with GS (72.90%) and MLP (74.30). For the German dataset, the most convenient method is the XGB which yields 81.67% of accuracy, and the rest of the top three approaches are MLP with GS (81.00%) and MLP with WFS (79.67%). The least successful three methods are ordered hierarchically DT (66.33% of accuracy), SVM (71.00%), and KNN (71.33%) in the Table 7.

Table 7

Banchmark comparison of ML algorithms on German and Austrian credit dataset
		German					Australian
Model	Accuracy	Sensitivity	Specificity	Mean Ranks	Model	Accuracy	Sensitivity	Specificity	Mean Ranks
XGB-GS	81.67	54.44	93.34	2.33	LGBM-GS	90.60	92.20	88.70	3.33
MLP-GS	81.00	53.33	92.86	3.67	CB-GS	89.50	90.40	88.40	5.00
MLP-WFS	79.67	51.11	91.91	5.00	XGB-GS	89.10	88.00	89.90	5.33
XGB	79.67	48.89	92.86	6.00	LGBM	88.90	89.40	88.40	6.67
SVM-GS	79.33	50.00	91.90	6.33	CB-WFS	88.92	90.10	87.40	7.00
RF-GS	79.00	44.44	93.81	7.00	GNB-WFS	88.10	82.90	92.30	8.00
MLP	78.67	51.11	90.48	7.00	CB	88.10	88.30	87.70	8.00
XGB-WFS	78.67	48.89	91.43	8.00	XGB-WFS	87.14	83.80	89.80	9.00
LR-GS	78.00	46.67	91.43	9.00	DT-WFS	87.14	83.80	89.80	9.00
DT-GS	77.00	43.33	91.43	10.33	LGBM-WFS	88.35	90.60	85.50	9.00
LR-WFS	77.00	45.56	90.47	10.33	DT-GS	87.10	79.50	93.30	9.33
LGBM-GS	76.83	51.67	87.62	10.33	LR-WFS	86.67	82.10	90.40	10.00
CB-WFS	76.90	49.00	88.86	10.67	RF-GS	86.70	87.20	86.30	11.33
CB-GS	76.90	50.00	88.43	10.67	SVM-WFS	85.71	81.20	89.30	12.33
GNB-GS	76.67	47.78	89.05	11.67	LR-GS	86.20	85.50	86.80	12.33
GNB-WFS	76.00	53.33	85.72	12.00	DT	86.20	86.30	86.10	12.67
LGBM-WFS	76.40	50.67	87.43	12.33	XGB	85.70	84.60	86.60	13.67
RF-WFS	75.33	55.56	83.80	12.33	RF-WFS	85.71	85.50	85.90	14.00
GNB	75.33	61.11	81.42	12.33	MLP-WFS	84.76	81.20	87.60	14.33
RF	74.67	48.89	85.72	14.67	GNB-GS	80.50	91.50	71.70	14.67
LR	74.67	41.11	89.05	15.33	RF	83.80	85.50	82.50	15.67
SVM	71.00	7.78	98.09	16.00	GNB	78.60	91.50	68.20	16.00
LGBM	72.40	24.00	93.14	16.00	KNN-WFS	80.95	84.60	78.00	16.67
CB	74.50	42.00	88.43	16.00	KNN-GS	76.20	91.50	64.00	17.33
SVM-WFS	72.67	28.89	91.43	16.67	SVM	54.30	95.70	21.10	18.33
KNN-GS	73.00	34.44	89.53	17.00	MLP-GS	79.10	82.10	76.60	18.67
DT-WFS	72.67	32.22	90.01	17.33	LR	78.10	86.30	71.50	18.67
KNN-WFS	73.33	40.00	87.61	17.67	MLP	74.30	75.20	73.60	21.33
KNN	71.33	38.89	85.23	20.33	KNN	72.90	83.80	64.10	21.33
DT	66.33	42.22	76.66	20.67	SVM-GS	72.90	83.80	64.10	21.33

Table 8

Selected Features for German credit data set using ML methods
Algorithms	More than twice chosen features	Best accuracy
DT	1,4,8,9,12	1,4,8,9,12
KNN	4,8,9,10,12,14	3,4,8,9,10,12,13,14
LGBM	2,5,16	1,2,3,4,5,6,12,16,19
SVM	4,6,8,9	1,2,3,4,6,7,8,9,10,13
GNB	1,2,4,5,6,8,9,11,12,13	1,2,4,5,6,8,9,11,12,13
XGB	1,4,5,7,8,9,10,11,12,13	1,4,5,7,8,9,10,11,12,13
LR	4,5,6,8,10,14	1,2,3,4,5,6,8,9,10,12,14
MLP	1,2,3,4,5,6,7,8,9,11,12	1,2,3,4,5,6,7,8,9,11,12,13,14
RF	1,2,3,4,5,6,8,9,10,11,12,13,14	1,2,3,4,5,6,8,9,10,11,12,13,14
CB	1,2,3,4,5,6,7,8,12,15,16,19	1,2,3,4,5,6,7,8,9,11,12,14,15,16,19,20

Validation of the borrower is a critical part of the banking system. Banks enhance their credit default models to reduce the risk of a decrease in profitability and capital reserves, increase borrowing costs, and sustain their credit ratings and reputation to shareholders. Hence, it is essential for the banks to improve their endurance and eligibility. The purpose of scoring models is to provide statistical probability or ratios to determine whether there is a non-payment risk by revealing hidden links between variables. The ML algorithms delivers enhanced complex models that are superior than the traditional methods. In this paper, we examine several ML algoirthms in a wide range of aspects for decision-making in credit default risk. We analyze the algorithms in the impact of four feature scaling to the prediction powers and computational costs, wrapper feature selection, and imbalanced class distributions, and suggest a road map by constructing the models on two publicly available datasets.

As an ultimate conclusion, the findings suggest that SVM, LR, KNN, and GNB algorithms benefit from the data transformations. Both the prediction power and computational cost of these algorithms are greatly affected by the appropriate technique, which is Min-Max for SVM, the standard scale for LR, natural logarithm for KNN’s computational cost, and Min-Max for KNN’s prediction power, and Min-Max for GNB. Use of WFS improves the methods LR, KNN, MLP, GNB, and SVM dramatically. Hence, WFS can be seen as a silver bullet method for feature elimination. For the unbalanced class distribution problem, XGB is the best choice among the ten algorithms. In addition, the XGB is placed in the top three approaches in both datasets, making the algorithm superior to the rest of the algorithms.

Alpaydin E (2016) Machine learning: the new ai. MIT press
Anderson R (2007) The credit scoring toolkit: theory and practice for retail credit risk management and decision automation. Oxford University Press
Apté C, Weiss S (1997) Data mining with decision trees and decision rules. Future generation computer systems 13(2–3):197–210
Baesens B, Van Gestel T, Viaene S, Stepanova M, Suykens J, Vanthienen J (2003) Benchmarking state-of-the-art classification algorithms for credit scoring. J Oper Res Soc 54(6):627–635
Biswas SK, Chakraborty M, Purkayastha B (2018) A rule generation algorithm from neural network using classified and misclassified data. Int J Bio-Inspired Comput 11(1):60–70
Bolón-Canedo V, Sánchez-Maroño N, Alonso-Betanzos A (2016) Feature selection for high-dimensional data. Progress in Artificial Intelligence 5(2):65–75
Breiman L (2001) Random forests. Mach Learn 45(1):5–32
Breiman L, Cutler A (2015) Random forests. Retrieved from https://www.stat.berkeley.edu/breiman/RandomForests/cc_home.htm (Accessed: 2022-05-06)
Brown I, Mues C (2012) An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert Syst Appl 39(3):3446–3453
Chen F-L, Li F-C (2010) Combination of feature selection approaches with svm in credit scoring. Expert Syst Appl 37(7):4902–4909
Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785–794
Crook JN (1996) Credit scoring: an overview; paper prepared for the british association, festival of sciences, university of birmingham. University of Edinburgh, Department of Business Studies
Dorogush AV, Ershov V, Gulin A (2018) Catboost: gradient boosting with categorical features support. arXiv preprint arXiv :181011363
Dorogush AV, Ershov V, Gulin A (2023) Catboost
Dua D, Graff C (2017) UCI machine learning repository. Retrieved from http://archive.ics.uci.edu/ml
Friedman N, Geiger D, Goldszmidt M (1997) Bayesian network classifiers. Mach Learn 29(2–3):131–163
Gatzert N, Wesker H (2012) A comparative assessment of basel ii/iii and solvency ii. The Geneva Papers on Risk and Insurance-Issues and Practice, 37 (3), 539–570
Grus J (2019) Data science from scratch: first principles with python. O’Reilly Media
Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier
Hancock JT, Khoshgoftaar TM (2020) Catboost for big data: an interdisciplinary review. J big data 7(1):1–45
Ince H, Aktan B (2009) A comparison of data mining techniques for credit scoring in banking: A managerial perspective. J Bus Econ Manage 10(3):233–240
Jayalakshmi T, Santhakumaran A (2011) Statistical normalization and back propagation for classification. Int J Comput Theory Eng, 3
Ke G, Meng Q, Finley T et al (2023) Lightgbm’s documentation
Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Liu T-Y (2017) Lightgbm: A highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst, 30.
Khashei M, Rezvan MT, Hamadani AZ, Bijari M (2013) A bi-level neural-based fuzzy classification approach for credit scoring problems. Complexity 18(6):46–57
Kleinbaum DG, Dietz K, Gail M, Klein M, Klein M (2002) Logistic regression. Springer
Koc O (2019) Comparison of machine learning algorithms on consumer credit classification. M.Sc. Thesis, Middle East Technical University
Lessmann S, Baesens B, Seow H-V, Thomas LC (2015) Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. Eur J Oper Res 247(1):124–136
Li X-L, Zhong Y (2012) An overview of personal credit scoring: techniques and future work. Int J Intell Sci 2(04):181
Liang D, Tsai C-F, Wu H-T (2015) The effect of feature selection on financial distress prediction. Knowl Based Syst 73:289–297
Marcano-Cedeno A, Marin-De-La-Barcena A, Jiménez-Trillo J, Pinuela J, Andina D (2011) Artificial metaplasticity neural network applied to credit scoring. Int J Neural Syst 21(04):311–317
Marqués A, García V, Sánchez JS (2012) Exploring the behaviour of base classifiers in credit scoring ensembles. Expert Syst Appl 39(11):10244–10250
Mueller JP, Massaron L (2016) Machine learning for dummies. John Wiley & Sons
Munkhdalai L, Munkhdalai T, Namsrai O-E, Lee JY, Ryu KH (2019) An empirical comparison of machine-learning methods on bank client credit assessments. Sustainability 11(3):699
Nnamoko N, Arshad F, England D, Vora J, Norman J (2014) Evaluation of filter and wrapper methods for feature selection in supervised machine learning. Age 21(81):33–32
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Duchesnay E (2011) Scikit-learn: Machine learning in Python. J Mach Learn Res 12:2825–2830
Ramasamy S, Rajaraman K, Krishnaswamy P, Chandrasekhar V (2018) Online deep learning: growing rbm on the fly. arXiv preprint arXiv:1803.02043
Raschka S (2015) Python machine learning. Packt Publishing Ltd.
Raschka S, Mirjalili V (2017) Python machine learning. Packt Publishing Ltd.
Rish I (2001) An empirical study of the naive bayes classifier. IJCAI 2001 workshop on empirical methods in artificial intelligence, 3 (22), 41–46
Shen F, Zhao X, Li Z, Li K, Meng Z (2019) A novel ensemble classification model based on neural networks and a classifier optimisation technique for imbalanced credit risk evaluation, vol 526. Statistical Mechanics and its Applications, Physica A, p 121073
Stańczyk U, Jain LC (2015) Feature selection for data and pattern recognition. Springer
Suto J, Oniga S, Sitar PP (2016) Comparison of wrapper and filter feature selection algorithms on human activity recognition. 2016 6th International Conference on Computers Communications and Control (ICCCC), 124–129
Szymanski BK, Zhang Y (2004) Recursive data mining for masquerade detection and author identification. Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop, 424–431
Trevor H, Robert T, Jerome F (2009) The elements of statistical learning: data mining, inference, and prediction. Springer, New York, NY
Tsai C-F (2009) Feature selection in bankruptcy prediction. Knowl Based Syst 22(2):120–127
Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: Practical machine learning tools and techniques. Morgan Kaufmann
Xia Y, Liu C, Li Y, Liu N (2017) A boosted decision tree approach using bayesian hyper-parameter optimization for credit scoring. Expert Syst Appl 78:225–241
Xue B, Zhang M, Browne WN (2015) A comprehensive comparison on evolutionary feature selection approaches to classification. Int J Comput Intell Appl 14(02):1550008
Yeh I-C, Lien C-H (2009) The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Syst Appl 36(2):2473–2480
Yu H, Huang X, Hu X, Cai H (2010) A comparative study on data mining algorithms for individual credit risk evaluation. 2010 International Conference on Management of e-Commerce and e-Government, 35–38
Zheng A, Casari A (2018) Feature engineering for machine learning: Principles and techniques for data scientists. O’Reilly Media, Inc

No competing interests reported.

AppendixA.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Data Transformation and Wrapper Impact on Machine Learning Algorithms in Credit Evaluation

Status:

Version 1

Abstract

Figures

1. Introduction

2. Data Processing, Feature Selection and Parameter Optimization

3. Machine Learning Algorithms

Results

Conclusion

References

Additional Declarations

Supplementary Files

Status:

Version 1