In data mining, classification problems are the most commonly and frequently used problems, which can be best solved using classification and predication applications. Nevertheless, the development of an accurate classification and prediction application relies on the use of an accurate classification algorithm. However, in machine learning area, the researchers have designed a number of classifiers, which can broadly be categorized as decision tree-based (e.g., ID3[1], C4.5[2], and CART[3]), probability-based (e.g., Naive Bayes[4] and AODE[5]), rule-based (e.g., OneR[6] and Ripper[7]), and fuzzy logic based [8], [9], [10], instance-based and ensemble-based (e.g., Bagging, Boosting, and ). Moreover, the research community is continuously working in the area of designing new classifiers [11], [12], [13], and as a result, a number of competing classifiers can be found for a given classification problem with almost equal likelihood to be the best classifier for that problem. Selection of an appropriate classifier from this vast alternatives list is a challenging task. One of the approaches is the empirical evaluation of all the available classifiers on the given classification problem and selection of the classifier with the best results. However, the approach suffers from the problem of exhaustive search, i.e., computational complexity [2], and a number of studies [14], [15] have shown that there is no specific classification algorithm applicable to all classification problems. For example, if the same classifier is used for another problem, it may come up with worst results and hence validates the known theorem of “No Free Lunch” [1]. The reason is that classifiers’ results depend on the characteristics of a given problem and consequently changes from problem to problem, therefore the problem of classifier selection can be viewed as a meta-learning approach [3] [27]. In meta-learning approach, meta-characteristics of the classification problems are computed and the performance of classifiers is measured on these problems. After this, mapping between problem features and classifier(s) with best performance is learned for recommending appropriate classifier [4]. Thus the automatic algorithm selection task using meta-learning is basically a four-fold process model, with the processes enlisted below.
- Classifier characterization - evaluating classifier performance.
- Problem characterization – extracting problem inherited meta-characteristics
- Mapping and learning problems meta-characteristics against classifiers performance
- Recommending appropriate classifier(s) for new problem
Classifier characterization is basically the goal set by the user for development of the application, i.e., accurate or a computationally less expensive classifier etc. It can be measured in terms of classifiers performance metrics using performance evaluation. The research community has characterized classifiers from uni-metric and multi-metrics perspectives. They, sometimes, call it meta-target. Problem characterization is the process of extracting inherit behaviors of the data, which show its unseen nature. It is measured in-terms of meta-features or meta-characteristics of the problem. Research community has extracted different types of meta-characteristics that can be categorized into statistical, information theoretic, model-based, land-marking, and complexity [9]. Recently, Q. Song et al. [10] has used a new dataset characterization method for computing datasets features and computed performance of seventeen classification algorithms over 84 UCI publically available datasets[11]. Mapping meta-characteristics and classifiers performance is the process of aligning each problem against the appropriate classifier. The objective of the process is to make algorithm selection problem as a machine learning problem where meta-characteristics form a feature-vector and label(s) of the classifier(s), with best performance, as the class label. Identification of the class-label is a challenging task and researchers have approached the issue using various approaches, such as multiple comparison method. As a result of these methods, some of the problems have more than one applicable classifiers as the class label. This makes the problem of algorithm selection is a single-class and multi-class problem and research community has approached them using single-label learning and multi-label learning. For learning association or mapping function between problems meta-characteristics and class label(s), researcher have used different approaches that can broadly be categorized – define categories - as decision tree-based learner (e.g., C4.5 [5]), rule-based learner [6], linear regression [7] and instance-based learner (e.g., k-NN [8], [10]). Finally, for the selection of appropriate classifier(s) on the fly, researchers have used different approaches.
Summarizing the research work done so far, the key issues which still need to be worked are the unavailability of training data, use of uni-metric evaluation criteria for classifiers evaluation, and selection of conflicting classifiers, i.e., the classifiers which have equal likelihood of being the best classifiers shall need to be resolved. By looking algorithm selection as a machine learning problem, it is observed that it is impractical and impossible to find a reasonable number of classification datasets to act as training data for building an accurate algorithm recommendation model. At the same time, to set an accurate class label for an instance in the training dataset, it is very hard to select the correct label based on only a single evaluation metric of the classifier performance. The reason is that a classifier perform best on a dataset evaluated using one metric, e.g., accuracy, but poor evaluated on another metric, e.g., time complexity.
Therefore, this research work has proposed an edge ML based CBR methodology to overcome the aforementioned issues. The key contributions of the research work are enlisted as follows: (a) design of an incremental learning framework using edge ML-based CBR methodology, (b) design of a multi-metrics classifiers evaluation criteria, (c) design of an efficient algorithm conflict resolution criteria (ACR) and (d) implementation of the CBR methodology integrated with ACR to automatically select an appropriate classifier for a new classification problem.
In this paper, the idea of Case-based Mata-learning and Reasoning (CB-MLR) framework is extended by introducing the concept of multi-metrics criteria for classifiers evaluation and integrating it with CBR and Classifier Conflict Resolution (ACR) methodologies. The CBR methodology incrementally learns mappings of problems’ meta-characteristics and labels for the best classifiers. For the problems meta-characteristics, general, basic statistical, advanced statistical, and information theoretic characteristics are considered. For choosing label of the best classifier(s), a multi-metrics performance evaluation criteria consisting of classifiers’ accuracy and consistency are considered. Similarly, for the mapping purpose of problems characteristics and best classifiers labels, a propositional feature-vector scheme is used. To learn mappings, the CBR incremental learning methodology is adopted which ultimately recommends the appropriate classifier for a given new classification problem. Hence, the key contributions of the research work are enlisted as follows.
-
Extends CB-MLR to a flexible and incremental meta-learning and reasoning framework using edge ML and CBR-based methodology, which is integrated with multi-criteria decision making, for classifier evaluation, and data characterization using multi-view meta-features extraction.
-
Extending the conventional ML algorithm section problem to edge ML-based methodology for efficient and faster recommendation with least computational resources utilization.
-
A new multi-metrics criteria is proposed for the evaluation of decision tree classifiers to select the best classifier as class label for the cases in training dataset (i.e., resolved cases in the proposed CBR methodology). Classifiers are analyzed based on their predictive accuracy and standard deviation, called consistency to select the best classifier as class-label.
-
The idea of multi-view learning is proposed to learn the data from multiple perspectives, with each perspective representing a set of similar meta-features that reflects one kind of behaviors of the data. Each set of features is called a family that forms a view of dataset.
-
Proposed a CBR-based meta-reasoning methodology with a flexible and incremental learning model integrating CBR with the algorithm conflict resolving (ACR) method to accurately recommend the most similar case as the suggested classifiers for a given new dataset. For the CBR retrieval phase, accurate similarity matching functions are defined, while for the CCR method, weighted sum score and AMD method
The remaining of this paper is organized as follows. Section 2 briefly overview edge ML computing for algorithm selection. Section 3 describes the edge ML and CBR-based methodology. Section 4 describes the implementation and evaluation of the proposed methodology. Section 5 concludes the work done.