Interpretation Conclusion Stability of Software Defect Prediction over Time

doi:10.21203/rs.3.rs-3996923/v1

Model instability refers to where a machine learning model trained on historical data becomes less reliable over time due to Concept Drift (CD). CD refers to the phenomenon where the underlying data distribution changes over time. In this paper, we proposed a method for predicting CD in evolving software through the identification of inconsistencies in the instance interpretation over time for the first time. To this end, we obtained the instance interpretation vector for each newly created commit sample by developers over time. Wherever there is a significant difference in statistical distribution between the interpreted sample and previously ones, it is identified as CD. To evaluate our proposed method, we have conducted a comparison of the method's results with those of the baseline method. The baseline method locates CD points by monitoring the Error Rate (ER) over time. In the baseline method, CD is identified whenever there is a significant rise in the ER. In order to extend the evaluation of the proposed method, we have obtained the CD points by the baseline method based on monitoring additional efficiency measures over time besides the ER. Furthermore, this paper presents an experimental study to investigate the discovery of CD over time using the proposed method by taking into account resampled datasets for the first time. The results of our study conducted on 20 known datasets indicated that the model's instability over time can be predicted with a high degree of accuracy without requiring the labeling of newly entered data.

Just-in-time SDP

JIT-SDP

Concept Drift Detection

Incremental Learning

explainable artificial intelligence

XAI

Developers use software testing methodologies to identify software defects, which is both time-consuming and costly. For this reason, researchers have implemented diverse Software Defect Prediction (SDP) models, which diminishes the necessity for comprehensive software testing and consequently conserving time and resources. However, if practitioners and developers do not understand why the SDP model produces a given decision, they often do not trust it. To this end, some models, such as linear regression, decision trees, and naive Bayes, have understandable structure. However, complex models, mostly applied for prediction due to better performance, are unintelligible and treated in the same way as black boxes, e.g., neural networks, random forests, and support vector machines. For these models, post-hoc interpretation algorithms have been introduced by explainable artificial intelligence (XAI) techniques such as LIME [2] to provide human-understandable explanations for ML models’ decisions. Several studies have investigated the consistency between the model interpretation and the instance interpretation, as well as the inconsistency among the output of interpretation algorithms on different datasets of Just-In-Time SDP (JIT-SDP) [3–6] that is the concept drift (CD) of the Source Component Shift (SCS) type. In SCS, the collected data comprises multiple sources, each with their own characteristics [7, 8]. In reality, in SCS, the interpretations of data in various datasets exhibit dissimilarities, stemming from the inconsistent concepts across these datasets. Consistency is the degree to which something remains unchanged or steady in its behavior, or performance. It implies stability and predictability. The CD points identified in the data stream can also be employed for predicting the instability of model performance. This refers to the phenomenon where the statistical properties of a data stream change over time [9]. If CD is not detected, the model may become less accurate over time, leading to false positives or false negatives [10]. Therefore, there is a need for concept drift detection (CDD) methods that can identify when CD is occurring and adjust the model accordingly. Additionally, Inaccurate JIT-SDP models may result from imbalanced datasets where the ratio of clean and defective modules is not equal [11]. To mitigate the impact of class imbalance, previous studies have utilized class rebalancing techniques [1]. Motivated from these observations, in this paper, we have proposed a new approach for discovering CD points in evolving software defect predictions in sync with the arrival of new unlabeled data by discovering concept change over time. In summary, this paper made the following contributions to the best of our knowledge:

Previous papers have identified inconsistencies in interpretation among different models and datasets. This paper aims to captures the inconsistency of data interpretation over time.
Previous studies have discovered CD in the SDP problem with methods of monitoring performance degradation. This paper aims to explores CD based on concept change and instability in instance interpretation within the JIT-SDP problem. It is important to note that CD fundamentally refers to a change in concept.
Previous studies have detected CD through ER monitoring. We have also obtained it through the monitoring measures other than ER like four threshold-dependent measures namely Precision, Recall, Gmean, and F-measure, as well as three independent measures: Area Under the Curve (AUC), Matthews correlation coefficient (MCC), and ER [4].
Our method for CD discovery does not require the labeling of new data, unlike the baseline methods [12, 13].
In order to conduct a more accurate evaluation, we have compared our method with the outputs of 7 baseline methods using statistical tests.
This paper has also conducted an experimental study aimed at identifying a suitable interpretation algorithm for CDD using the proposed method.
Since our work also deals with datasets that have class imbalance issues, we have employed a resampling technique and evaluated its impact on the output the proposed CDD method.
This paper has conducted a study that compare the output of the CDD method based on simplified and resampled data with baseline methods that rely on threshold-dependent and threshold-independent criteria.

The rest of this paper is organized as follows. Section 2 provides an overview of previous works related to this study, Sections 3 and 4 describe the proposed method and results respectively, Section 5 presents the threats of the validity of this research. We have stated the conclusion and future work in the last section.

In this section, we briefly discussed previous related research in various fields, including concept drift, interpretation techniques, and rebalancing studies. Furthermore, we have examined the application of these fields in SDP.

2.1 Interpretation Techniques

Interpretation techniques refers to the process of understanding and explaining how a machine learning model makes predictions or classifications. For this purpose, the impact score of each feature is obtained for the model's decision-making (model interpretation) or on the prediction result of an instance (instance interpretation). The output of the interpretation algorithm can be coefficients or weights assigned to each feature in a linear regression or logistic regression model, analyzing the decision boundaries or impact score of each feature in a tree-based model like random forests or gradient boosting, or using techniques like partial dependence plots and SHAP values to understand how individual features contribute to predictions in more complex models like neural networks [14–16]. Zheng et al. has also used the model interpretation algorithm to extract the appropriate features to increase the efficiency of the SDP model [6]. They have achieved high efficiency with less effort. The main goal of model interpretation is to gain a deeper understanding of how a model works and why it makes certain predictions. This can help improve the accuracy, identify potential biases or errors, and build trust with stakeholders who may be relying on its results. Recent literature has proposed model-agnostic techniques for interpreting the prediction of black-box AI/ML algorithms by detecting the contribution of each metric (a feature of a sample) in predicting the class label of an instance [17]. Jiarpakdee et al. have experimentally evaluated three model-agnostic techniques: two advanced techniques of Local Interpretability Model (LIME) [18] and BreakDown technique [19] in their study. LIME utilizes a local regression model around the instance to calculate the contribution of each metric in predicting the instance class. On the other hand, BreakDown breaks down the prediction of the instance label into parts for interpretation, displaying the contribution of each metrics separately in predicting the target class [17]. The study by [20] utilizes the Interactions-based Method for Explanation (IME) algorithm to describe predictors and classifiers. The IME algorithm returns three types of interpretation vectors at the model level: positive effect size that features have on the target class label, negative effect size, and average effect size. In the relevant paper, it is stated that this interpretation algorithm works independently of the machine learning model and makes it compatible with any classifier. Therefore, we have also used this interpretation algorithm in our proposed method.

2.2 Concept Drift Detection

CDD methods are commonly used in streaming data analysis problems, which are complex to learn and analyze [21–24]. Gangwar and Ekanayake et al. [24, 25] examined the impact of CD on the performance of the SDP model. In JIT-SDP, CD can occur when the characteristics of software defects change over time due to changes in the software development process, changes in the software architecture, or changes in user behavior [3]. Moreover, CD can have a significant impact on the accuracy of JIT-SDP models. Through examining more than 15 chronological datasets, Krishna and Menzies [26] concluded that the accuracy of the SDP model changes over time [27]. The study by [28] used the DDM strategy [29] to detect CD in time-ordered defect datasets and employed the chi-square test with Yates continuity correction [30] to evaluate it. Kabir et al. utilized the DDM method at the inter-version level for CD detection [31]. As previously mentioned, CD can cause changes in interpretation and stability of model performance, where stability refers to consistent performance over time, even after a year [32]. Bangash et al. study aimed to examine the stability of performance measures of inter-project SDP models such as G-Measure, MCC, AUC, and f-score while considering version order since models should not train on future knowledge to predict past defects. Overall, detecting CD is crucial for ensuring model stability and reliability over time, particularly in real-world scenarios where data distributions are likely to change frequently.

2.3 The Effect of Rebalancing Techniques on Performance and Interpretation

The performance and interpretation of an SDP model heavily depend on the dataset used. SDP models trained on imbalanced datasets are highly prone to produce incorrect predictions. The proportion of defective and clean components differs in imbalanced datasets, resulting in poor identification of data containing defects. Previous studies have utilized class rebalancing techniques to reduce the effect of class imbalance in the data by generating equal numbers of defective and clean components before building defect prediction models [1]. Many studies have shown improved performance when using rebalancing techniques, such as Seiffert [33, 34], who demonstrated that AUC can improve by up to 40% by applying rebalancing techniques. Tantithamthavorn et al. [1] conducted a study on the impact of four commonly used rebalancing techniques, namely SMOTE, ROSE, over-sampling, and under-sampling, on the performance and interpretation of SDP models trained with seven well-known classifiers. They then evaluated the performance of these models using seven well-known measures: three threshold-independent measures (e.g., AUC) and four threshold-dependent measures (e.g., Precision, Recall). They found that the impact of rebalancing techniques on SDP models’ performance over time varies depending on the classification algorithm and evaluation metrics used. However, AUC remains unaffected by these techniques and should be considered a standard measure for evaluating SDP.

2.4 Evaluation Methods

In this section, we discuss the evaluation methods and criteria presented in previous related studies. The literature has considered the identification of CD points using monitoring ER trend and detected points of ER that have changed statistically significant using Page-Hinckley (PH) statistical algorithm [12]. The method was abbreviated "ER PH" by Gama. Demšar and Li et al. used this method to detect concept drift as a baseline method [20, 35]. Similarly, we have also used this baseline method to evaluate our work. Gama has conducted a comparison of three graphs depicting the trends of ER, cumulative ER, and the PH test implemented on ER trend over time for CD detection [12]. In these graphs, the slope change shows the CD, which is more noticeable in the graph of the PH test implemented on ER trend. Ekanayake [36] also utilized AUC prediction for change detection. McIntosh et al. [3] investigated the effect of fluctuating feature values on the performance and interpretation of the JIT-SDP model by monitoring the efficiency of AUC and Brier Score performance measures. Bangash et al. [32] evaluated the performance stability of cross-project defect prediction models by monitoring measures such as G-measure, MCC, AUC, and F-Score in different periods. Kabir et al. [37] assumed that previous releases of the software are labeled (clean or defective) to train inter-release DP models (IRDP), evaluating their strength against discovering CD using AUC, Recall, and pf performance measures.

In this section, we described the design of our proposed approach and experimental studies. As previously mentioned, the aim of this study is to propose a CDD method using the identification of inconsistencies in instances interpretation over time on change level software defect datasets (commit). Previous studies have successfully identified CD points based on performance loss. However, CD inherently signifies a change in the concept. Consequently, it is also possible to detect CD by tracking and analyzing changes in distribution or interpretation of data characteristics. Additionally, the output of the interpretation algorithm aids developers and practitioners in comprehending the root cause of the defects in different time periods. Furthermore, it should be noted that conventional approaches for CDD necessitate the availability of labeled data. However, our proposed method operates on unlabeled data, enabling it to predict CD points rather than merely detecting them. It is not necessary to wait for data labeling in order to make predictions. However, in order to make a diagnosis, we must await the accurate labeling of the newly entered data. The process of labeling can take a significant amount of time, resulting in a delay in detecting CD. Consequently, we considered the methods based on monitoring the model's performance over time, which require data labels, as the baseline method. For evaluation, we compared the results of our proposed method with those of the baseline methods.

3.1. Constructing Baseline Methods

In this section, we discuss the baseline methods that are used to evaluate our work. The running process of the baseline method is illustrated in Fig. 1.

The baseline method employed in this study derives CD points from labeled data using monitoring ER over time. Specifically, Whenever the ER value is significantly different from its previous values, that particular point is identified as CD. The determination of this significant distance between consecutive values is accomplished through the PH statistical test. Given that our experiment is conducted on simplified and resampled data, in order to further investigation, we have acquired CD points through error rate monitoring of three threshold-independent performance measures and four threshold-dependent measures, in addition to ER monitoring. These performance measures are obtained from the incremental classifier of the Random Forest (RF) machine learning algorithm. The RF algorithm has been previously introduced as a reliable classifier in prior studies on SDP models and has been consistently utilized [6, 38–40]. It is worth noting that future research may explore alternative machine learning algorithms, such as Naive Bayes or regression models.

Due to the fact that the PH algorithm is able to detect the change on the error rate values over time, not the trend of the performance measures themselves, we have implemented this algorithm on the error rate of the performance measures of the machine learning model. Specifically, we identified CD points by monitoring the error rates of various criteria of the RF algorithm over time using the PH algorithm as described in [41] study. A change in the error rate of learning model performance measures indicates a CD. Hypothetically, monitoring the error rate related to the Accuracy measure using PH algorithm is abbreviated as ER PH [12]. To evaluate model stability, different papers use varying performance measures of machine learning algorithms over time. In this study, we derived CD points using error rate monitoring of three threshold-independent performance measures (ER, AUC, and MCC) and four threshold-dependent measures [1], namely Recall, F Measure, Precision, and Gmean. We abbreviate ER PH, AUC-Er PH, MCC-Er PH, Recall-Er PH, F Measure-Er PH, Precision-Er PH, and Gmean-Er PH. We considered CDD method based on the error rate monitoring of these performance measures as the baseline methods. The formulas for these performance measures, along with their error rate formulas, is provided in Table 1.

Table 1

The formula for performance measures, along with their error rate formulas
Performance measures	Error rate of performance measures
Accuracy = \(\frac{tp+tn}{tp+tn+fp+fn}\)	ER = 1 - Accuracy
AUC : The values of AUC range between 0 (worst), 0.5 (no better than random guessing), and 1 (best)	AUC-ER = 1 – AUC
MCC = \(\frac{tp.tn-fp.fn}{\sqrt{\left(tp+fp\right)(tp+fn)(tn+fp)(tn+fn)}}\)	MCC-Er = 1 - MCC
Recall = \(\frac{tp}{tp+fn}\)	Recall-Er = 1 - Recall
precision = \(\frac{tp}{tp+fp}\)	precision-Er = 1 - precision
F-Measure = (2 Precision * Recall) / (Precision + Recall)*	F Measure-Er = 1 - F Measure
Gmean = \(\sqrt{ \frac{tp}{tp+fn}.\frac{tn}{tn+fp}}\)	Gmean-Er = 1 - Gmean

Where tp and fp are the numbers of the true and false positives respectively, whereas, tn and fn are the numbers of the true and false negatives. The AUC measure is a threshold-independent performance measure that evaluates the ability of models in discriminating between defective and clean instances. We employed incremental learning to obtain performance measures over time. During the process of incremental learning, samples are gradually incorporated into the training data, and the model is updated after each chunk of data is added (as depicted in Fig. 2). In Fig. 2, according to our experiment, the training data indicated by the orange box can be simplified or resampled.

Why did we divide the data into chunks of hundreds? Training samples generated in evolving software JIT-SDP problems often face the challenge of verification latency (VL). For instance, a software commit is labeled as either defect when a defect is found or clean after a specifically defined time has passed (whichever comes first). If no defects are found during this waiting period, the change is labeled as clean [42]. Therefore, any training sample that is not identified as defective according to the VL is available as a clean sample for model training after a specific time [43]. According to Cabral, Minku et al., "Defect discovery delays ranged from 1 day to over 11 years, and medians were typically close to or lower than 90 days. A waiting time of 90 days can be considered to provide a good trade-off between correct labeling and CD". For this reason, we considered this distance as VL between training and test data. Based on our investigations, we have found that the time interval between each 100 samples in our dataset is approximately 90 days or more.

3.2. Presentation of Proposed Methods

The purpose of this section is to provide a detailed description of the proposed method. The running process of the proposed method is illustrated in Fig. 3.

In order to examine the consistency of the instance interpretation results over time, we partitioned the data into sequential chunks within specific time intervals. Subsequently, we performed preprocessing on each data chunk and generated an instance interpretation vector for every individual sample. As previously mentioned, to investigate the effect of resampling on CDD performance, we examined the proposed method on both simplified and resampled training datasets. For example, the pseudocode for interpreting testdata samples using the IME algorithm in R is as follows.

Explain <-

explainVis(modelRF, trainData, testData, method="IME", visLevel="instance")

Where the variable Explain represents a two-dimensional vector. Its rows correspond to the number of samples in a chunk, while its columns represent the interpreted values of the features. These interpreted values indicate the impact of each feature on predicting the label of each sample. modelRF is the model trained on the trainData dataset. This training dataset can be simplified or resampled. In experiments, we examined simplified data both in raw (uninterpreted) and interpreted form. Then, we employed the MANOVA technique (Multivariate Analysis of Variance) to determine the chunks that exhibit significant differences in comparison to the preceding chunks. MANOVA is an extension of ANOVA (Analysis of Variance) that analyzes the relationship between multiple dependent variables and one independent variable (one-way MANOVA) [44]. In this paper, the dependent variables consist of the interpreted features of each sample, while the independent variable refers to the number of data groups that are chunked in chronological order (the first hundred samples are categorized as group 1). Then we identified the CD points using the PH change detection algorithm.

In order to demonstrate the accuracy of our method, it is imperative to use the suitable interpretation algorithm. Frequent interpretation algorithms, IME [45], LIME [18], and SHAP [46], explain outcomes for every individual instance. To this end, they measure the difference in prediction between the original instance and its neighboring. Vreš et al. demonstrated that IME is the most robust of all three tested interpretation algorithms [2]. Jiarpakdee [17] asserted that instance interpretation with the BreakDown algorithm has better stability than the famous LIME algorithm in SDP problems. Therefore, we experimentally compared the proposed method based on two interpretation algorithms, BreakDown (BD) and IME, to detect CD. First, vectors of the importance of features for every sample are extracted by instance interpretation algorithms. Then, in the proposed method, CD pointes are detected by deriving significant differences among these vectors. We compared the results of method based on these two interpretation algorithms (based on interpreted data) with each other and with the case where no interpretation was used (based on raw data). We named the methods based on these three types of data IME_base (the method based on IME algorithm), BD_base (the method based on BreakDown algorithm), and raw_base, respectively. Additionally, we compared these three methods with the interpretation-based methods performed on resampled data (rIME_base and rBD_base). If changes in ER of various performance measures are considered as original CD points, this work shows that IME performs better than BreakDown in detecting CD points because its output is more consistent with the CD points discovered from the baseline method. In conclusion, the matching rate of the proposed methods are compared with the baseline methods in terms of accuracy, sensitivity and robustness.

3.3. The Implementation of the Proposed Method on Resampled Data

As previously mentioned, we proposed a method to explore CD points by examining the consistency of interpretation over time. However, the dataset is imbalanced and causes performance loss of the training model. Moreover, it may affect the interpretation output [47]. Therefore, we employed resampling technique to tackle the challenge of imbalanced class distribution in the software defect training datasets. However, balancing the data changes the concept, leading to a shift in the model interpretation and causing CD [1]. As a result, it is not justifiable to employ model interpretation techniques on a model trained using resampled data. To address this problem, we used instance interpretation. In an empirical study on 20 datasets, we evaluated the effect of data resampling on the results of our proposed CDD method using baseline methods. In the results section, it will be demonstrated whether it is more advantageous to use simplified data or resampled data.

In this section, through a case study on 20 datasets of open-source projects we have evaluated the following research questions:

RQ1 How does the proposed method based on different interpretation algorithms perform compared to baseline methods for detecting CD?

RQ1: Given that one of the key advantages of our proposed method is its ability to locate CD on unlabeled data, we have developed an experimental study of the method based on two algorithms of instance interpretation and evaluated them using 7 baseline methods. We have reached the conclusion that our proposed method, which is based on detecting changes in instance interpretation, aligns well with the results obtained from baseline methods based on labeled data. Additionally, we concluded that among the interpretation algorithms introduced in literature, the use of the IME algorithm enhances the efficiency of the CDD method in comparison to the breakdown interpretation algorithm. Even in the interpretation-based method on resampled data, IME_based method still demonstrated superior performance.

RQ2 How does the proposed method based on resampled data perform compared to the baseline methods that rely on threshold-dependent and threshold-independent criteria?

4.1 Experimental Setup

In this section, we presented our experiment setup. Specifically, we described experimental parameters and tools used for implementation, indicated the benchmark datasets and preprocessing steps used in our experiments.

4.1.1 Experimental Parameters and Comparable Tool

In this study, we employed the default parameters for the used algorithms, except for the PH algorithm. When we used default parameters for the PH change detection algorithm, we encountered a challenge where CD occurs on all datasets at almost the same points. Hypothetically, a standard CDD method detects CD points on a stream containing 500 observations. Let us assume that CD points are identified at five different points. We want to determine whether these are false positives or actual CD points detected by the detector. If one CD is detected every 100 observations, all detected points are likely false positives. However, if a detector identifies CD points at different intervals, we can conclude with some degree of confidence that the detected CD points are likely genuine [48]. Our approach also controls false positive detection rates and enhances the CDD method performance by adjusting parameters. The default parameters for PH are a warning threshold (corresponding to the allowable false alarm rate) β = 0.01, sensitivity (acceptable error rate) δ = 0.001, and fading factor (update weight for historical values in the PH statistic) γ = 0.999 [12]. By configuring the parameters of the PH algorithm with these default values, our proposed method detected CD points at equidistant intervals. This indicates that these points are indeed false positives rather than genuine CD points. To address this issue, we increased the false alarm value of the PH algorithm to β = 0.1 and recommend checking the CD using other values of this parameter. This prompted us to explore larger parameter values for optimal results. For other domains with higher data entry rates, different parameter values may be useful.

The implementation of the studied model-agnostic techniques is ready-made in R (IME² and BreakDown³). We applied the SMOTE technique using the R package implementation of the SMOTE function [49].

4.1.2 Dataset

In this study, we present a proposed CDD method based on unlabeled data using examining instance interpretation consistency over time for the evolving JIT-SDP problem. Our study addressed the RQs by utilizing 20 GitHub repositories that were previously used in other studies (Lin, Tantithamthavorn et al. 2021). Lin et al. employed this dataset to derive the most important features in the JIT-SDP problem and investigated the consistency and stability of prediction performance across different datasets and models. Additionally, this dataset has two key characteristics: public availability and long-term development, making it an ideal resource for our study. Table 2 provides a summary of this dataset. The dataset contains commit level metrics that are divided into three categories (Table 3).

Diffusion category figures out how distributed a commit is. A heavily distributed commit is more defect-prone, as shown in [50, 51]. Measuring the diffusion of a commit is produced by collecting the number of modified subsystems (NS), the number of modified directories (ND), the number of modified files (NF), and the distribution of modified code across each file (Entropy). Due to differences in the number of files across changes, we normalized the entropy by the maximum entropy log₂n similar to Hassan [51].

Size category figures out the commit size utilizing the lines deleted (LD), lines added (LA), and lines total (LT).

Purpose category figures out whether a commit has fixed a defect. Research has demonstrated that a commit that fixes a defect is likely to cause another defect [52, 53].

4.1.3 Data Preprocessing

The datasets used in this study contain commit-level metrics that are skewed and vary in scale, with some metrics showing high correlation. To address this problem, we followed a similar approach to the work of Lin et al. [4]. Specifically, we centered and scaled the values of commit-level metrics using the scale function in R, except for the "FIX" metric which is Boolean. Moreover, because correlated and inconsistent features cause interpretation inconsistencies, it is necessary to remove correlated metrics in the pre-processing stage [54]. Jiarpakdee et al. have shown that the Spearman correlation method removes correlated metrics significantly better than feature selection techniques [55]. Therefore, we used the same method to remove correlated features with correlation coefficient greater than 0.7. Additionally, we normalized the entropy log₂ⁿ by the maximum entropy log₂ⁿ to consider differences in the number of files across changes, as suggested by Hassan [51]. We used the Synthetic Minority Oversampling Technique (SMOTE) proposed by Chawla et al. [56] for class rebalancing to overcome the limitations of traditional oversampling and undersampling techniques. This technique generates artificial data based on feature space, rather than data space, and combines synthetic oversampling of minority defective modules with undersampling of majority clean modules. The SMOTE technique has an adjustable parameter that needs to be specified. However, Tantithamthavorn et al. demonstrated that changing the parameter of the SMOTE technique does not affect their findings [1]. The literature employed SMOTE to create balanced class distribution. They found that SMOTE heavily improved the performance of SDP with more consistent results than that of SDP without the balance process [11, 57]. Therefore, we applied the SMOTE technique for resampling and we left the review of other resampling methods to future works.

Table 2

Summary of software projects from the studied dataset (The percentage of defect commits, compared to clean ones, is shown in parentheses) [4]
Project name	Date of first commit	Lines of code	# of changes
accumulo	Oct 4, 2011	600,191	9,175 (21%)
angular	Jan 5, 2010	249,520	8,720 (25%)
brackets	Dec 7, 2011	379,446	17,624 (24%)
bugzilla	Aug 26, 1998	78,448	9,795 (37%)
camel	Mar 19, 2007	1,310,869	31,369 (21%)
cinder	May 3, 2012	434,324	14,855 (23%)
django	Jul 13, 2005	468,100	25,453 (42%)
fastjson	Jul 31, 2011	169,410	2,684 (26%)
gephi	Mar 2, 2009	129,259	4,599 (37%)
hibernate-orm	Jun 29, 2007	711,086	8,429 (32%)
hibernate-search	Aug 15, 2007	174,475	6,022 (35%)
imglib2	Nov 2, 2009	45,935	4,891 (29%)
jetty	Mar 16, 2009	519,265	15,197 (29%)
kylin	May 13, 2014	214,983	7,112 (25%)
log4j	Nov 16, 2000	37,419	3,275 (46%)
nova	May 27, 2010	430,404	49,913 (26%)
osquery	Jul 30, 2014	91,133	4,190 (23%)
postgres	Jul 9, 1996	1,277,645	44,276 (33%)
tomcat	Mar 27, 2006	400,869	19,213 (28%)
wordpress	Apr 1, 2003	390,034	37,937 (47%)

Table 3

Summary of commit level metrics [4]
Category	Name	Description
Diffusion	NS	Number of modified subsystems
	ND	Number of modified directories
	NF	Number of modified files
	Entropy	Distribution of modified code across each file
Size	LA	Lines of code added
	LD	Lines of code deleted
	LT	Lines of code in a file before the commit
Purpose	FIX	Whether or not the commit is a defect fix

4.2 Experimental Results

In this section, we discuss our findings for each RQ.

4.2.1 RQ1 How does the proposed method based on different interpretation algorithms perform compared to baseline methods for detecting CD?

In this paper, we aim to discover CD by exploring the inconsistency of the instance interpretation over time. For this purpose, we obtained the instance interpretation vector for each newly generated sample. The instance interpretation vector comprises the contribution of every individual feature-value in predicting each sample. Zheng has acquired the most important features for more accurate detection of software defects through model interpretation. It has been demonstrated that by extracting these features, it is feasible to achieve 96% of the original efficiency by using 45% of the previous effort [6]. This observation led us to infer that changing these important features over time will result in CD and a subsequent decline in model performance in the near future, a matter that has been disregarded by SDP studies. Hence, through examining the significant change in the distance between consecutive model interpretation vectors over time, it becomes feasible to approximate the occurrence of CD. However, the challenge of model interpretation lies in its dependence on data labeling. As a result, it is not feasible to predict CD for the new incoming data lacking labels. As previously mentioned, the instance interpretation does not require a data label. We have provided the instance interpretation of two samples in Fig. 4. The subfigure. on the left shows that the sample is defective with a probability of 95% (red box). The effect size of each feature with a specific value on the sample class prediction is depicted by the blue and orange bars. For example, the effect size of the nld feature with the specific value 2.6 on the prediction, with a 95% probability belongs to the defect class, close to 0.2, which is more than other features. In general, positive effect sizes are represented by the blue bars, while negative effect sizes are indicated by the orange bar.

We partitioned the data vectors into chunks based on chronological order and derived the instance interpretation vector of each sample. In cases where the distribution of a data chunk exhibits a statistically significant difference from the preceding data chunks, it is identified as an inconsistency. This inconsistency is detected through the MANOVA algorithm. Then, in order to ascertain its validity and eliminate the possibility of randomness, the PH algorithm is employed. Finally, inconsistencies that are deemed valid (as determined by the output of the PH algorithm) are recognized as CD. In the JIT-SDP, there is no dataset with labeled CD points. Therefore, for evaluation, we compared the output of our work with the output of known baseline methods. These baseline methods were fully described in the previous section. Finally, in order to address the research questions, we evaluated the outcomes of our proposed methods using performance measures of the CDD method. According to Bifet, appropriate evaluation measures for detecting CD are outlined below [58, 59]:

CDD_Accuracy: The accuracy of the proposed method for detecting CD points.
Missed Detection Rate (MDR): The possibility of not receiving a warning when there is a CD by calculating the ratio of unknown CDs to complete CDs. a suitable detector would have small or zero MDR. It is obtained using Eq. (1).

MDR = 1- CDD_Accuracy

(1)

Mean Time to Detection (MTD): The average delay between the detection of a CD point and its original CD occurrence location. A suitable detector should have a small MTD value.
Mean Time between False Alarms (MTFA): This measure determines the duration of a false alarm in the absence of any CD occurrence. A suitable detector should have a high MTFA value.
Mean Time Ratio (MTR): This is a measure of the trade-off between sensitivity and robustness. A suitable detector should have a high MTR value which can be obtained through Eq. (2):

MTR = \(\frac{MTFA}{MTD}\times\)(1 - MDR)(2)

We calculated CDD method performance measures to evaluate the proposed method. The evaluation method has been indicated in Fig. 5. Considering what interpretation algorithm we have used for the proposed method or the implementation has been performed on uninterpreted raw data, three designations were utilized: IME_base, BD_base, and raw_base. In the event that the method is executed on resampled data, the designations rIME_base and rBD_base are employed. These methods are elaborated upon extensively in Section 3.2. Finally, we compared the results of the methods using the Friedman test. The Friedman test [60] is a statistical analysis employed to assess and contrast multiple sets of results. It is important to note that the minimum rank serves as the benchmark for evaluating performance in this particular test. In summary, our method involves the following steps:

Dividing the test data into groups of 100 for each of the 20 datasets as illustrated in Fig. 6 (As stated in Section 3.1). For example, if a dataset contains 9000 commits and we consider the test dataset to be 100, we would have 90 data groups.
Data preprocessing, such as the removal of correlated features and rebalancing the class distribution.
Deriving instance interpretation vector for each sample of test data. To this end, the contribution of each metric in predicting the instance class is calculated.
Identifying the test data groups that exhibit significant statistical differences with their previous groups using the MANOVA method. MANOVA analyzes the relationship between multiple dependent variables and one independent variable. Groups are considered independent variables, whereas software metrics (or data features) as dependent variables. A p-value less than 0.05 indicates a significant difference in distribution between the data within a group compared to the preceding groups. The process of this identification is shown in Fig. 7. These differences are presented in the table within Fig. 7. Each column in this table represents a feature, while each row corresponds to a test data group. The value within each cell signifies that the feature values within that group significantly differ from those of previous groups in terms of statistical distribution. Hypothetically, the statistical distribution of the "FIX" feature vector in the 29th group significantly differs from that of two of the previous groups. We calculated the sum for each row in this table and included it as the last column (sum column).
Detecting CD points using PH test: In the "sum" column, the row in which the values are significantly larger than the previous rows indicates the occurrence of CD. Wherever consecutive rows in the "sum" column have larger numbers than before, it indicates that this difference is non-random and significant. We located these phase difference points using the PH change detection algorithm. If only one row has a considerable difference, it may be a coincidence. For example, group 29 values in the "sum" column are increasing, representing a concept change.
Instead of implementing the PH algorithm to detect a significant change in the "sum", we can apply it to each individual feature. So, it is feasible to investigate the significant change of which feature has a more substantial impact on the occurrence of CD. We defer this notion to be explored in future research.
Identifying CD points using proposed method for all 20 datasets on uninterpreted raw and interpreted data.
Collecting values of performance measures of the RF trained model over time for each dataset, such as ER, AUC, Gmean, Precision, F-measure, Recall, and MCC. These measures are commonly used in research to assess the stability of the SDP model.
Identifying CD points for each of the 20 datasets by monitoring each of these criteria using the PH statistical test (baseline methods).
Calculating the performance measures of the evaluation of CDD methods including Accuracy, MDR, MTD, MTFA, and MTR (for evaluating results of our method (step 6) compared to results of the baseline method (step 8))
Evaluation of the proposed method by determining the degree of concurrence with the output of the baseline method using the Friedman test.

This was an overview of the design process. In the following, we provided an illustrative example for better understanding. For example, the ER PH baseline method identifies points of CD occurrences as presented in Table 4. Figure 8 displays the complete output of column "sum" (the table within Fig. 7) on the WordPress dataset. Table 5 displays the CD points obtained through applying the PH algorithm on the output of column "sum". Figure 9 illustrates the segmented diagram of Fig. 8, highlighting changes of concept in these areas that correspond to CDs obtained from the baseline method (Table 4).

Table 4

Discovered CD points of WordPress dataset using ER PH method
24	53	82	111	142	174	203	232	261	290	325	357

Table 5

Discovered CD points of WordPress dataset using proposed method
27	57	86	115	144	173	202	231	260	289	318	347

We evaluated the results of our proposed method based on the output of baseline methods using CDD evaluation measures such as cdAccuracy, MTFA, MTD, and MTR. As previously mentioned, baseline methods are acquired through the monitoring of performance measures, both threshold-dependent and threshold-independent, which are calculated over time. These criteria are obtained by training the RF machine learning algorithm. In this section, we examined the effect of training data resampling on these criteria.

Table 6 The performance of the RF classifier before and after resampling

	Before	After
Accuracy	12	3
AUC	13	3
Precision	15	1
Recall	4	14
F Measure	4	10
Gmean	5	11

First, we obtained the values of each performance measure from the RF incremental machine learning model over time on 20 datasets. Then, we conducted the Wilcoxon test to determine whether that performance measure exhibited a significantly superior performance on the 20 datasets prior to resampling or afterwards. Table 6 presents the number of times the performance of the classifier was better before or after training data resampling. For instance, the first row shows that out of 20 datasets, the accuracy measure for 12 datasets before resampling significantly outperformed over time, while the accuracy of 3 datasets improved after resampling. These results indicated that resampling reduces the value of threshold-independent performance criteria (the first two rows of Table 6, which are red), while enhances the value of threshold-dependent criteria (the last three rows of Table 6, which are green) after resampling. We conducted this analysis using the Wilcoxon statistical test [61]. Consequently, balancing has a positive effect on threshold-dependent criteria within our problem. As a result, our data is categorized as unbalanced. Hence, it is necessary to investigate the performance of the method on the resampled data. Furthermore, we presented the values analysis of these criteria over time and on 20 datasets in a box-plot format (Fig. 10). The results indicated that all criteria, with the exception of precision, exhibit commendable stability. Previous studies have also examined the stability of the AUC criterion [1].

In the following, we evaluated the proposed CDD method using cdAccuracy, MTFA, MTD, and MTR criteria. For example, Table 7 presents the evaluation results of the MTR criterion related to the output of the proposed CDD method in different modes compared to the baseline method based on ER monitoring. In other words, the CD points were acquired through the proposed method in different modes and the baseline method based on ER monitoring for every dataset. Then we computed the MTR value from the Eq. (2) for each dataset. For example, the orange cell in Table 7 indicates that the MTR value would be 1500 when comparing the output of the raw_base method on the dataset "accumulate" with the output of the baseline method based on ER monitoring. We computed the MTR for all five methods on each of the 20 datasets. Finally, we analyzed the MTR values to determine which method exhibited significantly superior performance (higher), employing the Friedman test. The last row of Table 7 provides the results of the Friedman test. The lower rank number indicates better performance. As only the results of the Friedman test (the last line of Table 7) are important for evaluation, we have not included Table 7 for other baseline methods in the paper. In summary, the final results obtained from the Friedman test have been compiled in Table 8 through Table 11. Each of these tables presents the evaluation result of one of the performance measures, namely cdAccuracy, MTFA, MTD, and MTR to compare the proposed methods with different baseline methods. According to the data presented in Table 8, for all baseline methods whose names are in the first column, the values in the IME_base column are consistently lower than those in the BD_base and raw_base. Therefore, we can deduce that the method based on IME interpretation outperforms BD_base and raw_base (Table 8). Even on resampled data, IME_base (rIME_base column) performed better than BD_base (rBD_base column). The output of the other three criteria (cdAccuracy, MTFA and MTD) in Table 9 to Table 11 is the same as MTR.

Table 7 The evaluation results of the MTR criterion, which compares CDD methods with method based on ER monitoring using the Friedman test

methodology/ dataset	raw_base	IME_base	BD_base	rIME_base	rBD_base
accumulo	1500	1500	1500	6000	6000
angular	221	239	221	239	221
brackets	279	139	279	11700	11700
bugzilla	4800	4800	4800	3200	3200
camel	2193	1919	2193	10233	10233
cinder	370	370	370	7400	7400
django	6300	4200	1232	4200	1938
gephi	643	643	643	643	643
hibernate-orm	4050	2700	4050	579	225
hibernate-search	750	750	750	857	857
imglib2	575	575	575	177	177
jetty	620	620	620	12400	1771
kylin	6000	6000	6000	6000	500
log4j	1000	1000	1000	1000	1000
nova	2690	6725	6725	13450	13450
postgres	732	2090	1819	4390	2665
tomcat	2350	2089	2089	895	1567
wordpress	4688	9375	4312	6250	4688
meanAvg	4.22	4.11	4.5	3.17	3.94

Table 8

The evaluation results of the **MTR** criterion, which compares CDD methods with methods based on various criteria monitoring using Friedman test
	raw_base	IME_base	BD_base	rIME_base	rBD_base
ER PH-Er PH (the last line of Table 7)	4.22	4.11	4.5	3.17	3.94
AUC-Er PH	4.33	3.75	4.67	3.42	3.61
Gmean-Er PH	4.06	3.17	4.06	3.83	4.75
Precision-Er PH	4.08	3.56	4.81	3.61	4.08
Recall-Er PH	4.28	3.28	3.89	3.89	4.61
MCC-Er PH	4.31	3.31	4.25	3.14	4.03
F Measure-Er PH	4.22	3.03	4.31	3.81	4.33

Table 9

The evaluation results of the **cdAccuracy** criterion, which compares CDD methods with methods based on various criteria monitoring using Friedman test (no significant difference compared to MTR)
	raw_base	IME_base	BD_base	rIME_base	rBD_base
ER PH-Er PH	3.69	3.89	4.33	3.89	4.22
AUC-Er PH	3.75	3.94	4.42	3.86	3.94
Gmean-Er PH	3.75	3.75	4.44	3.86	4.17
Precision-Er PH	3.75	3.86	4.44	3.75	4.14
Recall-Er PH	3.69	3.69	4.36	3.89	4.31
MCC-Er PH	3.72	3.92	4.42	3.64	3.94
F Measure-Er PH	3.75	3.75	4.44	3.86	4.19

Table 10

The evaluation results of the **MTD** criterion, which compares CDD methods with methods based on various criteria monitor using Friedman test (no significant difference compared to MTR)
	raw_base	IME_base	BD_base	rIME_base	rBD_base
ER PH	4.03	4.42	4.31	3.28	3.72
AUC-Er PH	4.19	3.81	4.58	3.31	3.94
Gmean-Er PH	3.89	3.5	3.94	4.03	4.56
Precision-Er PH	3.79	3.71	4.44	3.76	4.5
Recall-Er PH	4.03	3.64	3.89	3.83	4.5
MCC-Er PH	4.17	3.69	4.17	3.19	3.92
F Measure-Er PH	3.91	3.29	4.18	3.79	4.59

Table 11

The evaluation results of the **MTFA** criterion, which compares CDD methods with methods based on various criteria monitor using Friedman test (BD_base performed better than IME_base in rebalanced ones)
	raw_base	IME_base	BD_base	rIME_base	rBD_base
ER PH	4.64	4.08	4.36	3.33	3.28
AUC-Er PH	4.53	3.94	4.33	3.75	3.36
Gmean-Er PH	4.36	3.58	4.17	4.17	3.97
Precision-Er PH	4.35	3.74	4.56	3.74	3.74
Recall-Er PH	4.33	3.56	4.14	3.94	4.14
MCC-Er PH	4.33	3.75	4.14	3.94	3.75
F Measure-Er PH	4.5	3.72	4.31	3.72	3.72

In order to evaluate that the proposed method based on which interpretation algorithm and which type of data (interpreted or raw and simplified or resampled) yields superior results, we conducted an experimental study on 20 datasets with different baseline methods and using performance measures, namely cdAccuracy, MTFA, MTD, and MTR. For a better comparison, we obtained radar diagrams for each measure to illustrate which method performed better. A radar diagram is utilized to compare the values of three or more variables with respect to a central point. These diagrams prove to be highly effective when comparing the performance of an algorithm with that of the baseline method. Figure 11 displays a radar diagram that pertains to Table 8 through Table 11. This type of diagram is used to compare the consistency of the output of the proposed interpretation-based methods with that of the baseline methods, which were influenced by four well-known performance criteria of CDD methods: MTR, cdAccuracy, MTD, and MTFA (four subfigure). Since the diagram is associated with the Friedman test output, the closer it is to the center, the better that method performs. As depicted in Fig. 11, it is evident that the orange graph (i.e., IME-based method) consistently falls within the gray graph (i.e., BD-based) in every four subfigures. Consequently, it is apparent that the IME-based method consistently outperforms BD-based even in resampled mode (as indicated by the yellow graph (i.e., rIME-based method) always being inside the bold blue graph (i.e., rBD-based)).

4.2.2 RQ2 How does the proposed method based on resampled data perform compared to the baseline methods that rely on threshold-dependent and threshold-independent criteria?

In this section, we conduct a comparison between the performance of the proposed method, which was implemented on the resampled data, and the output of the baseline methods. The comparison is based on monitoring of both thresholds-dependent and threshold-independent criteria. The Friedman test revealed that our methods did not exhibit significant differences in findings but yielded promising results. In Section 4, we present exciting results from the ranking of the method outputs on 20 datasets. As previously mentioned, detection of CD points using ER monitoring is a common method. However, in this work, we have located CD points using monitoring of measures other than ER. Then, to more extensively examine the performance of the proposed methods based on interpretation, we have evaluated their consistency with the baseline methods based on monitoring the performance of different criteria using four well-known performance criteria of CDD methods; MTR, cdAccuracy, MTD, and MTFA. The criteria used in the baseline methods are either threshold-dependent or threshold-independent. In the previous section, we noted that studies have shown that the performance of each of these two categories on the stability of incremental models differs. Therefore, relevant studies avoid them. On the other hand, checking model stability on resampled data is also a challenge in this field. To address these two challenges, we posed RQ2. To address this RQ, we investigated the consistency of CD points obtained from baseline methods with those obtained from proposed methods implemented on simplified and resampled data. To provide a clearer response, we have categorized the results of Table 8 to Table 11 into two groups: threshold-dependent and threshold-independent criterion-based methods using radar diagrams. These diagrams are displayed in Fig. 12 (related to the method based on threshold-independent criteria) and Fig. 13 (related to the method based on threshold-dependent criteria). Since the diagram is associated with the Friedman test output, the closer it is to the center, the better that method performs. In view of Fig. 12, when using threshold-independent criteria for baseline methods, the output of proposed methods implemented on resampling data is more consistent with the results of these baseline methods according to all four famous criteria of CDD methods: MTR, cdAccuracy, MTD, and MTFA. As shown in Fig. 12, the yellow chart (rIME-base) which is based on resampled data falls within the orange chart (IME-base) which is based on simplified data, as well as the bold blue chart (rBD-base) falls within the gray chart (BD-base).

According to Fig. 13, when using threshold-dependent criteria for the baseline methods, the output of the proposed methods implemented on simplified data is more consistent with the results of these baseline methods than on the resampled data. The orange graph (IME-base) which is based on simplified data always falls within the yellow graph (rIME-base) which is based on resampled data as shown in Fig. 13. However, the accuracy of the CDD method based on BD_base interpretation is lower than rBD_base, which affects the false alarm criterion, i.e., MTFA.

RQ2: Threshold-independent performance measures of the classifier decrease after resampling, while its threshold-dependent criteria improve. Resampling also reduces the matching of the CD points discovered by proposed CDD methods with the output of baseline methods based on monitoring threshold-dependent criteria. However, it improves compared to baseline methods based on monitoring threshold-independent measures. Accordingly, if scholars are to predict the performance instability of threshold-dependent measures, they ought to employ the method based on simplified samples. Conversely, if they are to predict the performance instability of threshold-independent measures, they should use a method based on resampled samples. Hypothetically, When to improve the performance of threshold-dependent measures, such as Recall [1], and increase the completeness of detecting software defects is important, methods based on simplified and unbalanced data outperform, and the output of the proposed method is closer to the baseline method.

In this paper, since an extensive study is conducted to evaluate the proposed method, some potential threats to validity should be considered. The following are the threats of the validity of this research, which include three categories: construct, internal, and external.

5.1 Construct Validity

Threat to construct validity refers to how accurately a scientific study measures or evaluates the theoretical construct or concept it claims to study. In scientific papers, construct validity is typically discussed in relation to the design and implementation of research study. We need to establish that their chosen methods and measures are appropriate. The purpose of this paper is to introduce a proposed method for identifying CD with respect to concept change by discovering significant changes in the interpretation of instances over time. Given that CDD must be conducted simultaneously with the generation of new data, it is imperative that it does not rely on data labeling. An advantageous aspect of our approach lies in its independence from data labels. Moreover, some of the mentioned studies demonstrated that the presence of class imbalance leads to a decline in various performance metrics of the machine learning model. Consequently, we conducted an assessment of the proposed CDD method on simplified and resampled data using seven baseline methods based on different model performance criteria across 20 datasets. Subsequently, we obtained reliable results through statistical tests. We have employed a widely-used resampling algorithm called SMOTE. One of the mentioned studies demonstrated that the rebalancing techniques consistently work with different parameters. They found that SMOTE heavily improved the performance of SDP with more consistent results than that of SDP without the balance process [11, 57]. Therefore, we applied the SMOTE technique for resampling and we left the review of other resampling methods to future works. We conducted our analysis using Random Forest models as recommended by previous works, but regression models and other classification techniques such as Naive Bayes can also be examined.

5.2 Internal Validity

In terms of internal validity, we demonstrate the extent to which our hypotheses and methods can account for the obtained results. In this paper, we employed two valid interpretation algorithms that have been suggested in the literature to examine the impact of the proposed CDD method on simplified and resampled data. Furthermore, we utilized a large dataset that has been employed in various valid studies to investigate and validate our findings. In order to conduct a dependable evaluation, we employed the well-known change detection algorithm from existing literature as the baseline method and fine-tuned its parameters according to the relevant problem through experimental studies. This baseline method is based on error rate monitoring. Moreover, we proposed other baseline methods based on the error rate of threshold-dependent and threshold-independent measures in order to evaluate more reliably.

5.3 External Validity

External validity refers to the ability to generalize proposed methods to other studies and regions. Given that our research focused on some kind of online fault diagnosis system (FDS), it can be extended to other types of FDS systems, such as chemical fault diagnosis [62] and diagnosing defects in induction motors [63], among others. In regards to the evaluation methods and model algorithms, we have used more default settings. Therefore, we have refrained from modifying the parameters in order to enhance the outcomes. Consequently, our work is not constrained by specific parameters. However, since we conducted our experiments on 20 datasets of JIT-SDP and evaluated them with statistical tests on different efficiency criteria, we can conclude that it is efficient for datasets of this size in the JIT_SDP problem. Nevertheless, it can be separately examined on larger datasets in this area.

On the other hand, the temporal aspect refers to whether findings hold true over time. Longitudinal studies that follow participants over an extended period tend to have higher external validity compared to cross-sectional studies conducted at a single point in time. Given that the outputs of our method consistently align with those of various baseline methods over time, it possesses significant external validity.

Previous research has demonstrated that the performance of SDP models or data distribution may undergo alterations as time progresses. In other words, the data undergoes concept drift over time. Therefore, in this study, we introduced a proposed method to discover CD within the unlabeled JIT-SDP chronological dataset using detecting inconsistencies in instance interpretation over time. In this regard, we conducted a study on two types of interpretation algorithms that have been introduced in the literature and are more suitable. Additionally, in order to ensure a reliable evaluation, we employed seven baseline methods which were applied to 20 datasets. As well as, using suitable statistical methods, we demonstrated the significance of our results. On the other hand, as previously mentioned, the data of the JIT-SDP problem is unbalanced. The reliability of the SDP model's performance on unbalanced data, particularly in terms of predicting the target class (defect label), is questionable. Therefore, we conducted a study in this paper to implement the proposed method using a balanced training dataset. Finally, the output of the proposed method based on simplified and resampled data was compared to the baseline methods based on threshold-dependent and threshold-independent performance criteria. An intriguing finding emerged from this comparison was that the output of the resampled data-based method demonstrated greater consistency with the output of baseline methods based on threshold-independent criteria. Conversely, the simplified data-based method exhibited greater consistency with the output of baseline methods based on threshold-dependent criteria. When to improve the performance of threshold-dependent measures, such as Recall, and increase the completeness of detecting software defects is important [1], methods based on simplified and unbalanced data outperform, and the output of the proposed method is closer to the baseline method. Since we derived results using several threshold-dependent measures, it can be concluded that it is not random. However, if performance measures independent of threshold such as ER and AUC, which are considered standard measures for SDP models in many studies, are important, rebalancing the data leads to better performance for CDD. Furthermore, we have used methods based on instance interpretation that perform better than model interpretation and do not require labeling of input data to discover CD points.

This research has been conducted on within-project JIT-SDP datasets. In future works, the proposed methods can be investigated using integrated datasets from different projects. In related works, the model is trained using the latest input data before the testing data after concept drift occurs. To improve results, we can extract relevant data suitable for training the online classifier after detecting CD from other labeled datasets. In this study, we used a simplified classifier in the CDD method; however, in future works, more complex classifications such as deep learning or multi-objective optimization algorithms can be used to improve results. Additionally, the effectiveness of the proposed methods in this research can also be investigated on streaming data from non-software error detection systems.

Conflict of interest

The authors declare that they have no conflict of interest.

Author Contribution

Conceptualization, Methodology, Formal analysis and investigation and Writing - original draft preparation: Z. Chitsazian; Writing - review and editing: A. Nikanjam, Supervision: S. Sedighian Kashi and A. Nikanjam

Tantithamthavorn C, Hassan AE, Matsumoto K (2020) The impact of class rebalancing techniques on the performance and interpretation of defect prediction models. IEEE Trans Software Eng 46(11):1200–1219
Vreš D, Robnik-Šikonja M (2022) Preventing deception with explanation methods using focused sampling. Data Mining and Knowledge Discovery, : p. 1–46
McIntosh S, Kamei Y (2018) Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction. in Proceedings of the 40th International Conference on Software Engineering
Lin D, Tantithamthavorn C, Hassan AE (2021) The impact of data merging on the interpretation of cross-project just-in-time defect models. IEEE Trans Software Eng 48(8):2969–2986
Rajbahadur GK et al (2021) The impact of feature importance methods on the interpretation of defect classifiers. IEEE Trans Software Eng 48(7):2245–2261
Zheng W et al (2022) Interpretability application of the Just-in-Time software defect prediction model. J Syst Softw 188:111245
Fathi K et al (2023) Source Component Shift Detection & Classification for Improved Remaining Useful Life Estimation in Alarm-Based Predictive Maintenance. in. 23rd International Conference on Control, Automation and Systems (ICCAS). 2023. IEEE
Turhan B (2012) On the dataset shift problem in software engineering prediction models. Empir Softw Eng 17:62–74
Lu J et al (2018) Learning under concept drift: A review. IEEE Trans Knowl Data Eng 31(12):2346–2363
Dong F et al (2017) Concept drift region identification via competence-based discrepancy distribution estimation. in. 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE). 2017. IEEE
Chen L et al (2018) Tackling class overlap and imbalance problems in software defect prediction. Software Qual J 26:97–125
Gama J (2010) Knowledge discovery from data streams. CRC
Mahdi OA et al (2020) Fast reaction to sudden concept drift in the absence of class labels. Appl Sci 10(2):606
Chatzimparmpas A et al (2020) A survey of surveys on the use of visualization for interpreting machine learning models. Inform Visualization 19(3):207–233
Liang Y et al (2021) Explaining the black-box model: A survey of local interpretation methods for deep neural networks. Neurocomputing 419:168–182
Massey OT (2011) A proposed model for the analysis and interpretation of focus groups in evaluation research. Eval Program Plan 34(1):21–28
Jiarpakdee J et al (2020) An empirical study of model-agnostic techniques for defect prediction models. IEEE Trans Software Eng
Ribeiro MT, Singh S, Guestrin C (2016) Why should i trust you? Explaining the predictions of any classifier. in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining
Gosiewska A, Biecek P (2019) IBreakDown: Uncertainty of model explanations for non-additive predictive models. arXiv preprint arXiv:1903.11420
Demšar J, Bosnić Z (2018) Detecting concept drift in data streams using model explanation. Expert Syst Appl 92:546–559
Yang Z et al (2019) A novel concept drift detection method for incremental learning in nonstationary environments. IEEE Trans neural networks Learn Syst 31(1):309–320
Zenisek J, Holzinger F, Affenzeller M (2019) Machine learning based concept drift detection for predictive maintenance. Comput Ind Eng 137:106031
Abbasi A et al (2021) ElStream: An ensemble learning approach for concept drift detection in dynamic social big data stream learning. IEEE Access 9:66408–66419
Gangwar AK, Kumar S (2023) Concept Drift in Software Defect Prediction: A Method for Detecting and Handling the Drift. ACM Trans Internet Technol 23(2):1–28
Ekanayake J et al (2012) Time variance and defect prediction in software projects. Empir Softw Eng 17(4):348–389
Krishna R, Menzies T (2018) Bellwethers: A baseline method for transfer learning. IEEE Trans Software Eng 45(11):1081–1105
Kabir MA et al (2019) Assessing the significant impact of concept drift in software defect prediction. in 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC). IEEE
Baena-Garcıa M et al (2006) Early drift detection method. in Fourth international workshop on knowledge discovery from data streams. Citeseer
Gama J et al (2004) Learning with drift detection. in Brazilian symposium on artificial intelligence. Springer
Nishida K, Yamauchi K (2007) Detecting concept drift using statistical testing. in International conference on discovery science. Springer
Kabir MA et al (2020) A drift propensity detection technique to improve the performance for cross-version software defect prediction. in 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE
Bangash AA et al (2020) On the time-based conclusion stability of cross-project defect prediction models. Empir Softw Eng 25(6):5047–5083
Seiffert C et al (2009) RUSBoost: A hybrid approach to alleviating class imbalance. IEEE Trans Syst Man Cybernetics-Part A: Syst Hum 40(1):185–197
Cicak S, Avci U (2023) Handling Imbalanced Data in Predictive Maintenance: A Resampling-Based Approach. in 2023 5th International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA). IEEE
Li J et al (2024) Concept Drift Adaptation by Exploiting Drift Type. ACM Transactions on Knowledge Discovery from Data
Ekanayake J et al (2009) Tracking concept drift of software projects using defect prediction quality. in 2009 6th IEEE International Working Conference on Mining Software Repositories. IEEE
Kabir MA et al (2021) Inter-release defect prediction with feature selection using temporal chunk-based learning: An empirical study. Appl Soft Comput 113:107870
Kamei Y et al (2016) Studying just-in-time defect prediction using cross-project models. Empir Softw Eng 21:2072–2106
Chen X et al (2018) MULTI: Multi-objective effort-aware just-in-time software defect prediction. ‎Inf Softw Technol 93:1–13
Li W et al (2020) Effort-aware semi-supervised just-in-time defect prediction. ‎Inf Softw Technol 126:106364
Sheluhin OI, Sekretarev SA (2021) Concept drift detection in streaming classification of mobile application traffic. Autom Control Comput Sci 55:253–262
Tabassum S, Minku LL, Feng D (2022) Cross-Project Online Just-In-Time Software Defect Prediction. IEEE Transactions on Software Engineering
Cabral GG et al (2019) Class imbalance evolution and verification latency in just-in-time software defect prediction. in. IEEE/ACM 41st International Conference on Software Engineering (ICSE). 2019. IEEE
Haase RF, Ellis MV (1987) Multivariate analysis of variance. J Couns Psychol 34(4):404
Štrumbelj E, Kononenko I (2014) Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst 41:647–665
Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Adv Neural Inf Process Syst, 30
Gao Y, Zhu Y, Zhao Y (2022) Dealing with imbalanced data for interpretable defect prediction. ‎Inf Softw Technol 151:107016
Ross GJ et al (2012) Exponentially weighted moving average charts for detecting concept drift. Pattern Recognit Lett 33(2):191–198
Torgo L, Torgo M (2013) DMwR: Functions and data for data mining with R. R package version 0(4):1
Mockus A, Weiss DM (2000) Predicting risk of software changes. Bell Labs Tech J 5(2):169–180
Hassan AE (2009) Predicting faults using the complexity of code changes. in 2009 IEEE 31st international conference on software engineering. IEEE
Purushothaman R, Perry DE (2005) Toward understanding the rhetoric of small source code changes. IEEE Trans Software Eng 31(6):511–526
Guo PJ et al (2010) Characterizing and predicting which bugs get fixed: an empirical study of microsoft windows. in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1
Jiarpakdee J, Tantithamthavorn C, Hassan AE (2019) The impact of correlated metrics on the interpretation of defect models. IEEE Trans Software Eng 47(2):320–331
Jiarpakdee J, Tantithamthavorn C, Treude C (2020) The impact of automated feature selection techniques on the interpretation of defect models. Empir Softw Eng 25:3590–3638
Chawla NV et al (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357
Pelayo L, Dick S (2007) Applying novel resampling strategies to software defect prediction. in NAFIPS 2007–2007 Annual meeting of the North American fuzzy information processing society. IEEE
Suárez-Cetrulo AL, Quintana D, Cervantes A (2023) A survey on machine learning for recurring concept drifting data streams. Expert Syst Appl 213:118934
Bifet A et al (2013) CD-MOA: Change detection framework for massive online analysis. in Advances in Intelligent Data Analysis XII: 12th International Symposium, IDA 2013, London, UK, October 17–19, Proceedings 12. 2013. Springer
Conover WJ, Iman RL (1981) Rank transformations as a bridge between parametric and nonparametric statistics. American statistician, : p. 124–129
Blair RC, Higgins JJ (1980) The power of t and Wilcoxon statistics: A comparison. Eval Rev 4(5):645–656
Wu H, Zhao J (2018) Deep convolutional neural network model based chemical process fault diagnosis. Comput Chem Eng 115:185–197
Razavi-Far R, Farajzadeh-Zanjani M, Saif M (2017) An integrated class-imbalanced learning scheme for diagnosing bearing defects in induction motors. IEEE Trans Industr Inf 13(6):2758–2769

R package version 1.3.0.” Software available at URL: https://cran.r-project.org/package= ExplainPrediction.
(method="IME" from ExplainPrediction package)
R package version 0.1.6.” Software available at URL: https://cran.r-project.org/package=breakDown.

No competing interests reported.

Interpretation Conclusion Stability of Software Defect Prediction over Time

Status:

Version 1

Abstract

Figures

1 Introduction

2 Related Works

2.1 Interpretation Techniques

2.2 Concept Drift Detection

2.3 The Effect of Rebalancing Techniques on Performance and Interpretation

2.4 Evaluation Methods

3 Approach

3.1. Constructing Baseline Methods

3.2. Presentation of Proposed Methods

3.3. The Implementation of the Proposed Method on Resampled Data

4 Evaluation

4.1 Experimental Setup

4.1.1 Experimental Parameters and Comparable Tool

4.1.2 Dataset

4.1.3 Data Preprocessing

4.2 Experimental Results

5 Threats to validity

5.1 Construct Validity

5.2 Internal Validity

5.3 External Validity

6 Conclusion and Future Works

Declarations

Author Contribution

References

Footnotes

Additional Declarations

Status:

Version 1