Explainable analytics: Understanding causes, correcting errors, and increasingly achieving perfect accuracy from nature of distinguishable patterns

doi:10.21203/rs.3.rs-1130288/v1

Download PDF

Article

Explainable analytics: Understanding causes, correcting errors, and increasingly achieving perfect accuracy from nature of distinguishable patterns

https://doi.org/10.21203/rs.3.rs-1130288/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

In addition to pursue accurate analytics, it is invaluable to clarify how and why inaccuracy exists. We propose a transparent classification method (TC). In training, we discover patterns from positive and negative observations respectively; next, patterns are excluded if they appear in both types. In testing, observations are scored by the pure patterns and connected like social networks. Based on set theory, pure patterns have explanatory power for distinguishing tangled relationship between negative and positive observations. Experimental results demonstrate that TC can identify all positive (e.g., malignant) observations at low ratios of training to testing, e.g., 1:9 in Breast Cancer Wisconsin (Original) and 3:7 in Contraceptive Method Choice dataset. Without fine-tuned parameters and random selection, TC eliminates uncertainty of the methodology. TC can visualize causes, and therefore, prediction errors are traceable and can be corrected. Further, TC shows potential of identifying whether the ground truth is incorrect (e.g., diagnostic errors).

Oncology

Cancer Biology

Accurate prediction plays a pivotal role in analytics; however, in reality people usually face the challenge of explaining how and why the prediction is inaccurate^1-2. By the survey³, outpatient diagnostic errors are at a rate of 5.08% (around 12 million US adults) per year. Even reducing only 1% errors, the lives of million people are saved. We consider three major issues of errors. First, fault in data, human mistakes or defective machines can produce faulty data. Without domain knowledge, such a fault is difficult to correct. Nevertheless, we should remove inconsistency, i.e., observations of positive class are identical to those of negative one. In addition, positive and negative observations may have similar patterns that are inextricably interwoven, e.g., people with similar profiles may exhibit different behavior. Lim et al.⁴ show the contraceptive method choice (CMC) dataset⁵ is the most difficult to classify, and especially the minimum error rates are greater than 0.4. Second, mismatches between data and methods, data contains categorical (e.g., country), numerical (e.g., age), or both types, which have natural constraints on analysis. For categorical values, only the number of items and the mode are statistically relevant⁶ so that a numerical-orientated method is inherently inadequate. For numerical values, they can be transformed into categorical ones by discretization⁷, which have been widely applied to knowledge discovery and data mining (KDD) applications⁸. However, bias occurs if categories are unrepresentative of numerical values. Third, big data challenge, the complexity of data is determined by the number of rows and features (columns). Particularly, computation tasks increase rapidly with the number of features, as known curse of dimensionality (CoD)⁹. To cope with CoD, dimension reduction and feature selection methods reduce the complexity by extracting information that is practical for classification and cluster analysis. The extraction, a trade-off between efficiency and effectiveness, may involve pruning large amounts of data. Actually, such a pruning manner may have pitfalls¹⁰ or miss clues to errors.

We conducted experiments with two datasets: Breast Cancer Wisconsin (Original) (BCWO) and Contraceptive Method Choice (CMC)⁵. Figure 2 (A)(B) and (C)(D) show the results of BCWO and CMC respectively. In Figure 2 (A), TC achieves perfect recall (i.e., recall=1.0) at the lowest ratio (i.e., 1:9) and 7 other ratios. It means TC is not only accurate in small amounts of data but also stable when data increases. TC has one error at the ratios 2:8 and 3:7 because the positive observation PO₂₂₃ is predicted as a negative one. Since the ratio 4:6, PO₂₂₃ belongs to training data and is not used in testing. For further exploration, at the ratio 10:10^#, PO₂₂₃ is irrelevant to PPP except itself. Indeed, PO₂₂₃ is related to PP that are also relevant to NO. PPP can eliminate impingement of PO₂₂₃ on other observations. Novel observations are regarded as positive ones if they are unrelated to both PPP and PNP. Some of NO are predicted as positive so that TC gets lower performance in terms of PR. With increased training observations, TC may obtain sufficient PP or NP to provide accurate results and their causes, e.g., at the ratio 1:9, PO₆₂₇has a novelty score of 70 by rule 3; at the ratio 2:8, PO₆₂₇contains 3 patterns in PPP, namely, ['8|0.8'], ['6|1.5', '8|0.8'], and ['1|0.9', '8|0.8'] and has a positive score of 16 by rule 1. Figure 2 (B) shows the suitable granularity of discretization gives broad vision for discovering more clues. At the ratio 1:9 TC has one error, which is also caused by PO₂₂₃. At the ratio 2:8, PO₂₂₃ has 1 pattern ['3|0.0', '5|0.0'] in PPP and is related to PO₃₃ and PO₁₀₂. At the ratio 3:7, PO₂₂₃ has 1 pattern ['3|0.0', '5|0.0'] in PPP and related to PO₃₃, PO₁₀₂, and PO₁₄₃. Since the ratio 4:6, PO₂₂₃ belongs to training data and is not used in testing. In Figure 2 (C) (D) at the ratios 1:9 and 2:8, RE and PR are zero due to none of negative observations in training (TRN). At the ratio 3:7, insufficient TRN leads to extremes as perfect RE but relatively low PR. The error rate is less than 0.4 at the ratio 7:3. In CMC, profile data is incapable of explaining behavior because (D) indicates 33% (i.e., (1473−980)/1473) cases are inconsistent inherently. In Figure 3, we provide visualization of TC. Compared to images, analysis of categorical and numerical data was difficult to visualize how causes are related to results.

Figure 3 (A) shows TC successfully predicts O₁₀₄ is positive because O₁₀₄ is related to the six pure positive patterns that are obtained from respective positive observations of training. Moreover, the thickness of lines represents the score as well as the positive degree. Figure 3 (B) presents the association between O₁₀₄ and others. In the O₁₀₄-headed group, the most common positive pattern is '0|1.3', and O₂₄and O₅₇ have significant influence on O₁₀₄. Figure 4 (A) (B) (C) illustrate TC is tolerant of faulty class labels (ground truth) in terms of testing, training, and both cases. Figure 4 (D) demonstrates TC can correct errors in class labels. Specifically, observations of faulty class labels have extreme values of PS or NS, which becomes significant with increasing data. Based on profound knowledge of most cases, TC would be useful for checking whether the original judgement (e.g., diagnosis) deserves further inspection.

We propose a method of transparent classification, named TC, which not only pursues accuracy but also clarifies cause of inaccuracy. Further, the design principles of TC ensure reproducibility¹¹. Figure 1 shows processes of TC. In data preprocessing, TC handles missing values and mixed values. Without involving randomness and reduction, TC delivers intrinsic nature of data. In identifying distinguishable patterns, TC finds patterns from training observations, which are used for predicting which class a test observation belongs to, e.g., a malignant or benign tumor. Through increasing ratios of training to testing, TC represents the forest and the trees, and input data is given in sequence. To avoid CoD, TC finds patterns by intersecting pairwise observations in each of the classes, which possess essential features of data in miniature. In the worst case of TC, n observations produce only patterns. By contrast, KDD faces the challenge of CoD, i.e., given the lowest threshold, k items yield 2^k itemsets¹², and large amounts of itemsets are pruned if the threshold is high. In positive patterns (PP), TC obtains PP from positive training observations (PO). For pure PP (PPP), TC excludes any positive pattern that also appears in negative training observations (NO). By set theory, the exclusion implies none of PPP is included in any of NO and hence TC can distinguish between PO and NO. Analogously, Negative patterns (NP) and pure NP (PNP) are the counterpart of PP and PPP. Without involving either fine-tuned parameters or random selection, TC eliminates uncertainty of the methodology. In establishing the causes, TC accumulates positive, negative, and novel degrees of a test observation O_t by rule 1, 2, and 3 which associate patterns with the observation and provide obvious clues for judgement. In rule 1, O_t containing patterns in PPP gets a positive score (PS). In rule 2, O_t containing patterns in PNP gets a negative score (NS). In rule 3, O_t containing none of patterns in PPP and PNP is considered novel and gets a novelty score (NT) equal to the number of training observations. In understanding results of analytics, we evaluate performance of TC by three measures, Precision, Recall, and AUC (Area under Curve)^8,12. According to the standard of diagnostic medicine¹³: AUC=0.5, no discrimination; 0.7≤AUC<0.8, acceptable; 0.8≤AUC<0.9, excellent; and 0.9≤AUC≤1, outstanding. In cause for prediction errors, error 1 (false positive¹⁴) occurs if O_t is predicted as positive but actually negative, denoted by NO_t*. By cause 1.1, NO_t* contains pure positive patterns although it should not. By cause 1.2, NO_t* is novel, namely, containing no pattern in PPP and PNP. Error 2 (false negative¹⁴) occurs if O_t is predicted as negative but actually positive, denoted by PO_t*. By cause 2.1, PO_t* contains pure negative patterns but no pure positive pattern although it should. Prediction errors occur due to insufficient training data or labelling errors in training data. Increasing training data helps to reduce prediction errors. If the portion of labelling errors is small, TC has the potential of identifying labelling errors. Specifically, false negatives usually have a small NS.

Acknowledgments: We thank the editor and reviewers for offering comments on the manuscript.

Funding: This work was supported by the Ministry of Science and Technology (MOST) in Taiwan.

MOST (grant no. 110-2410-H-224-037-): H.T.P.

MOST (grant no. 110-2410-H-224-009): C.C.H.

Author contributions:

Conceptualization: H.T.P. conceived the idea and C.C.H. provided for guidance.

Methodology: H.T.P.

Experiments: H.T.P.

Visualization: H.T.P.

Funding acquisition: H.T.P., C.C.H.

Writing – original draft: H.T.P.

Writing – review & editing: H.T.P., C.C.H.

Competing interests: Authors declare that they have no competing interests.

Description, prediction, explanation. Nat Hum Behav 5, 1261 (2021).
D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, G.-Z. Yang, XAI—Explainable artificial intelligence. Sci. Robot. 4, eaay7120 (2019).
H. Singh, A. N. Meyer, E. J. Thomas, The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual. Saf. 23, 727-731 (2014).
T. S. Lim, W. Y. Loh, Y. S. Shih, A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach. Learn. 40, 203-228 (2000).
D. Dheeru, C. Graff, UCI Machine Learning Repository (2019); http://archive.ics.uci.edu/ml.
S. S. Stevens, On the theory of scales of measurement. Science. 103, 677-680 (1946).
S. Garcia, J. Luengo, J. A. Sáez, V. Lopez, F. Herrera, F. A survey of discretization techniques: Taxonomy and empirical analysis in supervised learning. IEEE Trans. Knowl. Data Eng. 25, 734-750 (2012).
P. N. Tan, M. Steinbach, V. Kumar, Introduction to Data Mining (Pearson, 2020).
N. Altman, M. Krzywinski, The curse(s) of dimensionality. Nat Methods. 15, 399-400 (2018).
A. Jain, D. Zongker, Feature selection: Evaluation, application, and small sample performance. IEEE Trans. Pattern Anal. Mach. Intell. 19, 153-158 (1997).
M. McNutt, Reproducibility. Science. 343, 229 (2014).
M. J. Zaki, W. Meira Jr, Data Mining and Machine Learning: Fundamental Concepts and Algorithms (Cambridge University Press, 2020).
J. N. Mandrekar, Receiver operating characteristic curve in diagnostic test assessment. J. Thorac. Oncol. 5, 1315-1316 (2010).
National Cancer Institute (2021); https://www.cancer.gov/publications/dictionaries/cancer-terms/def/false-positive-test-result.

There is NO Competing Interest.

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Explainable analytics: Understanding causes, correcting errors, and increasingly achieving perfect accuracy from nature of distinguishable patterns

Archived Versions:

Version 1

Abstract

Figures

Main Text

Results

Discussion

Methods

Declarations

References

Additional Declarations

Archived Versions:

Version 1