Handling missing data is indispensable in health-care real-world data processing. Imputing method may introduce error and multicollinearity. Therefore, we explored (Optimal Intact Subset Method, OIS.Method) to avoid the issues. By exploring an optimal deleting way of columns and rows with missing data, a subset retaining most information of original datasets was determined. Traditionally, we can traverse all deleting ways. But the computational cost is too high to use in large datasets. OIS.Method used an indicator to determine the optimal deleting order which can ascertain the optimal deleting way and simplify computing. In order to validate the effectiveness of OIS.Method, we compared OIS.Method with five other missing data handling methods in simulated real-world classification datasets. Additionally, we validated OIS.Method in two real-world classification tasks. In simulated datasets, the performance of OIS.Method was best(highest AUC was 1). In real-world datasets, OIS.Method could acquire better classification performance. Take AUC for an example: OIS.Method VS Simple Impute VS Random Forest VS Modified Random Forest, 0.8179±0.0005 VS 0.8116±0.0002 VS 0.8087±0.0009 VS 0.8093±0.0014 in task1, and 0.7028±0.0126 VS 0.6963±0.0231 VS 0.6957±0.0247 VS 0.6699±0.0249 in task2. The calculation of OIS.Method is smaller, and it is well-suited for large real-world datasets.
Article
A novel method for handling missing data in health care real-world study: Optimal Intact Subset Method
https://doi.org/10.21203/rs.3.rs-1761573/v2
This work is licensed under a CC BY 4.0 License
Version 2
posted
You are reading this latest preprint version
Missing data
real-world study
data preprocessing
missing data imputation
Missing data are inevitable in clinical study1. They occur in nearly all steps due to multiple reasons, such as participant death or dropouts2; investigators or mechanical error3. Particularly, considering the special characteristics of real-world data, redundant variables4, unbalanced distribution of missingness, large amount of data, the negative effect caused by missing data will be more complicated.
Missing data will pose challenges on the statistical analyze and potentially weaken the validity of conclusion due to the loss of information5. Take classifications as an example, missing data in training sets decreases the prediction accuracy of classifiers6. Therefore, handling missing data is necessary.
Currently, the commonly-used missing data handling methods can be grouped into two categories7. One is deleting method which will simply discard records with missing data8. The other one is to impute with a particular value. On one hand, fixed-value, the mean, median or mode of the observed values, can be used to impute missing values9. On the other hand, missing value can be imputed by predictive models. Robust Machine Learning techniques such as Support Vector Machine(SVM), Linear Regression have been used for predicting missing data10. Multiple imputation(MI), based on predicted imputing, will create several different plausible imputed data sets and appropriately combining results obtained from each of them11. MI can adequately account for the uncertainty of imputed values to reduce the bias in downstream analysis through the Rubin’s rule12.
Above methods, however, have following problems. Firstly, deleting method is not applicable to the dataset who doesn’t contain sufficent sample size after deleting13. Secondly, imputing fixed values will bring potential bias or unrealistic results on high-dimensional dataset14. Thirdly, multicollinearity issues would be the main source of errors when using observerd independent variables to predict unknown variables15. Lastly, MI is not suitable when the missingness is not-at-random16, and computational resources are wasted when lots of redundant variables were imputed. Thus, it is necessary to explore a new missing data handling method especially suitable for real world dataset.
We established and assessed a method which can find an optimal intact subset from original dataset(Optimal Intact Subset Method, OIS.Method), the detailed procedure and validation process were described in this article.
Algorithm
Intact subset can be obtained by deleting rows and columns containing missing data. Adopting different deleting ways will generate different intact subsets. The subset containing most figures will be chosen as the optimal one.
1. find the optimal intact subset by traversing method
The optimal intact subset can be determined by traversing all deleting methods. Take dataset A in Fig. 1 with 8 rows and 4 columns(X1, X2, X3, X4) for an example. In this example, symbols of ‘V’ represented non-missing values; blank cells represented missing values. In order to obtain intact subset, columns with missing values were deleted first, and then the rows containing missing values were deleted, too. Deleting different column(s) will result in different amount of data in intact subset. When deleting X4 and corresponding rows with missing values, four rows( n2, n5, n6, n8), three columns(X1, X2, X3) and 12 figures were remained. Similarly, deleting X1 and X2, two rows(n2, n8), two columns(X3, X4) and four figures were remained. Therefore, after traversing \({N=C}_{4}^{1}+{C}_{4}^{2}+{C}_{4}^{3}+{C}_{4}^{4}={2}^{4}=16\) kinds of deleting ways, we found that deleting X4 was the best way because it reserved most figures. See Table 1for further details.
|
The number of deleted columns |
The combination of deleted columns |
The number of reserved rows |
The number of reserved columns |
The number of reserved figures |
|---|---|---|---|---|
|
0 |
none |
2 |
4 |
8 |
|
1 |
X1 |
2 |
3 |
6 |
|
X2 |
2 |
3 |
6 |
|
|
X3 |
2 |
3 |
6 |
|
|
X4 |
4 |
3 |
12 |
|
|
2 |
X1、X2 |
2 |
2 |
4 |
|
X1、X3 |
2 |
2 |
4 |
|
|
X1、X4 |
4 |
2 |
8 |
|
|
X2、X3 |
2 |
2 |
4 |
|
|
X2、X4 |
4 |
2 |
8 |
|
|
X3、X4 |
5 |
2 |
10 |
|
|
3 |
X1、X2、X3 |
2 |
1 |
2 |
|
X1、X2、X4 |
5 |
1 |
5 |
|
|
X1、X3、X4 |
5 |
1 |
5 |
|
|
X2、X3、X4 |
6 |
1 |
6 |
|
|
4 |
X1、X2、X3、X4 |
0 |
0 |
0 |
| There were 16 kinds of deleting ways, among which deleting X4 is the best way. | ||||
2.The disadvantage of traversing method
High computational cost is the fatal shortcoming of traversing method. Generally, for a dataset with f columns, there will be N = 2f kinds of deleting ways. The times of traversing will increase exponentially with the increase of f value. When f = 12, \(N={2}^{12}=4096\), and when f = 50, \(N={2}^{50}=1.1\times {10}^{15}\). Obviously, it has far exceeded the computing power of ordinary computing devices. However, it is very common to include more than 50 variables in real-world study. Therefore, it’s necessary to explore a substitute algorithm to simplify computing.
3. The OIS.method is an alternative method to cuts out the defects
To avoid traversing all deleting ways, we defined an indicator containing the missing information of both rows and columns and used it to determine the deleting order of columns. Set the indicator as Sm.
Sm\(=\sum _{1}^{n}\frac{{a}_{i,j}}{f}\times 100+{b}_{j}\times 100\) (1)
n: the number of rows, f: the number of columns, \(a\): the number of non-missing values, \({a}_{i,j}\):the number of non-missing values in the i-th row, j -th column, bj: the number of non-missing values in the j -th column.
Taking dataset A in Fig. 1 as an example, the detailed steps of OIS.method were descripted as follows, and the flowchart of OIS.method was shown in Fig. 1.
Sum the number of non-missing values of each row and record as an
Step2: Replace
Replace all non-missing values with ‘100’, and replace missing data with ‘\(\frac{{a}_{i,j}}{f}\)’.
Step 3 Calculate the value of Sm
Calculate the value of Sm of each column according to Formula (1). Then, sort columns according to the Sm value from smallest to largest.
Step 4: Delete columns
For each combination of deleted columns, deleting column(s) according to the order in the step 3.
The value of “Cell” represents the amount of data of intact subset. \(Cell={n}_{subset}\times {f}_{subset}\). It has maximum value in a certain combination of deleted columns (Fig. 2).
Step 5: Choose the optimal deleting method
The value of Cell has the maximum when deleting X4.
OIS.method can avoid traversing all deleting ways by identifying the optimal deleting order via Sm value, and it has been validated in another two datasets, too. The validation process was shown in Supplementary1 Table 1–2.
Table 2
confusion matrix
Model validation
Before validating in simulated real-world datasets, OIS.Method has been validated in 700 simulated classification datasets with different missingness patterns. The detailed procedure was descripted in Supplementary2.
1.Validation in computer-generated simulated real-world datasets
Step1: generate simulated real-world datasets with missing values
In order to simulated real-world datasets, redundant variables were introduced in the dataset with 500 samples, 10 features, 50% missing rate and an exponential missingness distribution. The multiples of redundant variables were 0,1,1.5,3,8. In order to explore the importance of columns on the results, power exponents were added to\({f}_{subset}\) when using OIS.Method, so \(Cell={n}_{subset}\times {f}_{subset}^{e}\).
Step2: the validation and comparison of OIS.Method
Besides OIS.method, 5 control missing data handling methods, IterativeImputer, IterativeSVD, K-Nearest Neighbor(KNN), Random Forest, and Simple Fill, were used to handle missing data. The control methods were realized via Fancyinput package of python.
Step3: Build and evaluate machine learning models
Five classification algorithms were used on each dataset, respectively, to develop predictive models. Machine learning algorithms in this study were Logistic Regression, Multiayer Perceptrons(MLP), Support Vector Machine(SVM), Random Forest, and Extreme Gradient Boosting (XGBoost). Datasets were divided into training sets and testing sets. The training sets were used to train models, and the Area Under ROC Curve(AUC), Accuracy, Precision, Recall, F1-score of testing sets were used to evaluate the models. Confusion matrix in Table 2 can help to understand the indicators17. The modeling and evaluating processes were repeated 10 times.
Step4: Assess the performance of different missing data handling methods
A better performance model stands for a better missing data handling method. So, we use the assessment value of predictive performance as the indicators to reflect the abilities of missing data handling methods indirectly.
In order to evaluate the methods more intuitively, we sorted them according to the value of the five evaluation indicators when introducing different multiples of redundant variables respectively. The proportion of different methods in Top 10, Top20 and Top50 according to different indicators were calculated. The higher the ranking and the larger the proportion, the better the method.
2. Validation in real-world datasets
In this part, OIS.method and three other methods(simple impute, random forest, modified random forest) were assessed in two classification tasks in real-world data with missing values.
2.1 Task 1: predict the risk of hypotension during dialysis
A dialysis dataset with 314, 534 samples, 496 variables and a missing rate of 42.54% were collected in the study. The main target of this study was to predict the risk of hypotension during dialysis applying machine learning method.
After handling missing data, the dataset was divided into training set and testing set. The training set was applied for building machine learning models using Random Forest, Gradient Boosting, and Logistic Regression. The AUC(Area Under ROC Curve), Accuracy, Precision, Recall and F1-score of testing set were used to evaluate the models and reflect the performance of missing data handling method indirectly.
2.2 Task 2: predict the risk of drug adverse reaction in elderly patients with type 2 diabetes
A dataset of elderly patients with type 2 diabetes with 1,224 samples and 279 variables and a missing rate of 21.75% were collected in this study.
Different from task1, two machine learning algorithms, Ensemble Learning and Passive Aggressive, were used to build predictive models in this study. The other steps were the same as the above.
3. Statistical Analysis
Statistical analysis was performed using pycharm2021.2.3. Univariate multigroup data were analyzed by a one-way ANOVA. Spearman rank test was used to evaluate the correlations between continuous covariates. Kruskal-Wallis test was used to evaluate the correlations between discrete variables. A multivariate analysis was performed using multi-linear regression analysis. Exact P values and 95% confidence intervals were provided.
We built a novel missing data handling method, which can identify an optimal deleting order of columns and rows to generate an optimal intact subset.
Validation results in computer generated simulated real-world datasets
1.1 OIS. Method performed best when the power exponent of columns equaled 2
Five classification performance indicators were used to evaluate OIS.method. Through univariate analysis and multivariate analysis, we found that the multiple of redundant variables was negatively correlated with the performance of OIS. Method and the power exponent of columns was positively correlated with the performance of OIS.method(Supplementary1 Table 8–9).
OIS.method worked best when the power exponent of columns equaled 2 (AUC: 0.8195 ± 0.1903, accuracy: 0.7791 ± 0.1761, precision:0.7791 ± 0.1761, recall: 0.7714 ± 0.1930, f1-score: 0.7748 ± 0.1754).
1.2 OIS. Method performed best compared with control methods
Similarly, we use five classification performance indicators to evaluate the missing data handling methods. Through univariate analysis and multivariate analysis, we found that the multiple of redundant variables was negatively correctly with the performance of all methods(Supplementary1 Table 10–11). With the increase of redundant variables, the performance of all methods to deal with missing values decreased(Fig. 2).
The proportions of different methods in Top 10, Top20 and Top50 when introducing different multiples of redundant variables have been descripted in Radar charts(Fig. 3). There were15 radar charts have been generated totally. The larger the area of the radar chart, the greater the advantage of the missing data handling method. We can find that OIS. Method had the best performance due to the area under all radar charts were largest and the highest AUC was 1.
Validation results in real-world datasets
1 predict the risk of hypotension during dialysis in dialysis patients
Twelve machine learning models have been built for predicting the risk of hypotension during dialysis using four missing data handling methods and three machine learning algorithms. The evaluation results using five classification performance indicators were descripted in Table 3. We can find that OIS.method had the best performance(AUC: OIS.Method VS Simple Impute VS Random Forest VS Modified Random Forest, 0.8179 ± 0.0005 VS 0.8116 ± 0.0002 VS 0.8087 ± 0.0009 VS 0.8093 ± 0.0014).
| AUC | Accuracy | Precision | Recall | F1-Score | Imputing methods | Machine learning method |
|---|---|---|---|---|---|---|
| 0.8179 ± 0.0005 | 0.7459 ± 0.0020 | 0.7370 ± 0.0027 | 0.6797 ± 0.0031 | 0.7072 ± 0.0025 | OIS.method | Random Forest |
| 0.8116 ± 0.0002 | 0.7405 ± 0.0010 | 0.7333 ± 0.0015 | 0.6696 ± 0.0003 | 0.7000 ± 0.0008 | Random Forest | Random Forest |
| 0.8087 ± 0.0009 | 0.7367 ± 0.0079 | 0.7302 ± 0.0012 | 0.6625 ± 0.0013 | 0.6947 ± 0.0009 | Modified Random Forest | Random Forest |
| 0.8093 ± 0.0014 | 0.7375 ± 0.0012 | 0.7297 ± 0.0017 | 0.6662 ± 0.0026 | 0.6965 ± 0.0016 | Simple impute | Random Forest |
| 0.7468 ± 0.0026 | 0.6854 ± 0.0011 | 0.6674 ± 0.0009 | 0.6041 ± 0.0025 | 0.6342 ± 0.0017 | OIS.method | Logistic Regression |
| 0.7370 ± 0.0022 | 0.6771 ± 0.0016 | 0.6591 ± 0.0018 | 0.5923 ± 0.0023 | 0.6239 ± 0.0020 | Random Forest | Logistic Regression |
| 0.7374 ± 0.0026 | 0.6781 ± 0.0022 | 0.6601 ± 0.0025 | 0.5940 ± 0.0030 | 0.6253 ± 0.0027 | Modified Random Forest | Logistic Regression |
| 0.7362 ± 0.0029 | 0.6765 ± 0.0021 | 0.6585 ± 0.0025 | 0.5914 ± 0.0029 | 0.6231 ± 0.0027 | Simple impute | Logistic Regression |
| 0.7527 ± 0.0021 | 0.6897 ± 0.0020 | 0.6937 ± 0.0024 | 0.5600 ± 0.0030 | 0.6197 ± 0.0028 | OIS.method | Gradient Boosting |
| 0.7415 ± 0.0009 | 0.6807 ± 0.0011 | 0.6819 ± 0.0017 | 0.5510 ± 0.0016 | 0.6095 ± 0.0013 | Random Forest | Gradient Boosting |
| 0.7428 ± 0.0009 | 0.6815 ± 0.0007 | 0.6823 ± 0.0012 | 0.5535 ± 0.0010 | 0.6112 ± 0.0008 | Modified Random Forest | Gradient Boosting |
| 0.7415 ± 0.0011 | 0.6796 ± 0.0010 | 0.6797 ± 0.0017 | 0.5511 ± 0.0011 | 0.6087 ± 0.0011 | Simple impute | Gradient Boosting |
| OIS.method worked best in three machining learning models. | ||||||
2 predict the risk of drug adverse reaction in elderly patients with type 2 diabetes
Similarly, eight machine learning models have been built for predicting the risk of drug adverse reaction in elderly patients with type 2 diabetes using four missing data handling methods and two machine learning algorithms. The evaluation results using five classification performance indicators were descripted in Table 4. In Ensemble Learning model, OIS. Method can acquire the highest AUC, Recall and f1-score(0.7028 ± 0.0126, 0.43599 ± 0.0597, 0.3423 ± 0.0348). And in Passive Aggressive model, OIS.method worked best at all evaluating dimensions(AUC: 0.5831 ± 0.0513, Accuracy: 0.6505 ± 0.0747, Precision: 0.2652 ± 0.0734, Recall: 0.4102 ± 0.1038, f1-score: 0.3181 ± 0.0799). It should be noted that Ensemble Learning algorithm was the integration of Logistic Regression, Quadratic Discriminant Analysis, SVM, XGBoost and latent dirichlet allocation.
| AUC | Accuracy | Precision | Recall | F1-Score | Imputing methods | Machine learning method |
|---|---|---|---|---|---|---|
| 0.7028 ± 0.0126 | 0.7293 ± 0.013 | 0.3078 ± 0.0335 | 0.43599 ± 0.0597 | 0.3423 ± 0.0348 | OIS.method | Ensemble Learning |
| 0.6963 ± 0.0231 | 0.7118 ± 0.0696 | 0.2787 ± 0.1451 | 0.3875 ± 0.2506 | 0.2903 ± 0.1487 | Random Forest | Ensemble Learning |
| 0.6957 ± 0.0247 | 0.7483 ± 0.0706 | 0.3664 ± 0.1509 | 0.3055 ± 0.2556 | 0.2791 ± 0.1541 | Modified Random Forest | Ensemble Learning |
| 0.6699 ± 0.0249 | 0.7339 ± 0.0708 | 0.2921 ± 0.1532 | 0.2903 ± 0.2479 | 0.2454 ± 0.1498 | Simple impute | Ensemble Learning |
| 0.5831 ± 0.0513 | 0.6505 ± 0.0747 | 0.2652 ± 0.0734 | 0.4102 ± 0.1038 | 0.3181 ± 0.0799 | OIS.method | Passive Aggressive |
| 0.5507 ± 0.0625 | 0.6011 ± 0.0838 | 0.2124 ± 0.0605 | 0.3681 ± 0.1383 | 0.2609 ± 0.0695 | Random Forest | Passive Aggressive |
| 0.5625 ± 0.042 | 0.6215 ± 0.0552 | 0.2224 ± 0.0312 | 0.3792 ± 0.1401 | 0.2733 ± 0.0533 | Modified Random Forest | Passive Aggressive |
| 0.5554 ± 0.0354 | 0.6267 ± 0.0628 | 0.2317 ± 0.033 | 0.3875 ± 0.1317 | 0.2828 ± 0.0491 | Simple impute | Passive Aggressive |
| OIS.method had best performance in AUC, Recall and f1-score in Ensemble Learning model, and OIS.method had best performance in all evaluation indicators in Passive Aggressive. | ||||||
In summary, in both two real-world classification tasks, OIS. Method showed best performance on handling missing data.
The main contribution of our study is to propose a novel missing data handling method, which is more suitable for real-world datasets with large sample size and large variable size due to the small calculation. Compared with other methods, OIS.method can avoid the error caused by imputation which is especially suitable for dealing with the dataset whose distribution of missing values is unbalanced. Moreover, OIS.Method can reduce the computational complexity drastically because it will delete redundant variables to decrease the data size. We have validated the performance of OIS.Method in binary classification datasets.
Currently, many missing data handling methods have been proposed. Missing data can be imputed by calculating the similarity between each possible value of the missing data and all the values of an attribute via IS Measure and Weighted Similarity Measure18. But the process of calculation was too complicate to use in large datasets. The method based on cluster such as k-means can also be used for imputing, and the missing data will be imputed with the cluster center mean values19. However, the efficiency will degrade when a large number of samples contain missing values, and the performance on numerical data was unknown.
Additionally, some studies developed missing data imputing framework to enhance the usability. A framework incorporating KNN, decision tree, and SVM to impute the missing data was suitable for both classification and regression tasks20. It should be noted that KNN is not suitable for large dataset because for each record with missing values ,it finds k number of similar records by searching the whole dataset18.
Our study had several limitations. For one thing, OIS.Method have not been validated in regression tasks which is what we will study in future. For another thing, it is inevitable that some information will be lost after deleting columns. But OIS.Method will delete redundant variables preferentially and retain the variables that contribute significantly to the outcome.
In practice, the existence of missing values is a common problem, which makes it difficult to build prediction models that perform well21. The selection of appropriate missing value handling method is crucial for downstream statistical analysis. OIS.method can be one of the options when handle complex real-world datasets.
Data availability
Simulated datasets underlying this article will be shared on reasonable request to the corresponding author. Real-world datasets underling this article were collected from a private database, therefore, they cannot be shared publicly due to the privacy of patients who participated in this study.
Author contributions
W.J.X. authored drafts of the paper, prepared the figures and tables. C.H. analyzed the data. T.R.S. managed and coordinated responsibility for the research activity planning and execution. W.X.W. designed the experiment, and approved the final draft. All authors approved the final manuscript.
Conflict of interest: none declared.
Ethic Approval: The real-world datasets in our study were retrospectively collected. Due to the retrospective nature of the study, Ethics Committee of Sichuan Provincial People's Hospital waived the need of obtaining ethical approval, informed consent. All of steps in our study were in accordance with relevant guidelines and regulations.
Funding
This work was supported by National Natural Science Foundation of China. [grant number:72004020]
Supplementary data
Supplementary data are available in Supplementary file.
- Larkins, N. G., Craig, J. C. & Teixeira-Pinto, A. A guide to missing data for the pediatric nephrologist. Pediatr Nephrol 34, 223-231, doi:10.1007/s00467-018-3932-4 (2019).
- Narayan, S. W. et al. Missing data reporting in clinical pharmacy research. Am J Health Syst Pharm 76, 2048-2052, doi:10.1093/ajhp/zxz245 (2019).
- Austin, P. C., White, I. R., Lee, D. S. & van Buuren, S. Missing Data in Clinical Research: A Tutorial on Multiple Imputation. Can J Cardiol 37, 1322-1331, doi:10.1016/j.cjca.2020.11.010 (2021).
- Sıva Subramanıan, R., Prabha, D., Maheswari, B. & Aswini, J. in Innovative Data Communication Technologies and Application Lecture Notes on Data Engineering and Communications Technologies Ch. Chapter 2, 17-31 (2022).
- Pedersen, A. B. et al. Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol 9, 157-166, doi:10.2147/CLEP.S129785 (2017).
- Liu, Q. & Hauswirth, M. in 11th IEEE Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON). 349-358 (2020).
- Alamoodi, A. H. et al. Machine learning-based imputation soft computing approach for large missing scale and non-reference data imputation. Chaos, Solitons & Fractals 151, doi:10.1016/j.chaos.2021.111236 (2021).
- Li, J., Wang, M., Steinbach, M. S., Kumar, V. & Simon, G. J. in 2018 IEEE International Conference on Big Knowledge (ICBK) 415-422 (2018).
- Zhang, Z. Missing data imputation: focusing on single imputation. Ann Transl Med 4, 9, doi:10.3978/j.issn.2305-5839.2015.12.38 (2016).
- Ngueilbaye, A., Wang, H., Mahamat, D. A. & Junaidu, S. B. Modulo 9 model-based learning for missing data imputation. Applied Soft Computing 103, doi:10.1016/j.asoc.2021.107167 (2021).
- Sterne, J. A. et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338, b2393, doi:10.1136/bmj.b2393 (2009).
- Dai, Z., Bu, Z. & Long, Q. Multiple Imputation via Generative Adversarial Network for High-dimensional Blockwise Missing Value Problems. Proc Int Conf Mach Learn Appl 2021, 791-798, doi:10.1109/icmla52953.2021.00131 (2021).
- Tsvetanova, A. et al. Missing data was handled inconsistently in UK prediction models: a review of method used. J Clin Epidemiol 140, 149-158, doi:10.1016/j.jclinepi.2021.09.008 (2021).
- Emmanuel, T. et al. A survey on missing data in machine learning. J Big Data 8, 140, doi:10.1186/s40537-021-00516-9 (2021).
- Si, Y. et al. Multiple Imputation with Massive Data: An Application to the Panel Study of Income Dynamics. Journal of Survey Statistics and Methodology, doi:10.1093/jssam/smab038 (2021).
- Beesley, L. J. et al. Multiple imputation with missing data indicators. Stat Methods Med Res 30, 2685-2700, doi:10.1177/09622802211047346 (2021).
- Wu, M. T. Confusion matrix and minimum cross-entropy metrics based motion recognition system in the classroom. Sci Rep 12, 3095, doi:10.1038/s41598-022-07137-z (2022).
- Zahin, S. A., Ahmed, C. F. & Alam, T. An effective method for classification with missing values. Applied Intelligence 48, 3209-3230, doi:10.1007/s10489-018-1139-9 (2018).
- Myneni, M. B., Srividya, Y. & Dandamudi, A. in Proceedings of the First International Conference on Computational Intelligence and Informatics Advances in Intelligent Systems and Computing Ch. Chapter 17, 171-178 (2017).
- Bertsimas, D., Pawlowski, C. & Zhuo, Y. D. From Predictive Methods to Missing Data Imputation: An Optimization Approach. Journal of Machine Learning Research 18 (2018).
- Kang, S. Product failure prediction with missing data using graph neural networks. Neural Computing and Applications 33, 7225-7234, doi:10.1007/s00521-020-05486-2 (2020).
No competing interests reported.