Evaluations of imputation methods for missing data with random forest modeling: an application with unbalanced data with a three categories outcome, the linkage to HIV care Uganda study

doi:10.21203/rs.3.rs-1999473/v2

Download PDF

Research Article

Evaluations of imputation methods for missing data with random forest modeling: an application with unbalanced data with a three categories outcome, the linkage to HIV care Uganda study

https://doi.org/10.21203/rs.3.rs-1999473/v2

This work is licensed under a CC BY 4.0 License

Version 2

posted

You are reading this latest preprint version

Background: Incomplete observation units may contain important information about the population being studied, analysis conducted with the complete portion of the dataset, done in most standard statistical software, may produce biased or low statistical power results. In this paper we investigate the behavior of imputation methods for a linkage to HIV care study dataset with more than 60% incomplete cases.

Methods:Missing data imputation algorithms amelia, missForest, mice and hmisc were considered. Two sets of simulations were conducted: first with the subset of data containing only complete observations and second with the whole dataset. Imputation accuracy and general behavior for each imputation algorithm were accessed in the first set. Random forest models were fit to evaluate overall prediction accuracy and sensitivity in both sets of simulations.

Results: The imputed values by missForest, a single imputation method, were more accurate for all incomplete variables and scenarios. Median overall prediction accuracy of HIV status, a three levels outcome, was slightly higher after imputations with missForest and amelia (approximately 52%, 61% and 65% for samples of 350, 700 and 1050 units respectively). In general, different missing percentages (20%, 40% and 60%) did not result in large changes in prediction or imputation accuracy and larger sample sizes performance improved for imputation and overall prediction accuracy but mixed results were observed with sensitivity. For the second set of simulations, missForest and amelia produced better overall prediction accuracy with larger samples (1200 versus 350 units). For new HIV positive HIV class, hmiscand mice presented better sensitivity, with median sensitivity 65% versus 50% and approximately 90% versus 20% for samples with 350 and 1200 units respectively, compared to incomplete samples.

Conclusions: Imputation with mice and hmisc may be used for imputation when the main interest is to increase sensitivity in predicting new HIV positive. Furthermore, it is interesting to investigate which method gives more advantage for other performance measures of interest.

Trial registration: NCT02545673

missing data imputation

random forest classification

amelia

hmisc

mice

missForest

multiple imputation (MI)

single imputation (SI)

NRMSE (normalized root square mean squared errors).

Standard statistical software such as R, STATA and SAS fit most statistical models and perform analysis using only observations that contain values for all variables in the model; this is referred to as complete case analysis (CCA). Incomplete cases are excluded in CCA but may contain relevant information [1].

The linkage to HIV care dataset [2] is composed of information on 23,937 individuals, and each represents one observational unit. For the focus of this study, 27 variables are considered for predicting HIV status, of which, four have missing values for numerous observations. Although missing values represent less than 4% of all data points, they are present in about 61% of the observations. Hence, only 39% of the data is used if CCA is conducted.

In this study, HIV status assumes three categories or classes: newly diagnosed positive (new positive), known positive and negative, with the minority class of new positive being of special interest. In the complete portion of the dataset, there are over 8,000 negative cases but only 140 new and 413 known positive cases, representing approximately 94%, 1.5% and 4.5% of the observations. When including the incomplete portion of the dataset, these numbers increase to 454, 653 and over 22,000 cases, respectively. Because predictive models for categorical outcomes work best with balanced datasets, missing values in any of the covariates in a model for classification of HIV status, when outcome is in the minority classes, may disproportionately affect the ability to predict those classes. Imputation will increase the information available in the minority classes and potentially improve prediction accuracy.

To obtain insight and provide general guidance for imputation for missing data, R packages amelia [3], missForest [4], mice [5] and hmisc [6], including single and multiple imputation methods were evaluated. For this, a variety of simulation scenarios were considered using the complete portion of the dataset. Then, results from this analysis were used to support the imputation into the full dataset, that not only recovered well the missing data, but more importantly lead to accurate classification models with the HIV care dataset.

While most imputation for missing data publications usually have continuous or binary outcome, this study explores the imputation methods’ performance when a categorical outcome with three levels is of interest. This is done to investigate the efficacy of these methods for our dataset and analysis of interest, and to generate general guidelines and support literature for further investigation with this type of dataset and analysis, which sometimes provide unique challenges in producing satisfactory prediction results. Furthermore, we suggest a weighted ranking system for the imputation methods considering performance of all incomplete variables at once, this ranking system can be adapted for considering several performance measures of interest at once.

This paper is organized as follows: first, materials and methods are presented, followed by results and conclusions with discussion and general guidelines for imputation.

This section is divided in four subsections: the first one introduces the linkage to HIV care dataset, the second gives an overview on missing data imputation and imputation approaches. The third subsection presents an overview of the imputation algorithms and models considered in this study, the fourth describes the simulations performed and defines and discusses the performance measures chosen for evaluating imputation precision and prediction accuracy for modeling.

The Linkage to HIV Care Study Dataset

The data were collected in rural Uganda as part of the PATH/Ekkubo study [2] which tested an intervention to increase the percentage of individuals diagnosed with HIV who promptly enroll in HIV care, start treatment, and successfully achieve viral suppression indicating treatment success and reducing the likelihood of transmitting the virus to others. The study was implemented in the context of door-to-door home-based HIV testing. The present paper uses data from the baseline questionnaire collected between November 2015 and December 2018 from all individuals who consented to be tested for HIV in the home-based HIV testing and to complete a questionnaire interview.

For this study, 27 socio-demographic, psychosocial, and health variables were selected as possibly relevant when considering HIV status outcome. The complete sample consisted of 23,937 participants from 26 different villages in Butambala, Mpigi, Gomba and Mityana districts of Central Uganda. Of which, 9,099 cases are completely observed. The 14,838 cases left do not have information for alcohol consumption risk level and depressive symptoms score because these measures were added to the questionnaire in November 2017. Additionally, of the 14,838 incomplete cases, 2,612 also have missing information for occupation because this measure was changed during the course of the study and 76 were also incomplete for pregnancy status, for unknown reason. Alcohol consumption risk level and depressive symptoms score are represented by an integer ranging from 0 to 12, and 0 and 30, respectively. Depressive symptoms were measured using the 10-item Center for the Epidemiological Studies of Depression Short Form (CESD-10) as defined in [7]. Alcohol consumption risk level was assessed using the Alcohol Use Disorders Identification Test-Concise (AUDIT-C), defined in [8]. Pregnancy and occupation are categorical variables with three and six unordered classes, respectively.

Descriptive statistics for numeric variables such as mean, standard deviation and range, in addition to 95% confidence interval for the difference of means, by complete and incomplete cases are presented in Table 1. A few variables presented a relatively larger difference of means - mean monthly income, individual level anticipated stigma from family or health care workers due to HIV status and years of education values were higher for incomplete cases, and age was lower.

The distributions of complete and incomplete cases for log of income, stigma from family and health care workers and years of education are shown in Figure 1. The plots suggest that the incomplete portion of the sample represents a more affluent sector of the population that might coincide with more years of education, and also that a portion of the incomplete cases anticipated a higher stigma from health care workers and family due to HIV status.

Table 1: Baseline mean and means difference for continuous variables, by completeness of cases.

	Complete	Incomplete	Total	Means Difference
	n = 9099	n = 14838	n = 23937
Variable	mean (sd) min; max	mean (sd) min; max	mean (sd) min; max	95% CI
village population (x 1000)	2.86 (0.92) 0.55; 3.97	2.82 (1.15) 0.44; 4.42	2.84 (1.07) 0.44; 4.42	(0.01; 0.07)*
age centered at 14	17.34 (11.11) 0; 45	15.05 (10.07) 0; 45	15.92 (10.5) 0; 45	(2.01; 2.57)**
monthly income in dollars	24.68 (41.71) 0; 833	38.77 (125.14) 0; 11111	33.42 (102) 0, 11111	(-16.28; -1.9)**
time to health clinic (x 15 minutes)	1.96 (1.99) 0.07; 28	1.65 (1.45) 0; 30	1.77 (1.68) 0; 30.33	(0.27; 0.37)**
Anticipated stigma disclosure concern	2.57 (1.06) 0; 4	2.57 (1.05) 0; 4	2.57 (1.05) 0; 4	(-0.03; 0.03)
Anticipated HIV stigma from family	1.08 (0.89) 0; 4	1.26 (0.99) 0; 4	1.19 (0.96) 0; 4	(-0.21; -0.17)**
Anticipated HIV stigma from health care workers	0.81 (0.64) 0; 4	0.91 (0.74) 0; 4	0.87 (0.70) 0;4	(-0.12; -0.08)**
Depressive symptoms	5.83 (5.65) 0; 30	-	5.83 (5.65) 0; 30	-
Alcohol consumption risk level score	1.09 (2.2) 0; 12	-	1.09 (2.2) 0; 12	-
Village HIV prevalence	0.06 (0.02) 0.03; 0.1	0.04 (0.01) 0.02; 0.05	0.05 (0.02) 0.02; 0.1	(0.02; 0.02)**
Enacted stigma village level	0.98 (0.12) 0.8; 1.4	1.05 (0.16) 0.8; 1.4	1.03 (0.15) 0.8; 1:43	(-0.07; -0.07)**
Anticipated stigma due to HIV disclosure concern village level	2.57 (0.26) 2.2; 3.0	2.57 (0.49) 1.9; 3.3	2.57 (0.42) 1.9; 3.3	(-0.01; 0.01)**
Anticipated stigma public attitudes village level	1.65 (0.27) 1.4; 2.3	1.62 (0.21) 1.4; 2.0	1.64 (0.23) 1.4; 2.3	(0.02; 0.04)
Anticipated HIV stigma from family village level	1.07 (0.13) 0.9; 1.5	1.26 (0.12) 1.1; 1.7	1.19 (0.15) 0.9; 1.7	(-0.19; -0.19)**
Anticipated HIV stigma from health care workers village level	0.81 (0.09) 0.6; 0.9	0.92 (0.16) 0.7; 1.4	0.88 (0.15) 0.6; 1.4	(-0.11; -0.11)**
Years of education	6.55 (3.49) 0; 15	8.08 (3.35) 0; 15	7.5 (3.48) 0; 15	(-1.61; -1.43)**
Note. Significance for difference of means: * p-value <0.001, p-value <<0.00001, CI = Confidence Interval for mean difference, sd = standard deviation, min = minimum, max = maximum, bold** shows values with larger differences.

Categorical variables are presented in Table 2, along with frequencies and percentages. P-values for the difference in frequency between complete and incomplete were all smaller than 0.001 and are not shown in the table. Some variables and categories with seemingly larger differences are emphasized in bold such as wealth, occupation, HIV and marital status.

The distributions of the incomplete variables, by HIV status are shown in Figures 2 and 3. For depressive symptoms score, the distribution is similar for negatives (-) and new positives (new +), and presents higher values and more spread for known positives (known +). For alcohol consumption risk level, new positives have wider spread and higher median values than known positives, which, in their turn have wider spread and higher values than negatives. For categorical variables, the figures are also grouped by complete and incomplete cases; incomplete cases being the ones that have information for the categorical variable but not for alcohol and depressive symptoms variables. Figure 3 shows there is a slight difference in proportions of pregnancy status for complete (C) and incomplete (I) cases across the three classes of HIV status is observed. And there is a higher difference in the proportion of pregnancy in complete versus incomplete cases for new positives than in the other two HIV classes. For occupation variable, there is a difference in distribution in each HIV class in occupation levels when comparing complete and incomplete cases. For new positives, there is a larger proportion in the fisherman and peasant farmer categories in complete cases. There is a higher proportion of salaried tradesperson in incomplete cases, while the proportion of people not employed outside of the house seems to be similar across complete and incomplete cases for all HIV classes.

Table 2: Baseline frequency distribution for categorical variables by completeness of cases.

Variable	Categories	Complete (n = 9099) n (row %)	Incomplete (n = 14838) n (row %)	Total (n = 23937) n (row %)
HIV status	Negative	8546 (93.9)	14284 (96.3)	22830 (95.4)
	New Positive	140 (1.5)	314 (2.1)	454 (1.9)
	Known Positive	413 (4.5)	240 (1.6)	653 (2.7)
gender	Male	3342 (36.7)	6707 (45.2)	10049 (42.0)
	Female	5757 (63.3)	8131 (54.8)	13888 (58.0)
religion	Muslim	2184 (24.0)	3870 (26.1)	6054 (25.3)
	Christian & non-Muslim	6915 (76.0)	10968 (73.9)	17883 (74.7)
Wealth index	lowest quintile	3356 (36.9)	1517 (10.2)	4873 (20.4)
	2nd lowest quintile	2311 (25.4)	2971 (20.0)	5282 (22.1)
	3rd lowest quintile	1590 (17.5)	2867 (19.3)	4457 (18.6)
	4^th lowest quintile	984 (10.8)	2850 (19.2)	3834 (16.0)
	highest quintile	858 (9.4)	4633 (31.2)	5491 (22.9)
away for work 1 month or more	not away	8039 (88.4)	12677 (85.4)	20716 (86.5)
away for work 1 month or more	one or more times	1060 (11.6)	2161 (14.6)	3221 (13.5)
transportation	free: walking/bike	4882 (53.7)	7370 (49.7)	12252 (51.2)
	low cost: taxi	958 (10.5)	2870 (19.3)	3828 (16.0)
	high: boda/car	3259 (35.8)	4598 (31.0)	7857 (32.8)
marital status	Never married	1618 (17.8)	4498 (30.3)	6116 (25.6)
	married	1556 (17.1)	2204 (14.9)	3760 (15.7)
	widowed/divorced	5925 (65.1)	8136 (54.8)	14061 (58.7)
another household member is HIV +	No	7260 (79.8)	12067 (81.3)	19327 (80.7)
	Yes	390 (4.3)	442 (3.0)	832 (3.5)
	Do not know	1449 (15.9)	2329 (15.7)	3778 (15.8)
		n = 9099	n = 14762	n = 23861
pregnancy of self or partner	No	7272 (79.9)	10918 (74.0)	18190 (76.2)
	Yes	906 (10.0)	1236 (8.4)	2142 (9.0)
	No partner	921 (10.1)	2608 (17.7)	3529 (14.8)
		n = 9099	n = 12226	n = 21325
occupation	Peasant farmer	3998 (43.9)	3046 (24.9)	7044 (33.0)
	Casual worker	839 (9.2)	1182 (9.7)	2021 (9.5)
	Salaried trade	750 (8.2)	2988 (24.4)	3738 (17.5)
	fish/csw/rest/bar/attendant	430 (4.7)	83 (0.7)	513 (2.4)
	business selling	1287 (14.1)	1799 (14.7)	3086 (14.5)
	not employed out home	1795 (19.7)	3128 (25.6)	4923 (23.1)
Note. n = sample size, bold shows values with larger differences.

As mentioned before, if CCA is used for this dataset, more than 60% of the cases will be discarded when alcohol consumption or depressive symptoms variables are included in the analysis model. Bias may be introduced due to some systematic differences between complete and incomplete cases. Furthermore, critical information is potentially lost when attempting to predict new positive HIV cases via any classification method because of the small number of complete cases in this category. To predict HIV status, random forest (RF) models [9] were used for the prediction models because they were found to perform well in preliminary analysis.

Missing Data Imputation

As in our linkage to HIV care dataset, missing values are common in socio-demographic, environmental, medical, epidemiologic, political and biological sciences research. Therefore, when exploring a dataset, if missing data is present, it is important to determine how to proceed. If missing data is random, sparse and does not decrease the sample size significantly, CCA may be efficient. On the other hand, when there are nonrandom missing patterns or a high percentage of incomplete cases, CCA may produce biased results and loss of statistical power [10,11]. When adequate, CCA is advantageous for its simplicity and ease of implementation. There are also some specific cases in which CCA is appropriate even when missing data is not random. For example, coefficient estimates may be unbiased when linear or logistic regression is the analysis model and missing data does not depend on the outcome, after taking covariates into consideration by including them in the model. Hughes et al. [12] presents a rich discussion of when CCA leads to bias or loss of precision.

When the impact of missing data is not negligible, missing data imputation may be a suitable alternative. Single imputation (SI) and multiple imputation (MI) techniques use information from observed values to fill in values for the unobserved ones.

Single imputation consists of replacing each missing value in the dataset by a plausible one, and then CCA is conducted in this newly completed dataset [13]. The imputed value may be the target variable mean, range or some type of regression prediction, for example. The main problem found with this approach is that the uncertainty due to guessing the unobserved values is not accounted for, leading to two main issues: variances of regression estimates tend to be underestimated, generating low coverage confidence intervals and coefficient estimates may be biased, as Buuren explains in [14]. More efficient SI techniques such as Maximum Likelihood (ML) and Expectation Maximization (EM) algorithms have been developed and generally result in less bias, but according to Clavel et al. [10] the reliability of results is hard to assess depending on patterns and percentage of missing data. The missForest package in R [4] is the SI method used for this study.

Multiple imputation consists in creating several complete datasets. Each dataset contains the same information on the observed values, and imputed values for the missing ones, these may be distinct in each complete dataset, so that the datasets generated are potentially different [14]. This difference inflates the variance of model parameters, therefore taking into account the uncertainty of the imputations [1, 13,14]. A CCA analysis is carried out in each complete dataset and results are combined into one. The rule presented by Rubin, D.B. [15] is applied for averaging analysis model parameters and combining their variances. Most MI methods generate five complete data sets as a default, and according to Buuren [14] and Clavel et al. [10], this is usually sufficient to get a valid estimate for the imputation variance. For this study, the MI approach will be evaluated with R packages amelia [3,16,17], mice [5,14] and hmisc [6].

Different imputation methods included in this study use different imputation models for imputation of missing data. An imputation model has a target variable as its outcome and all the other variables as predictors or auxiliary. Predictions from the imputation model, which is fit using the observed values, are used to impute into the missing values. More information in the imputation model will create better imputations because imputation is predictive [17], therefore, the imputation model should include at least as much information as the analysis model of interest, including transformations and interactions of variables. In general, all variables should be considered as auxiliary, even when there is not a difference in distribution between complete and incomplete data, and also include the other incomplete variables, which can have their own imputed values in the missing ones.

The two main MI approaches are Joint multivariate (JM) and fully conditional specification (FCS). In JM, it is assumed that the data follows an approximately joint distribution. The algorithm uses observed data to estimate parameters for the joint distribution, randomly samples from it and then imputes missing values to create complete data sets. The amelia package follows the JM approach. With the FCS approach, each target variable has its own imputation model defined, conditional on auxiliary variables from the dataset, which may or may not be predictors for the analysis model. Mice and hmisc packages follow an FCS approach.

In general, when doing multiple imputation, the missing data may depend on observed values but cannot depend on any unobserved data, and this missing data mechanism is called missing at random (MAR). The linkage to HIV care dataset contains data with MAR characteristics; incomplete cases come from specific villages and from a set period of the study that collected the data, which are both known information, along with several observed variables that contain information that may support and inform the imputation process.

Imputation Algorithms and Models

The amelia package was first presented by King et al. [17], and later discussed for time-series cross-section data and its algorithm updated by Honaker & King [16,3]. Amelia assumes missing data MAR mechanism and joint multivariate normal data distribution. For binary and ordered categorical variables, continuous values are imputed and then rounded to the nearest category. Unordered categorical variables are broken into binary indicators for each category, they are then manipulated in a fashion where imputed values represent parameters for a multinomial distribution and then categories are imputed based on the distribution probabilities [1]. Amelia runs an EM algorithm to estimate the mean and variance of the joint normal distribution. This is done once for each imputed dataset, each based on a different bootstrap sample of the data. Bootstrapping is done so that each complete dataset generated has potentially different imputed values.

The multivariate imputation by chained equations (mice) package in R has consistently and recently been updated for functionalities such as MI for data that is missing not at random (NMAR), tree based models imputation and multi-level data imputation. In mice, an imputation model for the conditional distribution estimation for each type of target variable is defined. Several options are available, depending on the apparent distribution and prior knowledge of each variable. The parameters of the imputation model are estimated via a pseudo Gibbs Sampler algorithm. The imputation models in mice investigated in this study are linear and multinomial regression for numeric and categorical variables respectively, and classification and regression trees for both variable types.

The Harrell Miscellaneous (hmisc) package in R, presented by Alzola & Harrell [6] has numerous functionalities for data analyses, advanced graphics and tables, sample size, power computation, missing data imputation among several others. The aregImpute function from hmisc conducts MI and follows the same FCS approach as mice, imputing one variable at a time, estimating its distribution conditional on other variables. The main difference from mice is in its imputation model, which is based on additive semi-parametric regression, where the models are defined based on transformations of the target variable and the predictors. The function aregImpute uses alternating conditional expectations (ACE) or additive and variance stabilization for regression (AVAS) functions to find which transformations to apply. Details for these functions were presented by Breiman [18], Tibshirani [19] and Banks [20]. Yadav & Roychoudhury [21] assessed imputation time for different size dataset and found hmisc to be a much faster algorithm than missForest and mice, so it is a good alternative if computation time is a concern.

Both mice and hmisc use predictive mean matching in their algorithms. Predictive mean matching works with the imputation model to impute values that are similar to an observed value. First, the predicted values for all cases in the target variable are calculated via the imputation model; then a donor is randomly selected from a small group of complete cases. The donor is an observed case whose predicted value is similar to the predicted value for the incomplete case. This works well under the assumption that missing values and complete cases with close predicted values follow similar distributions.

The missForest package in R also follows a somewhat FCS approach: it conducts imputation one target variable at a time instead of assuming a joint distribution for the data, but imputes only once. It is a nonparametric SI algorithm, generating a single imputed dataset based on predicted values from an RF model fit for each target variable. The algorithm was introduced with details by Stekhoven & Bühlmann [4], and presented promising results; it produced more accurate imputations than parametric mice, especially when dealing with categorical and numeric variables at once, and it was also found to be particularly useful in the presence of complex interactions and non-linear relationships amongst variables. MissForest was found to perform similarly to mice by Penone et al. [22] and Waljee et al. [23] with a life-history traits database and a missing laboratory data application, respectively. Although Shah et al. [24] found that imputation with missForest resulted in biased regression parameters on the analysis model when compared to mice, the analysis model used for our linkage to HIV care dataset is an RF, and no applied study was found with similar setup. The linkage to HIV care dataset may contain complex relationships amongst variables, difficult to explain with a parametric model, and has numeric and categorical variables that are incomplete, which are situations in which missForest is designed to perform well.

Simulations

Two sets of simulations were considered: the first one with samples taken from the complete portion of the dataset, with simulated missing data and then imputations were made considering only the information in each selected sample, and the second with samples taken from the full dataset, with missing data as observed in the linkage to HIV care dataset and imputations made considering information from the full dataset, not just the selected sample, taking advantage of a larger amount of data to inform the imputation process.

For the first set of simulations, nine scenarios were set up for each imputation method, using samples from the complete portion of the linkage to HIV care dataset. Samples sizes were 350, 700 and 1,050. Missing values were introduced in 20%, 40% and 60% of observations in the four target variables – alcohol consumption, depression symptoms, pregnancy and occupation. To simulate missing data, a completely randomized list of the same observed cases was set to have missing values for all four variables at once. Different configurations of tuning parameters were explored in preliminary simulations for each imputation method, and the best performing ones were selected. For MI methods, five multiple imputed datasets were generated each time. One thousand simulation runs were performed for each scenario.

To illustrate one simulation with an MI method, consider one sample with 350 cases taken from the original 9,099 complete cases via stratified random sampling to have a sample balanced for the output HIV status. Then, 20% of the cases are randomly selected to have missing values in the four target variables. Next, the MI algorithms generates five complete (imputed) datasets, using information contained in the 27 variables in the 350 cases. Then, five estimates of imputation and modeling precision are calculated and averaged; the measures used to evaluate precision are explained later in this section. Similar steps are repeated for 1,000 samples. The same is done for 40% and 60% missing values, and samples with 700 and 1,050 cases.

Because mice offers several options, three different configurations were tested and are referred to as mice, mice.rf and mice.cart. For mice, parameters were kept with their default values, so categorical variables were imputed using a politomic logistic regression model and numeric variables based on a linear regression model, both combined with predictive mean matching. When using mice.rf, a random forest was created to predict and impute in each target variable. For mice.cart, one optimal tree was created to impute the data for each of the five complete generated datasets.

For imputation with hmisc, transformations in the numeric variables were set to be linear. The predictive mean matching type was set in a way that predicted values for both target incomplete and complete observations come from a fit from the same bootstrap sample. For some samples, after simulating missing data, there were not enough values in all classes for all categorical variables for the hmisc algorithm to fit its imputation model. Because of that, the total number of simulations with this method were slightly smaller than 1,000 for the following specific scenarios - simulations with 350 cases, with 40% and 60% missing values successfully ran 995 and 861 times, respectively, and with 700 cases and 60% missing, 998 times.

For imputations with missForest, a single imputation method, the results of each simulation are based on one single forest with ten trees generated for each target variable. Other parameters were kept with their default values. Forests with up to 100 trees were evaluated, but differences in imputation and prediction accuracy were minimal and are not reported. Therefore, the number of trees for the forests in missForest and mice.rf imputation was set to ten, which allowed for similar precision to larger forests but improved imputation speed.

In this first set of simulations, mice.rf and mice.cart did not offer consistent advantages over mice with default configurations and were more time consuming, so these configurations were not included in the simulations with the full dataset. This first set of simulations was designed with the goal of investigating if any of the imputation methods performed significantly better or were more time efficient for selected sample sizes, missing percentages or performance measures, given the dataset of interest with its intrinsic characteristics of mixed numeric and categorical variables, collinearity, possible nonlinear relations and interaction between the variables.

Based on the results and information acquired from the initial imputation simulations, a second set of simulations was designed with the main goal being to assess the performance of imputation given the incomplete HIV dataset to be analyzed. In this step, two scenarios were considered after imputing into the full linkage to HIV care dataset; samples with 350 and 1,200 observations. These were chosen because there is a limited number of units in the positive classes for the HIV status variable, so in order to select samples with no class imbalance without oversampling or using synthetic samples, which was found to be not very effective for this dataset, and also with different HIV new positive cases between different samples, samples sizes were chosen to be slightly smaller than three times the number of observations in the minority class. Missing values were as observed for each target variable – about 60% for alcohol consumption and depression symptoms, 11% for occupation and 0.3% for pregnancy. Here, amelia, mice, hmisc and missForest were used to impute into the full linkage to HIV care dataset. The selected samples included negative HIV from the complete portion of the dataset only, and new and known positive HIV from the complete and imputed data. Imputation increases cases available for modeling in the HIV positive classes, from 140 to 454 new positives and from 413 to 653 known positives. Imputation adds some variability due to uncertainty of imputed values. Therefore, we chose to limit this variability to the inclusion of the additional positive cases, which are needed for increasing sample size for modeling, but there was no need to include the 14,284 originally incomplete cases in the negative HIV class, this is done under the assumption that negatives are well represented in the 8,546 originally complete cases. The setting including imputed observations from all HIV classes was tested and it resulted in worse results altogether - lower prediction accuracy, and no improvement in sensitivity, with much larger variability for predicting HIV negative and no advantage on predicting HIV positive classes.

The performance measures of interest are described in the following paragraphs. For the first set of simulations, because the real observed values are known, the precision of imputed values can be evaluated via misclassification rate, normalized root square mean squared errors (NRMSE), and mean rank. The calculations for obtaining these measures are described below:

categorical target variables - pregnancy and occupation: misclassification rate is calculated by comparing the imputed value to the original value and obtaining the wrongly classified percentage,
continuous target variables - alcohol consumption risk level and depressive symptoms score: NMRSE, which is obtained by calculating the mean squared distance between observed and imputed values, then taking the square root divided by the standard deviation. With V representing the variance, the i^th imputed value, y_i the i^th observed value and n the total number of cases, then,
all target variables together: mean rank for the methods, which is a proposed measure used to summarize how each method performs compared to the others. It is based on the NRMSE and misclassification rate mean values considering all simulations at once, and is obtained for one simulation scenario by the following steps:

Obtain mean NRMSE for alcohol consumption and depression symptoms, and mean misclassification rate for pregnancy and occupation, separately by imputation method.
Sort the methods by mean NRMSE for alcohol consumption. Rank methods from smallest to largest mean NRMSE, obtaining numbers 1 through 6 (there are six methods configurations), call this column Rank 1.
Repeat step 2 for mean NRMSE for depressive symptoms, and call this Rank 2.
Repeat step 2 for mean misclassification rate for pregnancy and occupation, to obtain Ranks 3 and 4.
Take the average of the four ranks to obtain the mean rank for each method.

Although measurements based on MSE and misclassification rate can quantify the ability of each method to impute values similar to the observed ones, such measures could not answer practical analytical questions such as if the quality of the analysis models improved after imputing data versus when conducting CCA. Thus, we evaluate the performance of the analysis model before and after imputation as well, and this is done for both sets of simulations.

Several measures for assessing an RF model’s performance can be obtained via the information from the confusion matrix, such as accuracy, sensitivity, specificity and Receiving Operating Characteristic (ROC) curve. For our analysis model, prediction accuracy and sensitivity were the measures chosen to be extracted before and after imputation using the function confusionMatrix from the caret package in R [25]. Accuracy is the measure that represents the percentage of all cases in the test sample that are predicted in their correct HIV status by the analysis model. Sensitivity for a specific class is the ratio of the number of cases in that class that are correctly predicted over the total of cases observed in the class, and it varies from 0 to 1, being 1 perfect prediction, and it provides a way of evaluating the performance considering each HIV status separately. Accuracy and sensitivity calculations were done using the RF fit to predict a test sample with 500 observations, with 2% in each positive HIV status. This percentage was selected in order to be able to estimate how well the models might perform if applied for a population with approximately similar percentages in each HIV status to the ones observed in the full linkage to HIV care dataset - 1.9% and 2.7% for new and known HIV positive, respectively.

When modeling with the linkage to HIV care dataset, it was important to take into account that class imbalance is present in the outcome variable HIV status and, because of that, caution must be taken when evaluating prediction accuracy for the analysis models. Leevy et al. (2018) argued that in such cases, it is challenging to identify the minority class because a high class imbalance introduces a bias in favor of the majority class, and a classifier that predicts all cases in the majority class may produce a deceptively high accuracy rate. Thus, in order to avoid models that may attempt to maximize prediction accuracy by classifying all observations in the negative HIV class, samples for training the RF models are balanced for HIV status variable, with approximately a third in each class. When there are not enough observations in the new HIV positive class, synthetic cases are created via SMOTE (Synthetic Minority Over-sampling Technique). Smote is done via the SmoteClassif function from the UBL package in R. This is an attempt to increase the decision region for the minority class. This technique usually helps to increase the proportion of true positives predicted by the classification model, with increased proportion of false negatives being the payoff [26].

In preliminary simulations, the sensitivity for the class of the new HIV positive got increasingly worse with larger sample sizes, even when class imbalance was not present via the use of oversampling or SMOTE. Therefore, taking into consideration the limitations of modeling with a small number of cases in the minority class, the sample sizes for simulations were kept between about three times the number of available cases in the new HIV positive class.

For the first set of simulations, the NRMSE and misclassification rates were averaged by method for each of the four target variables in each scenario to summarize precision of imputation. Results for samples with 1,050 cases are presented in Table 3, along with the overall mean rank of each imputation method. NRMSE is shown in the original scale of the dataset, and misclassification rates are presented in percentage of incorrect predictions. Smaller values represent more precise imputation. Throughout Table 3, it can be seen that for every simulation scenario, missForest imputed values closest to the actual ones, followed by mice configurations and amelia, then hmisc. Scenarios with 40% missing values in the target variables, presented intermediate results between ones with 20% and 60%, for all sample sizes and methods, and on that account, for clearer visualization and interpretation, results with 40% missing were omitted from upcoming plots. For smaller samples, imputation precision results are not shown but are similar when comparing methods’ performance via ranks.

Table 3: Mean NRMSE and standard deviation for alcohol consumption and depressive symptoms, misclassification rate for pregnancy and occupation and mean rank by imputation method, for samples with 1,050 cases and missing percentages of 20%, 40% and 60%.

Imputation method	missing %	alcohol consumption nrmse mean (sd)	depressive symptoms nrmse mean (sd)	pregnancy misclassif. rate (%) mean (sd)	occupation misclassif. rate (%) mean (sd)	mean rank
missForest	20	0.91 (0.07)	0.91 (0.05)	16 (2.1)	50 (1.6)	1.00
mice.cart		1.26 (0.04)	1.27 (0.04)	29 (2.5)	68 (3.5)	2.25
mice.rf		1.23 (0.07)	1.29 (0.05)	31 (2.3)	70 (2.0)	3.25
mice		1.30 (0.07)	1.35 (0.06)	29 (2.7)	70 (1.7)	4.00
amelia		1.28 (0.04)	1.32 (0.05)	38 (2.3)	76 (2.2)	5.00
hmisc		1.41 (0.08)	1.41 (0.06)	32 (2.3)	75 (2.5)	5.50
missForest	40	0.92 (0.05)	0.92 (0.04)	16 (1.7)	52 (1.4)	1.00
mice.cart		1.27 (0.03)	1.28 (0.03)	28 (1.9)	68 (2.5)	2.25
mice.rf		1.23 (0.05)	1.29 (0.04)	30 (1.8)	70 (1.2)	3.00
mice		1.30 (0.06)	1.35 (0.05)	29 (1.8)	70 (1.7)	4.25
amelia		1.28 (0.03)	1.32 (0.04)	38 (2.0)	76 (2.2)	5.00
hmisc		1.42 (0.06)	1.41 (0.05)	32 (2.0)	75 (2.3)	5.50
missForest	60	0.95 (0.05)	0.94 (0.04)	17 (1.8)	54 (1.4)	1.00
mice.cart		1.28 (0.02)	1.29 (0.02)	28 (1.7)	68 (2.3)	2.25
mice.rf		1.24 (0.05)	1.30 (0.04)	29 (1.6)	69 (1.1)	2.75
mice		1.30 (0.05)	1.35 (0.05)	29 (1.9)	70 (2.0)	4.50
amelia		1.28 (0.03)	1.32 (0.04)	38 (2.3)	75 (2.5)	5.00
hmisc		1.41 (0.06)	1.42 (0.05)	32 (2.1)	75 (2.5)	5.50
Note. sd = standard deviation, misclassif. = percentage missclassified.

Box-plots that show the distribution of the NRMSE and misclassification rates obtained from all simulations for each scenario are shown in Figure 4. The two panels for each variable show precision by imputation method for samples with 350 and 1,050 cases, with each panel showing results for 20% and 60% missing, simulations with 700 units as well as with 40% missing are omitted for clarity. The precision of imputation was slightly better for all methods when considering larger samples or smaller missing percentages. Depressive symptoms NRMSE results presented an overall similar distribution shape to alcohol consumption NRMSE when considering imputation methods, sample sizes and missing percentage. Misclassification rates for occupation and pregnancy also have similar general behavior, although much higher misclassification percentages are observed for occupation. This happens because occupation variable has more classes; Kropko et al. [1] observed the same behavior when imputing missing data while methodically increasing number of classes in simulated categorical variables.

It can also be observed in Figure 4 that missForest is the method that imputes values closer to the real ones and with smaller variability between simulations for imputation in numeric variables. Mice.rf configuration of mice also presented variability smaller than other methods. Considering the MI methods, hmisc presented the largest NRMSE values but smaller misclassification rates than amelia for the pregnancy variable. Mice package provided slightly different results across the three configurations tested, depending on sample size and variables. Amelia seemed to be a competitive MI method when imputing numeric variables. Simulations comparing amelia to FCS method via the mi package in R were done by Kropko et al. [1], and the results obtained were comparable with ours - when considering numeric target variables FCS and JM performed similarly and FCS approach was better when imputing categorical variables.

To assess performance on predicting HIV status via the RF model before and after imputation with each scenario, the distribution for prediction accuracy is shown in Figure 5, with one panel for each sample size. Prediction accuracy increased sharply with increasing sample sizes: median overall accuracy values with the observed complete sample, in dark gray, were about 51.5%, 61.5% and 65.5% for samples sizes 350, 700 and 1,050 respectively. CCA, in light gray, represented by incomplete samples before imputation, presented lower median accuracy with large variability when missing 60%, while for 20% the difference between accuracy for CCA and observed sample is not very large. The missForest method seemed to slightly increase median prediction accuracy even when compared to the observed samples, which may be explained by noise present in the dataset that might be decreased with this type of imputation. All three mice configurations and hmisc performed about the same, just slightly inferior when compared to observed samples, and even though mice.rf presented a slight advantage in imputation precision, this advantage was not reflected when using imputed data for modeling. Amelia, despite its relatively poor performance in imputing categorical variables, achieved median prediction accuracy consistently higher than the other MI methods, comparable to missForest.

Next, to evaluate prediction considering each HIV class separately, Figure 6 shows sensitivity distributions, by method and missing percentage, with each panel showing results for a sample size. For new HIV positives with 350 cases, the median observed sensitivity was about 0.5, and it fell to about 0.4 with CCA with 20% and 60% missing. With imputation, missForest recovers the sensitivity with 20% missing, with smaller variability, but not at 60%. All MI methods performed about the same, with a slightly lower median sensitivity of about 0.45, for both missing percentages.

For larger samples, sensitivity falls sharply for new HIV positives, and was about 0.3 and 0.2 with 700 and 1,050 units, respectively. This is due to having only 140 new HIV positive cases in the dataset, which is about a third of 350, allowing models with this sample size to be balanced. When cases were replicated or used to generate synthetic ones via SMOTE in order to balance the dataset and then fit an RF model, the information in this class was limited when compared to the other two classes which had actual extra information.

For HIV negative, median sensitivities of about 0.52, 0.62 and 0.66 for samples with 350, 700 and 1,050 observations respectively were well recovered after imputation and all methods seem to perform about the same, with amelia and missForest having a small advantage. Also, the variability in sensitivity across simulations was smaller for negative than for positive HIV classes. For known HIV positive, the observed median sensitivity of about 0.7 was recovered after imputation in samples with 350 and 700 cases. For samples with 1,050 cases, observed median sensitivity of about 0.8 was closely recovered via all methods except when missing 60% and imputation with missForest. In cases where there were enough units in each HIV class and missing was limited to 20%, CCA performed similarly to after imputation, suggesting that imputation might not be necessary when missing percentage is not too large and imputation does not increase the number of available units in each outcome class. Figures 5 and 6 show that larger samples improved overall prediction accuracy and sensitivity for negatives and known positives while decreasing sensitivity for new positives. Therefore, if maximum sensitivity is desired for the new positives, a sample that is about three times the number of new HIV positive cases is probably the best option for modeling with this dataset. On larger samples simulations, results were similar for the three mice configurations, larger missing percentages slightly worsened imputation performance. As for amelia and hmisc, even though they presented lower imputation precision, as seen in Figure 4, their modeling accuracy and sensitivity values were comparable to the other methods.

For the second set of simulations, because the values are actually missing in the full linkage to HIV dataset, it was not possible to assess imputation precision via NRMSE and misclassification rate, but model prediction accuracy evaluation after imputation and comparison to CCA was performed in the same way as before. Recall that these simulations differ from previous ones in the percentage of missing data in the target variables, and that when doing imputation, the information from the full linkage to HIV care dataset is included. As a consequence, the only results that are comparable to previous ones are the ones for CCA with sample size 350, shown in dark gray in Figures 5, 6 and 7. Model prediction accuracy is shown in the top left panel of Figure 7. Median overall accuracy for samples with 350 was about 51.5% with CCA, and after imputation, it increased to about 67%, so even though the sample size is the same, the inclusion of new information in the HIV positive classes improved prediction accuracy. With larger samples, CCA provided about 64% median accuracy versus 74% and 73% after missForest and amelia and 72% after hmisc and mice imputations. Sensitivity for new HIV positive is presented on the top right panel. For CCA, sensitivity was worse for larger samples, likely because the new positives in the sample were mostly synthetic cases. Once real observations were included after imputation, median sensitivity improved to about 0.65 and 0.92 with mice imputation into small and large samples, respectively. The gain in sensitivity for the new HIV positive class after imputation with mice and hmisc seems to be a trade-off by a loss in the known HIV positive sensitivity, shown on the bottom right panel. Sensitivity results for HIV negative, in the bottom left panel, were improved after imputation, which is interesting because there is no new information being included in this class, this suggests that if the new HIV positive class is more easily classified with more information, then the negative class also is more easily classified. Sensitivity for the negative class presented smaller variability when compared to the other classes; the large amount of available information for this class produced more stable results across simulations.

Although several existing imputation methods and possible scenarios were not evaluated here, and different issues might arise when working with other datasets and analysis model other than RF, this study provides a few insights that may be useful to researchers who are exploring options other than CCA.

Single imputation method missForest provided more precise imputations than MI methods, but this did not translate to a consistent advantage when looking at prediction accuracy and sensitivity after RF modeling. This is reasonable because this method imputes one single time, aiming for precision but by doing so, it does not account for uncertainty of imputation, therefore model results may be precise but biased towards specific measures, in our case it provided better overall accuracy, and sensitivity was stable across classes, without favoring one class over others.

Considering multiple imputation methods, among mice different configurations considered, there were slight differences in precision of imputation, but almost no difference in model prediction accuracy and sensitivity. Amelia seems to be more sensitive to sample size and missing percentage, imputing better with larger samples and lower missing percentage in the first set of simulations, but presented the best overall prediction accuracy for both sets of simulation. Hmisc imputes less precise values than all methods, but performs similarly to mice when considering prediction accuracy and sensitivity. Interestingly, values for prediction accuracy and sensitivity after amelia imputation are similar to the ones obtained after missForest, with higher overall accuracy than the other methods and similar sensitivity across classes. Mice and hmisc provided lower overall accuracy and lower sensitivity for detecting known HIV positives, but a much higher sensitivity for new HIV positives. This result may be relevant to a researcher who is interested in studying specifics such as population features to focus on when trying to identify people that have not yet been diagnosed but have a higher chance of being HIV positive. This difference in the methods’ performance may be an advantage if the main interest of a study is in increasing sensitivity of one specific class of the outcome variable.

In general, all methods performed badly when imputing categorical variables, especially with occupation which has six classes, versus pregnancy with three classes. Therefore, if possible, one should avoid imputation into multi class categorical variables, or possibly attempt to decrease the number of classes in a variable if some of them show similar characteristics. Furthermore, if needed to prioritize, efforts to avoid or recover missing data in future or ongoing research should be concentrated in categorical features.

When considering the different simulation scenarios designed for this study, there was overall consistency in methods’ performance independent from sample size and missing percentage. Therefore, the selection of a single scenario with missing percentage and sample size adequate to the target study may provide enough information when considering different methods. These results are useful for the data analyst who is usually working with incomplete data, and does not necessarily have sufficient data and time to perform extensive simulations to compare different imputation methods. Hence, having the analysis model defined in advance, a couple of candidate imputation methods may be tested in the following way: perform imputation with each method, separate data into a testing and training set, fit model with training data and use testing data to evaluate the main measures of interest, similarly to how sensitivity and overall prediction accuracy were used in this study. Repeat this process a number of times to get the general behavior of these measures for each method. Then choose the method that presents better results for the measures of interest, without the need to evaluate imputation precision. To our study, the evaluation of imputation precision did not provide substantial additional information and if it was taken into account on its own, could cause misleading conclusions in regards to which method would give better results.

If more than one performance measure is of interest in a study, instead of using the mean rank presented in Table 3 to evaluate imputation accuracy for individual variables as it was done here, ranks may be assigned to each performance measures of interest by method. Equal or weighted rank may be used– in this study all variables received equal weights. The imputation method with the smallest mean rank may be selected.

Although simulation running time for the imputations was not discussed in this study, it was easy to notice that in general, algorithms that used tree methods took significantly more time to run than others - amelia and hmisc imputed in as little as a few seconds to a minute, while mice with random forest and with cart and missForest took a minute to several minutes for imputation, depending on sample size and missing percentage. Mice with default settings ran much faster than methods with trees but significantly slower than amelia and hmisc. In general, larger samples resulted in slower imputation. Additionally, Yadav & Roychoudhury [21] found that imputation time for missForest decreased when missing percentage increased and the opposite was true for mice and hmisc. Therefore, if computational resources are limited, data is too large or missing percentages are high, imputation time is another aspect to consider when choosing an imputation method.

Specific to the linkage to HIV care study dataset, modeling results are remarkably improved after imputation increases the number of observations available in the minority categories of positive HIV status. While overall prediction accuracy and sensitivity were selected here to compare results before and after imputation, any other measure of interest can be evaluated following the same steps – impute, select train and test samples, fit a model with train sample, predict test sample and evaluate performance measure. Sensitivity for predicting the known HIV positive class deteriorated after imputation so an additional option to be considered in future work is to only model with observations that were complete in this class, as it was done with the negative class. Therefore, when considering analysis for imbalanced datasets, imputation for missing data is especially advantageous when it increases the number of observations in the minority class(es), as it was observed for the HIV dataset.

For future work, one or two simulations may be selected to have its RF model further analyzed and one or more imputation methods may be considered depending on the goals of the analysis. Interpretation of the RF model via variable importance will support the understanding of social demographic aspects of the HIV epidemic in Uganda and which features may help in the identification of new HIV positive cases. Social interventions in an attempt to decrease the rate of new HIV positive cases overtime could be based on specific relationships observed between HIV positive rates and features, such as discouraging alcohol consumption, which was observed to be higher amongst new positive cases. Interventions in addition to HIV care could also be aimed at the known HIV positive population to improve their quality of life, such as addressing depression symptoms via public services that could offer counseling or other support therapies.

The findings of this study will be used for the selection of an adequate imputation method in future research to support the analysis of the data post interventions from the PATH/Ekkubo study [2]. The analysis of the follow-up data will help quantify how successful the PATH/Ekkubo study was in increasing the percentage of diagnosed HIV positive people that were linked to HIV care, followed treatment and achieved viral suppression, therefore decreasing the odds of spreading the virus further.

CCA – Complete Case Analysis; HIV – Human Immunodeficiency virus; PATH – Providing access to HIV care; CESD – Center for the Epidemiological Studies of Depression Short Form; AUDIT-C – Alcohol use Disorders Identification Test – Concise; MICE – Multiple Imptation by Chained Equations; HMISC – Harrell Miscellaneous; RF – Random Forest; SI – Single Imputation; MI – Multiple Imputation; ML – Maximum Likelihood; EM – Expectation Maximization; JM – Joint Multivariate; FCS – Fully conditional specification; MAR – Missing at random; NMAR – Missing not at random; ACE – alternating conditional expectations; AVAS - additive and variance stabilization for regression; NRMSE - normalized root square mean squared errors; MSE – Minimum squared errors; ROC - Receiving Operating Characteristic; SMOTE - Synthetic Minority Over-sampling Technique.

Ethical approval and consent to participate

All study and experiments procedures were performed in accordance with relevant guidelines and regulations, approval of all study and experiments procedures was obtained from the institutional review boards at San Diego State University (#2095100) in the US and Makerere University School of Public Health in Uganda (#296) and the Uganda National Council for Science and Technology. Informed consent was obtained from all participants before collection of all data and intervention participation.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analyzed during the current study are not going to be made public but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding:

This study was supported by the National Institute of Mental Health: Grant R01MH106391 awarded to Susan M. Kiene and Rhoda K. Wanyenze. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Authors' contributions

NBM ran simulations, performed statistical analysis and figures. NBM, CDL and BAB conceived design of simulations, methodology and presentation and interpretations of results. NBM, BAB, CDL and SMK led the draft of the manuscript. In regards to the linkage to HIV care study, which provided us with the HIV dataset in this manuscript, SMK and RKW conceived and designed the design and oversaw project implementation. SCK provided input on the study design and intervention protocol. NAM provided input on the overall study design, and developed the cost-effectiveness analysis section. RN, KAS, and ME contributed to project implementation. All authors approved the final manuscript.

Acknowledgements

Not applicable.

Kropko, J., Goodrich, B., Gelman, A., & Hill, J. (2017). Multiple Imputation for Continuous and Categorical Data: Comparing Joint Multivariate Normal and Conditional Approaches. Political Analysis, 22(4), 497-519. doi:10.1093/pan/mpu007
Kiene, S. M., Kalichman, S. C., Sileo, K. M., Menzies, N. A., Naigino, R., Lin, C.D., Bateganya, M. H., Lule, H & Wanyenze, R. K. (2017). Efficacy of an enhanced linkage to HIV care intervention at improving linkage to HIV care and achieving viral suppression following home-based HIV testing in rural Uganda: study protocol for the Ekkubo/PATH cluster randomized controlled trial. BMC Infectious Diseases, 17, 460.
Honaker, J., King, G & Blackwell, M. (2011). Amelia II: A Program for Missing Data. Journal of Statistical Software, Vol. 45, No 7.
Stekhoven, D. J. & Bühlmann, P. (2012). MissForest - non-parametric missing value imputation for mixed-type data. Bioinformatics, Vol. 28, Issue 1, pp. 112118.
Buuren, S. V., K. Groothuis-Oudshoorn (2011). mice: Multivariate imputation by chained equations in R, Journal of Statistical Software, Vol. 45, Issue 3.
Alzola , C. & Harrell, F., (2002). An introduction to S and the Hmisc and Design Libraries.
Andresen, E. M., Malmgren, J. A., Carter, W. B. & Patrick, D. L. (1994). Screening for depression in well older adults: evaluation of a short form of the CES-D (Center for Epidemiologic Studies Depression Scale). Am J Prev Med. 1994; 10(2): 77-84.
Bush, K., Kivlahan, D. R., McDonell, M. B., Fihn, S. D. & Bradley K. A. (1998). The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Arch Intern Med. 158:1789-95.
Breiman, L. (2001. Random Forests. Machine learning, 45(1), 5-32.
Clavel, J., Merceron, G. & Escarguel, G. (2014). Missing data estimation in morphometrics: how much is too much? Systematic Biology, Vol. 63, Issue 2, pp. 203218.
Nguyen, C. D., Carlin, J. B. & Lee, K. J. (2017). Model checking in multiple imputation: an overview and case study. Emerging Themes in Epidemiology 14, No. 8.
Hughes, R. A., Heron, J., Sterne, J. A. C. & Tilling, K. (2019). Accounting for missing data in statistical analysis: multiple imputations is not always the answer. International Journal of Epidemiology, pp. 1294-1304.
Plaia, A., Bondì, A.L., (2006). Single imputation method of missing values in environmental pollution data sets, Atmospheric Environment, Volume 40, Issue 38.
Buuren, S. V. (2018). Flexible imputation of missing data. Chapman & Hall, 2^ndedition.
Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley, New York.
Honaker, J. & King, G (2010). What to do about missing values in time-series cross-section data. American Journal of Political Science, Vol. 54, No. 2, pp. 561-581.
King, G, Honaker, J., Joseph, A. & Scheve, K. (2001). Analyzing incomplete political science data: an alternative algorithm for multiple imputation. American Political Science Review, Vol. 95, No 1, pp. 46-69.
Breiman, L. & Friedman, J. H. (1985). Estimating Optimal Transformations for Multiple Regression and Correlation, Journal of the American Statistical Association, pp. 580-598.
Tibshirani, R. (1988). Estimating Transformations for Regression Via Additivity and Variance Stabilization, Journal of the American Statistical Association Vol. 83, No. 402, pp. 394-405
Banks, D. (2009). Duke University Lectures: Nonparametric Regression.
Yadav, M. L. & Roychoudhury, B. (2018). Handling missing values: A study of popular imputation packages in R, Knowledge-Based Systems, Volume 160.
Penone, C., Davidson, A. D., Shoemaker, K. T, (2014). Imputation of missing data in life‐history trait datasets: which approach performs the best? Methods in ecology and evolution, Vol 5, issue 9.
Waljee AK, Mukherjee A, Singal AG, et al, (2013). Comparison of imputation methods for missing laboratory data in medicine, BMJ Open 2013;3:doi: 10.1136/bmjopen-2013-002847.
Shah, A. D., Bartlett, J. W., Carpenter, J., Nicholas, O. & Hemingway, H. (2014). Comparison of random forest and parametric imputation models for imputing missing data using mice: a caliber study. American Journal of Epidemiology, Vol. 179, No. 6, pp. 764-774.
Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1-26. doi:http://dx.doi.org/10.18637/jss.v028.i05.
Chawla, NV., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique, Journal of Artificial Intelligence Research, Vol. 16, pp. 321-405.

No competing interests reported.

Download PDF

Version 2

posted

You are reading this latest preprint version

Evaluations of imputation methods for missing data with random forest modeling: an application with unbalanced data with a three categories outcome, the linkage to HIV care Uganda study

Status:

Version 2

Abstract

Figures

Background

METHODS

The Linkage to HIV Care Study Dataset

Missing Data Imputation

Imputation Algorithms and Models

Simulations

Results

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 2