Synthetic Control Analysis Method
This paper uses the synthetic control analysis (SCA) method to retrospectively quantify the impact of USAID funding on under-five mortality across several low- and middle-income countries. While other approaches also quantify impact (e.g., difference-in-difference, propensity scores), the SCA method does not require some of the critical assumptions of these other approaches.[15] The SCA method uses a non-parametric, data-driven procedure to create a control group (a synthetic control) that is similar to a treatment group in a pre-intervention period.[14] Outcomes for the control group are then compared to the outcomes in the treatment group during the intervention period to quantify the impact of the treatment. In this case, the units comprising the control and treatment groups are countries, the intervention is USAID investment in maternal and child health during the first 15-20 years of the IMCI initiative, and the outcome is the U5MR, as described in more detail below. The SCA method assesses the outcomes (and predictive factors) of a group of non-treatment countries and identifies a subset of countries that are similar to the treatment countries. The SCA process assembles the counterfactual by weighting the outcomes of this subset of countries to produce a synthetic control: the outcome in the treatment countries in the absence of treatment.[15]
In sum, the SCA presented here provides a data-driven strategy to produce a counterfactual to enable the quantification of impact of USAID child health programs under the conditions described. In this section, we describe in more detail the SCA method including assumptions, construction of the counterfactual, quantification of the impact and sensitivity analyses.
Treatment Unit
A synthetic control analysis (SCA) requires a treatment unit, a donor pool of similar units, a treatment period, an outcome variable, and a set of predictor variables to construct the synthetic control. We purposively selected countries with the highest amount of continuous USAID investments in child health during the IMCI period to make up the treatment unit. This is an example of “testing at the margins.” If we do not see quantifiable treatment effects in these high investment countries, then we would not expect to see quantifiable effects in countries where USAID invested at lower levels. The authors did not expect the investment of one US dollar to have an impact, and we did not have any knowledge of a threshold beyond which investments would trigger a net positive impact; therefore, quantifying the impact in countries with relatively high levels of investment made most sense as the starting point for selecting the treatment units under this novel analysis approach for quantifying the impact of donor funding.
The selection criteria for high investment countries that comprise the treatment unit entailed a two-step process. First, we selected countries that received continuous maternal and child health (MCH) funding during the period 1999-2016, based on the USAID’s annual Reports to Congress on the Child Survival and Health Programs Fund for 1999-2004 and later from the US State Department’s Foreign Assistance Coordination and Tracking System financial reporting system from 2007-2016 (not available to the public). In total, 25 countries satisfied this condition. In the 1999-2004 period, funding of malaria programs was included in the Child Survival and Health Program funding and not available separately. From 2007-2016, MCH and malaria funds were separated and therefore, these two funds were combined to provide a consistent tracking of continuous funds for the 1999-2016 period.
For the second step of the process, we examined the distribution of these countries along two parameters: MCH plus malaria funding in total and per capita amounts. In Figure 2, the axes represent the median amounts for total (x-axis) and per capita (y-axis) funding. The median amount received by these countries is $32.5M and per capita amount is $1.19. From this, we identified eight high-investment countries that fall in the first quadrant of the chart, i.e. the countries that received above the median amounts on both axes: Senegal, Zambia, Mali, Malawi, Madagascar, Ghana, Mozambique, and Uganda. These eight countries make up the Treatment Unit.
Rather than use an individual country as the treatment unit, the treatment unit in this analysis was comprised of multiple treated units (the eight countries listed above), as done previously.[16, 17] We followed the alternative approach used by Lepine et al. to construct a single treated unit from the average of the outcomes of the eight high-investment USAID countries and then calculate the treatment effect compared to the synthetic control. This was done instead of pooling the individual treatment effect of each country, since the approach we used was found by to lead to similar estimates, but with a more precise counterfactual that is less influenced by outliers.[16] The treatment unit each year was constructed from the average of the outcome variable (U5MR) and the covariates, with the average of each variable weighted by the number of live births.[25]
Donor Pool
All countries classified by the World Bank as low- or low-middle income countries in 2016 were considered eligible to be donors. However, because SCA requires that donors not receive exposure to the treatment, countries were excluded from being in the donor pool if they received USAID financial assistance earmarked for either maternal and child health or malaria for more than half (>9) of the sixteen years from 1999-2016. Although we originally hoped to exclude countries that received any USAID funding in the treatment period, setting this strict criterion would have eliminated almost all countries from the donor pool. Our criteria allowed 48 countries to remain in the donor pool (Supplementary Table S1).
Another consideration with the donor pool of countries is that, as low and lower-middle income countries, they are likely to have received financial and technical support from other donors in the treatment period that contributed to reductions in child mortality. This leads back again to one rationale for this analysis: the need to find alternative ways to assess the impact of donor programs when there is no pure control. The potential contamination of this impact analysis from some donor pool countries that received either some USAID funds and/or other kinds of technical support for child mortality reduction is a real possibility. To account for potential differences between the donor pool and the treatment group of countries, development assistance from donors, political stability, and other factors were controlled for in this analysis (see section on Predictor Variables below). Population size was also accounted for in the calculation of the outcome measure.
Time Period
We considered the treatment to have started in 1999 since many of the initial progress reports of IMCI implementation were published that year. Few, if any, countries would have started implementation sooner at any scale. Some countries in this analysis may have started IMCI later than 1999, but that is consistent with the hypothesis that USAID engagement leads to earlier and more intense implementation of new policies. The analysis scans a relatively long intervention period (1999-2016). The trends in outcome between treatment and control, during the intervention period, will reflect smaller periods of increasing and decreasing intensity in support of IMCI within the period. Choosing 1980 as the pre-intervention start year permits a long pre-intervention period (1980-1998), which helps SCA optimize its control. The year 1980 was also a good start year for the pre-intervention since data on many covariates only began to become widely available in the 1980s, due in part to the advent of the Demographic and Health Surveys. Because this analysis was initially conceptualized in June of 2017 and the dataset was compiled at that time, 2016 was chosen as the end-date of the analysis.
Dependent Variable
The dependent variable is the under-five mortality rate (U5MR), since it provides an overall measure of child health and was the variable used for MDG4.[26]
The median value of U5MR is estimated annually for all countries by the UN Interagency Group for Mortality Estimation (UN IGME), which provides a consistent approach across countries, was used as the data source for the dependent variable in this analysis.[26]
In 1999, the unweighted mean U5MR of the eight countries in the treatment group was 161.7, while the unweighted mean U5MR of the 48 countries in the donor pool was 80.1.[26] This difference is expected that USAID would invest child health resources in high-mortality countries. However, the purpose of the synthetic control method is to equalize the dependent and predictor variables in the pre-intervention period (see more below).
Predictor Variables
SCA requires that the treated unit and the synthetic control be similar during the pre-intervention period when comparing across measures that may predict the outcome variable. However, child mortality reduction is multifactorial. The Success Factors Study for Women’s and Children’s Health examined over 250 indicators for data availability and potential to associate with declines in child mortality.[27, 28] It divided these many variables into 11 different policy areas (Table 1). It found that these policy areas contributed additively to child mortality reduction, and that approximately half of the gains in child mortality came from improvements in coverage in the health sector (e.g. immunizations, fertility reduction), and the other half came from gains in coverage outside the health sector (water and sanitation, per capita GDP growth). Note that within these policy areas, variables were often highly correlated (e.g. between antenatal care and skilled birth attendance). For that reason, to avoid known multi-collinearity in the initial model, our initial synthetic control model included one variable from each policy area identified by the Success Factors Study, which was used as the starting point for further optimization as described below. During optimization, we did not require that the final model keep exactly one variable from each policy area, since some variables from into the same policy area were only weakly correlated with each other, while others from different policy areas may have been either highly correlated to each other or of low predictive value for U5MR.
Assumptions
SCA makes several assumptions. For accurate estimation of effects, only one unit (or one group of units) under study are treated to the intervention. The donor units cannot be exposed to the same/similar intervention, defined here as above median absolute and per capita MCH and malaria funding from USAID throughout the 2000-16 period. Additionally, the values of the predictor variables must be comparable for both the treated and the synthetic control.
Model Optimization
SCA provides an unbiased method for choosing an appropriate counterfactual for non-random treatment assignment. We iteratively added or replaced different candidate predictor variables from the model and selected the model which had the lowest root-mean-squared-prediction-error (RMSPE). A previous analysis that used U5MR as an outcome variable found that models with an RMSPE < 3 show a good fit between the treated unit and the synthetic control.[17] We constrained this optimization by insisting that predictors likely to confound our results (namely, polity score and total foreign aid received per capita) be included in the final model. We followed a previous SCA analysis by including three lags of the dependent variable as predictor variables.[14]
Procedure
We conducted all analyses in Stata version 14 SE using the synth command and the following code:
synth [dependent variable] [control variables], trunit() trperiod(1999) xperiod(1980(1)1998) counit([donor countries]) nested fig allopt keep(filename.dta, replace)
Statistical Analysis and Inference
The synth procedure calculates a difference in the outcome variable between the treatment and control group in the post-intervention period, but on its own the significance of this gap is unclear. The synth_runner procedure in STATA permits the direct calculation of the statistical significance of the measured gap in outcomes after the intervention. Synth_runner performs the synth procedure for the treatment unit and for each unit of the donor pool (placebos), calculating the size of the gap for each placebo each year.
Because we are testing the hypothesis that U5MR declined faster in USAID-supported countries than in similar donor countries, we use one-sided p-values to test statistical significance. The one-sided p-value is the number of placebos whose measured treatment gap in a given year was larger in the same direction as for the treatment unit, divided by the total number of placebos. Since the placebo effect may be quite large if the units were not matched well in the pre-treatment period, the measured gap for each placebo in the post-intervention period is standardized by dividing it by the pre-treatment gap size.[29]
Sensitivity Analysis
We carried out several sensitivity analyses, including the exclusion of donor pool countries with relatively large weight in the creation of the synthetic control. This was done to check for the undue influence of one country on our results. Other sensitivity analyses are described in Supplementary Section S2.
Individual Country Analysis
In addition to the weighted average of the treatment unit, we also ran SCA for each of the eight countries in Quadrant 1 of Figure 2 individually; these are provided in Supplementary Section S2.