Estimating the impact of donor programs on child mortality in low- and middle-income countries: a synthetic control analysis of child health programs funded by the United States Agency for International Development

doi:10.21203/rs.3.rs-32400/v1

PDF

Research article

Estimating the impact of donor programs on child mortality in low- and middle-income countries: a synthetic control analysis of child health programs funded by the United States Agency for International Development

https://doi.org/10.21203/rs.3.rs-32400/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted 09 Jun, 2020

You are reading this latest preprint version

Background Significant levels of funding have been provided to low- and middle-income countries for development assistance for health, with most funds coming through direct bilateral investment led by the United States and the United Kingdom. Direct attribution of impact to large scale programs funded by donors remains elusive due the difficulty of knowing what would have happened without those programs, and the lack of detailed contextual information to support causal interpretation of changes.

Methods This study uses the synthetic control analysis method to estimate the impact of one donor’s funding (United States Agency for International Development, USAID) on under-five mortality across several low- and middle-income countries that received above average levels of USAID funding for maternal and child health programs between 2000 and 2016.

Results In the study period (2000-16), countries with above average USAID funding had an under-fiver mortality rate lower than the synthetic control by an average of 29 deaths per 1,000 live births (year to year range of 2 – 38). This finding was stable across several sensitivity analyses.

Conclusions The synthetic control method is a valuable addition to the range of approaches for quantifying the impact of large-scale health programs in low- and middle-income countries. The findings suggest that adequately funded donor programs (in this case USAID) help countries to reduce child mortality to significantly lower rates than would have occurred without those investments.

Pediatrics

Health Policy

Synthetic control analysis

Integrated Management of Childhood Illness (IMCI)

Low- and middle-income countries (LMICs)

United States Agency for International Development (USAID)

Child mortality

Quasi-experimental methods

Maternal and child health (MCH)

Impact evaluation

Donor assistance

Substantial resources have been provided to low- and middle-income countries (LMICs) to support the development of the health sector. Close to $600 billion of development assistance for health (DAH) was provided between 1995 and 2018.[1] In 2018 alone, the DAH was estimated at $38.9 billion with the majority of funds coming through direct bilateral assistance, with the USA leading at 34% of total DAH followed by the UK at 8.4%. Of the total 2018 DAH, 24% ($9.5 billion) went to HIV/AIDS, 20.1% ($7.8 billion) went to newborn and child health, 14.3% ($5.6 billion) went to health systems strengthening, and 12.1% ($4.7 billion) went to reproductive and maternal health.[1] Multilateral development agencies and other private-public partnerships, such as the Global Fund, The World Health Organization (WHO), World Bank, the Bill and Melinda Gates Foundation, and the United Nations Children's Fund (Unicef) jointly disbursed approximately 31% of total DAH in 2018.[2] Single-year estimates of DAH specifically for reproductive, maternal and child health vary by source with estimates ranging from $10.8 to $13.1 billion, with the US consistently as the single largest contributor across sources.[3]

Given this large investment, donors of health programs want evidence that the patient and population health have improved in the communities served by these donors’ programs. This is desired in order to be transparent and to be accountable for the funding that donor agencies receive as well as to justify continued funding.[4] For example, the Bill and Melinda Gates Foundation (BMGF) has a website for press releases that allow the Foundation to publicize its maternal and child health programs.[5] The United Kingdom’s Department for International Development (DFID) has a Development Tracker where one can find reports on the results of the health programs it funds.[6] The United States Agency for International Development (USAID) has a document library called the Development Experience Clearinghouse with annual reports to Congress that have information about programs aiming to prevent child deaths and their results.[7]

In addition, to make an even stronger case to constituencies that a donor’s program is valuable, it is desirable to have evidence that can directly attribute the cause of positive changes in patient or population health to that specific donor’s program. Evidence of such attribution is rare, however, as it requires a difficult-to-achieve evaluation design that justifies causal inference about the effects of the program.[8] Evidence of attribution is particularly difficult in low- and middle- income countries with a history of donor support.[9] As Victora et al. articulate very well, traditional evaluation designs that compare a donor-funded intervention in a program area with no intervention in a control area may not be feasible. Health programs, including child health programs relevant to this study, have been and are being scaled up nationwide, and the existence of potential comparison areas or populations---that have not experienced donor-supported child health interventions---are rare. Thus, there is often no option for a strong counterfactual necessary for causal inference. And, it is hard to tease out benefits of a program from one donor when there are many donor programs and most/all work in partnership with host government and communities, without detailed information about these programs and contextual factors allowing for plausible interpretations of impact.[10, 11]

Victora et al. argue for the development of a national evaluation platform in each country with the district as the main unit of analysis that “includes documentation of contextual factors and implementation of many programmes—and indicators of coverage, impact, and cost” (pg 94).[9] In each country, data are compiled from existing population-based sources and from routine collected data at health facilities and administrative sources data sources that match a conceptual model around maternal and child health interventions leading to reduced mortality.[12] The quality of data are assessed and improved. Data are then organized to facilitate cross-district analysis by time and equity. This approach may be considered an optimal approach given the difficulties with evaluation in LMICs using traditional methods. However, this approach might be considered a long-term solution as the platform would need to integrate population information (vital registration, household surveys) facility-based and administrative statistics at the district-level on a routine basis; in addition, functionality would also require expertise and other resources to analyze the information and redirect programs.[13]

This paper provides an alternative, intermediate method to directly estimate the impact of donor funding on health outcomes in the absence of detailed contextual information. Using the case of a single donor’s effort (USAID), we use a synthetic cohort analysis methodology to study the impact of USAID funding on under-five mortality across several low- and middle-income countries. The analysis presented here provides a data-driven strategy to produce a counterfactual (i.e., a circumstance with no intervention), to quantify impact. The synthetic control method, has been used to evaluate programs in high, middle and low-income countries.[14-17] Abadie, Diamond, and Hainmueller describe the utility of the synthetic control method (Supplementary Section S1).[14]

In this study, we apply a novel method (synthetic control) to estimate the impact of donor investments in child health. Specifically, we attempt to quantify the impact of USAID investment and support on child mortality. We test the method by observing a scenario where we believe the impact of USAID investments can be quantified using a synthetic control approach. The scenario takes place in countries with relatively high levels of USAID support to the World Health Organization’s initiative to accelerate reductions in child mortality, called the Integrated Management of Childhood Illness (IMCI) and Integrated Community Case Management (iCCM) (1999-2016).

Estimating the impact of United States Agency for International Development child health programs

In 1995, the World Health Organization convened global partners to develop a new approach to child healthcare in developing countries, called Integrated Management of Childhood Illness (IMCI). An integrated and holistic approach to child health, IMCI aims to reduce child mortality and morbidity while promoting health and well-being of children under five years of age.[18] Shortly after development, different countries moved with varying speed to implement the recommendations of IMCI.[19] By mid-1999, twelve countries had reached the expansion phase, while twenty-nine LMICs had not yet introduced IMCI.[20] In September 2000, 189 nations met at the United Nations to sign the Millennium Declaration, committing them to try to achieve the Millennium Development Goals (MDG). MDG4 aimed to reduce child mortality by two-thirds from 1990 levels by 2015.

USAID was well-positioned to contribute to these initiatives. To better describe here USAID’s approach to child mortality reduction, we reviewed USAID’s annual Reports to Congress on the Child Survival and Health Programs Fund from 2001-2004 as these were the early years of IMCI and the MDGs.[21-24] We reviewed the activities that USAID invested in during this period (e.g., social and behavioral change), the specific problems being addressed (e.g., treatment of diarrhea), and USAID’s approaches (e.g., quality improvement). One common attribute across USAID child health programs (including malaria) was that activities were directed at all levels of the health system, from household level health promotion to national policy. Another common attribute was a focus on introducing and scaling up high-impact interventions for the major causes of child mortality and morbidity, including developing the evidence to support scale and quality. Most programs addressed the treatment of the sick child, including diarrhea and pneumonia, and malaria where relevant. The final common attribute was USAID support for three broad groupings of activities: developing and leveraging partnerships with the government, civil society and donors; health system strengthening; and targeting cost-effective high-impact interventions to those in most need. These common attributes across countries and other review findings are used to describe the theory of change for this analysis (Figure 1).

[Insert Figure 1]

We hypothesize that US investment, engagement, and partnership with countries, during the IMCI period, would accelerate reductions in child mortality in those countries, as visualized through the theory of change (Figure 1): (1) increased capacity building and health promotion around IMCI (improving both supply and demand); (2) encouraging partners and other stakeholders to increase engagement with IMCI including more rapid policy change; and, (3) facilitating development of IMCI evidence and/or more rapid introduction of evidence-based practices. These efforts would lead to earlier and more successful scale-up of IMCI interventions, and faster under-five mortality reduction, than without these investments. More specifically, we hypothesize that under-five mortality rates (U5MR) in countries with consistent and strong USAID investment would, over the last two decades, be quantifiably lower than they would have been if USAID had not invested and engaged.

Synthetic Control Analysis Method

This paper uses the synthetic control analysis (SCA) method to retrospectively quantify the impact of USAID funding on under-five mortality across several low- and middle-income countries. While other approaches also quantify impact (e.g., difference-in-difference, propensity scores), the SCA method does not require some of the critical assumptions of these other approaches.[15] The SCA method uses a non-parametric, data-driven procedure to create a control group (a synthetic control) that is similar to a treatment group in a pre-intervention period.[14] Outcomes for the control group are then compared to the outcomes in the treatment group during the intervention period to quantify the impact of the treatment. In this case, the units comprising the control and treatment groups are countries, the intervention is USAID investment in maternal and child health during the first 15-20 years of the IMCI initiative, and the outcome is the U5MR, as described in more detail below. The SCA method assesses the outcomes (and predictive factors) of a group of non-treatment countries and identifies a subset of countries that are similar to the treatment countries. The SCA process assembles the counterfactual by weighting the outcomes of this subset of countries to produce a synthetic control: the outcome in the treatment countries in the absence of treatment.[15]

In sum, the SCA presented here provides a data-driven strategy to produce a counterfactual to enable the quantification of impact of USAID child health programs under the conditions described. In this section, we describe in more detail the SCA method including assumptions, construction of the counterfactual, quantification of the impact and sensitivity analyses.

Treatment Unit

A synthetic control analysis (SCA) requires a treatment unit, a donor pool of similar units, a treatment period, an outcome variable, and a set of predictor variables to construct the synthetic control. We purposively selected countries with the highest amount of continuous USAID investments in child health during the IMCI period to make up the treatment unit. This is an example of “testing at the margins.” If we do not see quantifiable treatment effects in these high investment countries, then we would not expect to see quantifiable effects in countries where USAID invested at lower levels. The authors did not expect the investment of one US dollar to have an impact, and we did not have any knowledge of a threshold beyond which investments would trigger a net positive impact; therefore, quantifying the impact in countries with relatively high levels of investment made most sense as the starting point for selecting the treatment units under this novel analysis approach for quantifying the impact of donor funding.

The selection criteria for high investment countries that comprise the treatment unit entailed a two-step process. First, we selected countries that received continuous maternal and child health (MCH) funding during the period 1999-2016, based on the USAID’s annual Reports to Congress on the Child Survival and Health Programs Fund for 1999-2004 and later from the US State Department’s Foreign Assistance Coordination and Tracking System financial reporting system from 2007-2016 (not available to the public). In total, 25 countries satisfied this condition. In the 1999-2004 period, funding of malaria programs was included in the Child Survival and Health Program funding and not available separately. From 2007-2016, MCH and malaria funds were separated and therefore, these two funds were combined to provide a consistent tracking of continuous funds for the 1999-2016 period.

For the second step of the process, we examined the distribution of these countries along two parameters: MCH plus malaria funding in total and per capita amounts. In Figure 2, the axes represent the median amounts for total (x-axis) and per capita (y-axis) funding. The median amount received by these countries is $32.5M and per capita amount is $1.19. From this, we identified eight high-investment countries that fall in the first quadrant of the chart, i.e. the countries that received above the median amounts on both axes: Senegal, Zambia, Mali, Malawi, Madagascar, Ghana, Mozambique, and Uganda. These eight countries make up the Treatment Unit.

Rather than use an individual country as the treatment unit, the treatment unit in this analysis was comprised of multiple treated units (the eight countries listed above), as done previously.[16, 17] We followed the alternative approach used by Lepine et al. to construct a single treated unit from the average of the outcomes of the eight high-investment USAID countries and then calculate the treatment effect compared to the synthetic control. This was done instead of pooling the individual treatment effect of each country, since the approach we used was found by to lead to similar estimates, but with a more precise counterfactual that is less influenced by outliers.[16] The treatment unit each year was constructed from the average of the outcome variable (U5MR) and the covariates, with the average of each variable weighted by the number of live births.[25]

[Insert Figure 2]

Donor Pool

All countries classified by the World Bank as low- or low-middle income countries in 2016 were considered eligible to be donors. However, because SCA requires that donors not receive exposure to the treatment, countries were excluded from being in the donor pool if they received USAID financial assistance earmarked for either maternal and child health or malaria for more than half (>9) of the sixteen years from 1999-2016. Although we originally hoped to exclude countries that received any USAID funding in the treatment period, setting this strict criterion would have eliminated almost all countries from the donor pool. Our criteria allowed 48 countries to remain in the donor pool (Supplementary Table S1).

Another consideration with the donor pool of countries is that, as low and lower-middle income countries, they are likely to have received financial and technical support from other donors in the treatment period that contributed to reductions in child mortality. This leads back again to one rationale for this analysis: the need to find alternative ways to assess the impact of donor programs when there is no pure control. The potential contamination of this impact analysis from some donor pool countries that received either some USAID funds and/or other kinds of technical support for child mortality reduction is a real possibility. To account for potential differences between the donor pool and the treatment group of countries, development assistance from donors, political stability, and other factors were controlled for in this analysis (see section on Predictor Variables below). Population size was also accounted for in the calculation of the outcome measure.

Time Period

We considered the treatment to have started in 1999 since many of the initial progress reports of IMCI implementation were published that year. Few, if any, countries would have started implementation sooner at any scale. Some countries in this analysis may have started IMCI later than 1999, but that is consistent with the hypothesis that USAID engagement leads to earlier and more intense implementation of new policies. The analysis scans a relatively long intervention period (1999-2016). The trends in outcome between treatment and control, during the intervention period, will reflect smaller periods of increasing and decreasing intensity in support of IMCI within the period. Choosing 1980 as the pre-intervention start year permits a long pre-intervention period (1980-1998), which helps SCA optimize its control. The year 1980 was also a good start year for the pre-intervention since data on many covariates only began to become widely available in the 1980s, due in part to the advent of the Demographic and Health Surveys. Because this analysis was initially conceptualized in June of 2017 and the dataset was compiled at that time, 2016 was chosen as the end-date of the analysis.

Dependent Variable

The dependent variable is the under-five mortality rate (U5MR), since it provides an overall measure of child health and was the variable used for MDG4.[26]

The median value of U5MR is estimated annually for all countries by the UN Interagency Group for Mortality Estimation (UN IGME), which provides a consistent approach across countries, was used as the data source for the dependent variable in this analysis.[26]

In 1999, the unweighted mean U5MR of the eight countries in the treatment group was 161.7, while the unweighted mean U5MR of the 48 countries in the donor pool was 80.1.[26] This difference is expected that USAID would invest child health resources in high-mortality countries. However, the purpose of the synthetic control method is to equalize the dependent and predictor variables in the pre-intervention period (see more below).

Predictor Variables

SCA requires that the treated unit and the synthetic control be similar during the pre-intervention period when comparing across measures that may predict the outcome variable. However, child mortality reduction is multifactorial. The Success Factors Study for Women’s and Children’s Health examined over 250 indicators for data availability and potential to associate with declines in child mortality.[27, 28] It divided these many variables into 11 different policy areas (Table 1). It found that these policy areas contributed additively to child mortality reduction, and that approximately half of the gains in child mortality came from improvements in coverage in the health sector (e.g. immunizations, fertility reduction), and the other half came from gains in coverage outside the health sector (water and sanitation, per capita GDP growth). Note that within these policy areas, variables were often highly correlated (e.g. between antenatal care and skilled birth attendance). For that reason, to avoid known multi-collinearity in the initial model, our initial synthetic control model included one variable from each policy area identified by the Success Factors Study, which was used as the starting point for further optimization as described below. During optimization, we did not require that the final model keep exactly one variable from each policy area, since some variables from into the same policy area were only weakly correlated with each other, while others from different policy areas may have been either highly correlated to each other or of low predictive value for U5MR.

[Insert Table 1 here]

Assumptions

SCA makes several assumptions. For accurate estimation of effects, only one unit (or one group of units) under study are treated to the intervention. The donor units cannot be exposed to the same/similar intervention, defined here as above median absolute and per capita MCH and malaria funding from USAID throughout the 2000-16 period. Additionally, the values of the predictor variables must be comparable for both the treated and the synthetic control.

Model Optimization

SCA provides an unbiased method for choosing an appropriate counterfactual for non-random treatment assignment. We iteratively added or replaced different candidate predictor variables from the model and selected the model which had the lowest root-mean-squared-prediction-error (RMSPE). A previous analysis that used U5MR as an outcome variable found that models with an RMSPE < 3 show a good fit between the treated unit and the synthetic control.[17] We constrained this optimization by insisting that predictors likely to confound our results (namely, polity score and total foreign aid received per capita) be included in the final model. We followed a previous SCA analysis by including three lags of the dependent variable as predictor variables.[14]

Procedure

We conducted all analyses in Stata version 14 SE using the synth command and the following code:

synth [dependent variable] [control variables], trunit() trperiod(1999) xperiod(1980(1)1998) counit([donor countries]) nested fig allopt keep(filename.dta, replace)

Statistical Analysis and Inference

The synth procedure calculates a difference in the outcome variable between the treatment and control group in the post-intervention period, but on its own the significance of this gap is unclear. The synth_runner procedure in STATA permits the direct calculation of the statistical significance of the measured gap in outcomes after the intervention. Synth_runner performs the synth procedure for the treatment unit and for each unit of the donor pool (placebos), calculating the size of the gap for each placebo each year.

Because we are testing the hypothesis that U5MR declined faster in USAID-supported countries than in similar donor countries, we use one-sided p-values to test statistical significance. The one-sided p-value is the number of placebos whose measured treatment gap in a given year was larger in the same direction as for the treatment unit, divided by the total number of placebos. Since the placebo effect may be quite large if the units were not matched well in the pre-treatment period, the measured gap for each placebo in the post-intervention period is standardized by dividing it by the pre-treatment gap size.[29]

Sensitivity Analysis

We carried out several sensitivity analyses, including the exclusion of donor pool countries with relatively large weight in the creation of the synthetic control. This was done to check for the undue influence of one country on our results. Other sensitivity analyses are described in Supplementary Section S2.

Individual Country Analysis

In addition to the weighted average of the treatment unit, we also ran SCA for each of the eight countries in Quadrant 1 of Figure 2 individually; these are provided in Supplementary Section S2.

An optimized model

Using an iterative model evaluation process we identified a best fit model with the lowest RMSPE of 0·5969 (Table 2) that left 10 of the original 20 predictors remaining. For example, potential predictors such as skilled birth attendance, physicians per 1,000 population, and four or more antenatal care visits were removed by the model optimization to achieve the minimal difference (the RMSPE) between the treatment and control in the pre-intervention period. In this best fit model using this unbiased approach, there is close agreement between the treatment unit and the synthetic control for most predictor variables, with HIV prevalence and DPT (diphtheria, pertussis, and tetanus toxoid) vaccination coverage the main exceptions (Table 3). The synthetic control consisted of eight countries: Chad, Eritrea, Gambia, Guinea Bissau, Mongolia, Namibia, Niger, and Swaziland/Eswatini (Table 4). This final model showed a very close agreement in U5MR between the synthetic control and treatment unit in the pre-intervention period up to 1999 (Figure 3). Values to the left of the red vertical line reflect the pre-intervention period. The blue line shows the real U5MR trend (weighted average) of the treatment countries in Quadrant 1 of Figure 2. Red dashed line shows the real U5MR trend of the synthetic control. There is very little difference between the blue and red lines reflecting a very low RMSPE and a good model fit.

[Insert Tables 2, 3 and 4]

Quantifying the impact of USAID investments to reduce child mortality in high-investment countries

A significant divergence between the treatment unit (blue line: countries with relatively high USAID investment in child health) and the synthetic control (red line) emerges at the start of the intervention period (1999) and continues through the end of the treatment period (Figure 3). Countries with relatively high levels of USAID investment in child health (the treatment unit) had a U5MR that was between 2 and 38 deaths per 1,000 live births lower (mean = 29) than the synthetic control in the treatment period (Table 5).

[Insert Figure 3 and Table 5]

Placebo Testing and Inference

To calculate the statistical significance of the difference between the treatment unit and the synthetic control, we check if our estimate of the difference could be due entirely to chance using what is called a placebo test (see Abadie, et al. 2010 for more details than provided below).[30] We check how often we would find the same or greater difference if we used each of the other countries in the donor pool at random as the treatment unit rather than the countries with high levels of USAID investment in child health that make up the treatment unit in this analysis. The placebo test provides a distribution of the U5MR differences between all the countries in the donor pool that are not part of the treatment unit and the synthetic control. If we find that the other countries in the donor pool have similar differences in U5MR compared to the synthetic control, then our inference is that there is insufficient statistical evidence that our treatment unit (countries with high levels of USAID investment in child health) has a significantly lower U5MR than the control unit. The results of the placebo testing are visualized in Figure 4. The red vertical line represents the intervention start year of 1999. All lines represent the mean difference between U5MR of the tested country and its synthetic control. The black line represents the difference from the synthetic control and the weighted average U5MR of the original treatment unit countries (the countries in Quadrant 1 of Figure 2 that have relatively high levels of USAID investment in child health). The grey lines represent the difference in U5MR between the 48 donor countries used as placebos and the synthetic control. The statistical significance of the placebo testing is provided in Table 5. The difference in U5MR between the synthetic control and the treatment unit (treatment unit countries with high levels of USAID investment in child health) is statistically significant in the 2000-16 treatment period (one-tailed p < .01).

[Insert Figure 4]

Sensitivity Analysis

In one sensitivity analysis, we excluded Chad from the donor pool as it had a relatively large weight among countries making up the synthetic control for a reason unclear to the authors. This was done to check if our results would provide the same inference when a highly weighted country was removed from the donor pool. As in the main analysis, there is also a significant divergence between the treatment unit (blue line: countries with relatively high USAID investment in child health excluding Chad) and the synthetic control (red line). See Figure 5. Countries with relatively high levels of USAID investment in child health (the treatment unit) had a U5MR that was between 7 and 26 deaths per 1,000 live births lower in the treatment period than the synthetic control (without Chad). See Supplementary Table S3. The estimated impact of high-level USAID investment in this sensitivity analysis was 19 deaths per 1,000 births lower, on average, as compared to the synthetic control without Chad in the donor pool. The RMSPE is only slightly higher (0·6034) in the sensitivity analysis as compared to the main analysis (Table 2). The difference in U5MR between the synthetic control (without Chad) and the treatment unit is also statistically significant (one-tailed p < .01) in the 2000-16 treatment period (Supplementary Table S3).

Another sensitivity analysis of countries with a relatively low but consistent level of USAID investments in child health (see countries in Quadrant 3 of Figures 2) also found significant, positive impacts on under-five mortality as compared to control during the treatment period (see Supplementary Table S5). The mean difference between the treatment unit made up countries with low levels of USAID investment was an under-five mortality rate that was 1.48 per 1,000 live births lower compared to its synthetic control. The difference was statistically significantly lower in the treatment unit for 16 of 17 years (2001-03 and 2005-06, p. < .05; 2007-16, p. < .01). See Supplementary Materials, Section S2 for additional sensitivity and individual country analyses.

[Insert Figure 5]

Although considered the gold standard evaluation method for drawing inferences about causality, prospective, randomized trials are often not practical for evaluation of most large-scale public health programs in low- and middle-income countries. Another program may be scaling up across a country at the same time. Or, similar programs, other than the one being evaluated, may be ongoing or completed in areas that would need to be part of a control. For these reasons, observational studies are commonly used for evaluating large-scale public health programs. In order to draw inferences from observational studies, however, steps must be taken to minimize potential biases. One common step to minimize bias in observational studies is the provision of a counterfactual through use of modeled controls or comparison units. This is done to simulate or model the change in the outcome variable that would have occurred in the absence of the intervention or treatment.

The synthetic control method provides a novel approach that uses a data-driven (unbiased) method to construct a counterfactual or control by creating a weighted average of the outcome variable from units in a donor pool that are most similar to the treatment unit(s). The synthetic control method then models the continuation of the trend of the pre-intervention period in both a separate treatment unit and a control.[31] The two trends are then compared. The difference between the synthetic control trend and the trend of the treated units is considered the treatment effect and this difference is tested for statistical significance using placebo tests. Another advantage of the synthetic control method is its ability to account for observed and unobserved time-varying confounders that may influence the outcome variable, an advantage that other key inferential methods for analyzing observational data, such as the ‘difference-in-difference’ approach, do not have.[31] In addition, model testing procedures (placebo testing) allow the analyst to assess whether or not the control units provide an adequate counterfactual to the treatment or intervention unit(s).[15] More traditional approaches (e.g., regression) were not considered for this analysis as these approaches do not provide sufficient methods to provide an unbiased selection of counterfactual countries, or easily allow differential weighting and contribution from the donor pool of countries.

The results of this synthetic control analysis provide a quantitative estimate of the impact on under-five mortality, on average, of USAID investments focused on child health among countries with relatively high levels of USAID investment (the treatment group) during the treatment period (2000-16). This period coincides with the World Health Organization’s initiative to accelerate reductions in child mortality, called the Integrated Management of Childhood Illness (IMCI) and Integrated Community Case Management (iCCM). In this analysis, the under-five mortality rate was more than 20 per 1000 live births lower, on average, in the treatment group as compared to the synthetic control (mean = 29, range 2-38). The difference is statistically significant (one tailed, p. < 01).

The inference that USAID investments have a positive impact on under-five mortality, under the conditions described above, is supported by placebo testing used to check the model used in this analysis, as well as several sensitivity analyses (see supplementary materials). For example, one sensitivity analysis with the treatment unit made up of countries with a relatively low but consistent level of USAID investments in child health also found significant, positive impacts on under-five mortality. This finding is consistent with a dose-response effect of USAID investments in child health---that greater investment is associated with a greater treatment effect---lending additional support for a causal relationship.

There is currently insufficient information available about contextual factors and program implementation across the countries with USAID child health programs, nor comparable health indicators of coverage, impact, and cost, that would allow a better understanding the relative contribution of program interventions and context to the impact quantified in this analysis, as argued by Bryce et al. and Victora et al.[9-11] As noted in the Introduction, this is the hope and desire for the future of program evaluation in low- and middle-income countries. The authors’ hypothesis, consistent with the high-level theory of change described above, is that a combination of the following types of USAID investments in child health contributed to the impact quantified in the analysis: (1) capacity building and health promotion around evidence-based interventions for reducing child mortality; (2) encouraging stakeholders to be engaged in scaling up and improving quality of these evidence-based interventions, including policy change; and, (3) facilitating the development of evidence for interventions that prevent child deaths, and for effective implementation of these interventions. While this analysis cannot identify the relative contribution of these approaches, we believe that substantial USAID investments in MCH (especially Child Survival programs) led to countries adopting IMCI and other evidence-based MCH interventions earlier and at a higher level of intensity, scale and quality than would have happened without such investments. This assumption is supported when looking at the treatment effects by individual country that make up the treated units. The two countries with the largest difference between its trend in under-five mortality and the trend of that country’s synthetic control (Uganda, Zambia) are countries that initiated IMCI earlier and were already at the expansion phase by June 1999.[20] (See supplementary figures S7 and S8). It is also important to remember that net per capita foreign aid in the pre-intervention period, among other likely confounders, were controlled for in the SCA.

Limitations

One limitation of the SCA method is that it does not control for potential confounding variables during the treatment period. It is possible that some other factor appeared in treatment countries after 1999 besides significant USAID MCH and malaria investments and national implementation of IMCI that contributed to the observed results. Arguing against this is the strong match between the treatment unit and synthetic control in a long pre-intervention period, the long-standing USAID engagement with many of the treatment countries, and the close correlation between the introduction of IMCI and the MDGs, and the treatment year. Many countries were excluded from the donor pool because of possible confounding from occasional USAID investment during the treatment period.

One particular SCA assumption relevant to many countries in the analysis is the absence of ‘shocks’ that might differentially affect the outcome variable---in this case child mortality rates---between the treated units and the countries making up the donor pool.[31] Such shocks might include governance crises, wars, pandemics, or natural disasters. However, the countries in quadrant 1 that make up the Treatment Unit did not experience substantial country-wide shocks in U5MR in the intervention period.[26] Countries that received considerable USAID funding and experienced considerable shocks in U5MR[26], and that were not included in the analysis (Haiti, Burma, and Indonesia) were also excluded from the donor pool because they did not get 16 years of funding. In addition, the source of the outcome variable, under-five mortality, is UN IGME that includes adjustments for crisis mortality in its modeled estimates.[26]

The synthetic control method assumptions requires an adequate fit between the treatment unit(s) and the synthetic control in the pre-intervention period and that the potential comparison units in the donor pool are similar to the treated unit(s).[31] There is evidence that the pre-intervention fit in this analysis more than meets the standard (RMPSE <= 3).[17] It is unlikely that the pre-intervention fit would be this good, if the treated units were not similar to the units making up the donor pool. We controlled for confounding for many cross-national differences by requiring that the synthetic control match the treatment unit on a broad range of criteria in a long pre-intervention period, including indicators of governance, health system strength, disease burden and GDP per capita. A difference in the pre-intervention period between the treatment and synthetic control groups among the predictor variables were the DPT immunization rate and HIV prevalence. The countries which were selected as the treatment unit therefore may have had stronger health systems, but also would have had a greater HIV burden putting pressure on these systems as compared to the countries that comprised the synthetic control after unbiased model selection using RMSPE. The possible contributions of these initial differences during the post-intervention period are unclear. The former may have helped reduce U5MR while the latter may have blunted reductions. In the end, these differences did not affect U5MR trajectory during the pre-intervention period. The higher HIV prevalence may also have attracted greater health system investment under well-financed HIV programs (e.g. the US President's Emergency Plan for AIDS Relief or PEPFAR), although PEPFAR was enacted in 2004 and the figures show the greatest divergence in U5MR occurring in the years prior to that. Nevertheless, it remains possible that some other important characteristics were not included in this analysis that may have contributed to their differential trajectories during the treatment period. However, the placebo control analysis provides some additional evidence against this.

We were also limited in this analysis by the types of available data. For example, we used the polity score as an indicator of governance environment because it was available for every country since 1980, whereas the World Bank’s governance effectiveness and political stability indices only became available in 1996. “Cherry-picking” models to favor a particular outcome is a concern in SCA; we avoided this by using RMSPE as our unbiased criteria for selecting models.[32]

Synthetic control analysis (SCA) is a valuable addition to a range of approaches for quantifying the impact of donor programs, and for making causal inferences about the results of population level health interventions, in general, when a randomized trial is impractical. It has certain advantages over other more widely used approaches and is one of a small number of methods that may control for unmeasured time-varying confounders. Wider use of SCA, along with its dissemination and critique, will help to develop a better understanding of its strengths and limitations and optimal conditions for use.[31] The authors welcome additional attempts to replicate, publish and critique this approach.

Not unexpectedly, donor investments that complement host government child health programs appear to significantly accelerate reductions in under-five mortality when those investments are substantial in absolute amounts and per capita. Given the review of common USAID activities and approaches during the period of analysis, we hypothesize that a combination of the following approaches contributed to the observed impact, although the relative contributions of each cannot be assessed here: capacity building and community health promotion around evidence-based interventions; encouraging stakeholders to be engaged in scaling up and improving quality of these interventions; and facilitating intervention and implementation research in support of intervention scale and quality.

STATA statistical data files and code used in this analysis can be found in the following web folder: https://tinyurl.com/WeissSyntheticControl

BMGF: Bill and Melinda Gates Foundation

DAH: Development Assistance for Health

DFID: The United Kingdom's Department for International Development

DPT: Diphtheria, pertussis, and tetanus toxoid vaccine

iCCM: Integrated Community Case Management

IMCI: Integrated Management of Childhood Illnesses

LMICs: Low- and middle-income countries

MCH: Maternal and child health

MDG: Millennium Development Goals

PEPFAR: The President's Emergency Plan for AIDS Relief

RMSPE: Root-mean-squared-prediction-error

SCA: Synthetic Control Analysis

UNICEF: The United Nations Children's Fund

UN IGME: The United Nations Inter-Agency Group for Child Mortality Estimation

USAID: United States Agency for International Development

U5MR: Under-five mortality rate

WHO: The World Health Organization

Global Burden of Disease Health Financing Collaborator N: Past, present, and future of global health financing: a review of development assistance, government, out-of-pocket, and other private spending on health for 195 countries, 1995-2050. Lancet 2019, 393(10187):2233-2260.
Dieleman JL, Micah AE, Murray CJL: Global Health Spending and Development Assistance for Health. JAMA 2019, 321(21):2073-2074.
Pitt C, Grollman C, Martinez-Alvarez M, Arregoces L, Borghi J: Tracking aid for global health goals: a systematic comparison of four approaches applied to reproductive, maternal, newborn, and child health. Lancet Glob Health 2018, 6(8):e859-e874.
Chan M, Kazatchkine M, Lob-Levyt J, Obaid T, Schweizer J, Sidibe M, Veneman A, Yamada T: Meeting the demand for results and accountability: a call for action on health data from eight global health agencies. PLoS Med 2010, 7(1):e1000223.
What We Do, MATERNAL, NEWBORN & CHILD HEALTH, PRESS RELEASES [https://goo.gl/KDVj82]
Development Aid: Aid by Sector [https://devtracker.dfid.gov.uk/sector/2/categories/122/projects/12220]
Development Experience Clearinghouse [https://bit.ly/2ScIjMA]
Habicht JP, Victora CG, Vaughan JP: Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol 1999, 28(1):10-18.
Victora CG, Black RE, Boerma JT, Bryce J: Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations. Lancet 2011, 377(9759):85-95.
Bryce J, Victora CG, Habicht JP, Vaughan JP, Black RE: The multi-country evaluation of the integrated management of childhood illness strategy: lessons for the evaluation of public health interventions. Am J Public Health 2004, 94(3):406-415.
Victora CG, Schellenberg JA, Huicho L, Amaral J, El Arifeen S, Pariyo G, Manzi F, Scherpbier RW, Bryce J, Habicht JP: Context matters: interpreting impact findings in child survival evaluations. Health Policy Plan 2005, 20 Suppl 1:i18-i31.
Heidkamp R, Group NEPW: The National Evaluation Platform for Maternal, Newborn, and Child Health, and Nutrition: From idea to implementation. J Glob Health 2017, 7(2):020305.
Bryce J, Victora CG, Boerma T, Peters DH, Black RE: Evaluating the scale-up for maternal and child survival: a common framework. Int Health 2011, 3(3):139-146.
Abadie A, Diamond A, Hainmueller J: Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program. Cambridge, Mass.: National Bureau of Economic Research; 2007.
Kreif N, Grieve R, Hangartner D, Turner AJ, Nikolova S, Sutton M: Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units. Health Econ 2016, 25(12):1514-1528.
Lepine A, Lagarde M, Le Nestour A: Free primary care in Zambia: an impact evaluation using a pooled synthetic control method. In. Health, Econometrics and Data Group (HEDG) Working Papers 15/20: HEDG, c/o Department of Economics, University of York; 2015.
Pieters H, Curzi D, Olper A, Swinnen J: Effect of democratic reforms on child mortality: a synthetic control analysis. Lancet Glob Health 2016, 4(9):e627-632.
Tulloch J: Integrated approach to child health in developing countries. Lancet 1999, 354 Suppl 2:SII16-20.
Lambrechts T, Bryce J, Orinda V: Integrated management of childhood illness: a summary of first experiences. Bull World Health Organ 1999, 77(7):582-594.
WHO: Integrated Management of Childhood Illness: Global status of implementation. In: IMCI information. World Health Organization; 1999.
USAID: USAID Child Survival and Disease Programs Fund Progress Report Fiscal Year 2001. In.; 2002.
USAID: USAID Child Survival and Health Programs Fund Progress Report Fiscal Year 2002. In.; 2003.
USAID: USAID Child Survival and Health Programs Fund Progress Report Fiscal Year 2003. In.; 2004.
USAID: USAID Child Survival and Health Programs Fund Progress Report Fiscal Year 2004. In.; 2005.
UN: World Population Prospects: The 2017 Revision, Key Findings and Advance Tables. . In.: United Nations, Department of Economic and Social Affairs, Population Division 2017.
IGME UN Inter-agency Group for Child Mortality Estimation [http://childmortality.org/]
Kuruvilla S, Schweitzer J, Bishai D, Chowdhury S, Caramani D, Frost L, Cortez R, Daelmans B, de Francisco A, Adam T et al: Success factors for reducing maternal and child mortality. Bull World Health Organ 2014, 92(7):533-544B.
Cohen RL, Murray J, Jack S, Arscott-Mills S, Verardi V: Impact of multisectoral health determinants on child mortality 1980-2010: An analysis by country baseline mortality. PLoS One 2017, 12(12):e0188762.
Galiani S, Quistorff B: The synth_runner package: Utilities to automate synthetic control estimation using synth. Stata Journal 2017, 17(4):834-849.
Abadie A, Diamond A, Hainmueller J: Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program. Journal of the American Statistical Association 2010, 105(490):493-505.
Bouttell J, Craig P, Lewsey J, Robinson M, Popham F: Synthetic control methodology as a tool for evaluating population-level health interventions. Journal of Epidemiology and Community Health 2018, 72(8):673.
Ferman B, Pinto C, Possebom V: Cherry picking with synthetic controls. Textos para discussão 420, FGV/EESP - Escola de Economia de São Paulo, Getulio Vargas Foundation (Brazil) 2016, 420.

Acknowledgments

We thank Gustavo Angeles of the University of North Carolina for his advice on carrying out the synthetic control method. We thank Bhakti Hansoti of the Johns Hopkins University schools of Medicine and Public Health for comments on the manuscript. We thank Vanessa A. Vassall of Florida International University for proofreading the manuscript.

Funding

The authors received salary support from organizations with USAID contracts during the preparation of this manuscript. SAID, via contracts with the Public Health Institute (Cooperative agreement #7200AA18CA00001) and CAMRIS International (Contract #AID-OAA-C-16-00031), provided some salary support to the authors during the preparation of this manuscript. Note that this article reflects the views of the authors and does not necessarily reflect the views of USAID or the United States Government, the Public Health Institute, or Camris International. The corresponding author (WW) has full access to all the data in the study and has final responsibility for the decision to submit for publication. The funding organization (USAID) played no role in commissioning this analysis, in the study design, data collection, decision to publish, or preparation of the manuscript.

Authors' Contributions

WW conceived and initially designed the study and carried out the original model testing. BP and RC contributed to selection of countries and predictor variables and further model testing. WW and AA drafted the initial manuscript and developed the theory of change. All authors commented and revised the manuscript critically. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Corresponding author

Correspondence to William Weiss ([email protected])

Ethics Approval and Consent to Participate

Not applicable

Consent for Publication

Not applicable

Competing Interest

The authors declared no competing interests.

Supplementary Information

Supplementary information can be found at the following link: https://tinyurl.com/WeissSupplementaryMaterials. Section S1 summarizes the utility of the synthetic control method. Section S2 provides additional sensitivity analyses and individual country analyses. Additional supplementary tables are also included.

Table 1. Policy Areas Tested for Synthetic Cohort Model.

Policy Area	Variables Tested in Synthetic Cohort Model
Wealth	Log GDP per capita (constant 2010 USD)
	Official development assistance per capita (USD)
Service Delivery	Skilled Birth Attendance (%)
	Physicians per 1,000 population
	Antenatal care visits (4+)
Health Financing	Health Expenditure per capita (2011 international $
	Out-of-pocket health spending (% of total health spending)
Immunizations	DPT immunization (% of 12-23-month-olds)
Malaria or HIV	HIV prevalence (% 15-49 year olds)
Fertility	Total fertility rate (births per woman)
Nutrition	Stunting (% of children under 5)
Governance	Government effectiveness index
	Political stability index
	Polity score
Infrastructure	Percent of population in urban areas
	Land area
	Population density
Water, Sanitation, and Hygiene	Access to improved water source (%)
	Access to improved sanitation (%)
Education	Average years of female education, women aged 20-24

Table 2: Comparison of Synthetic Quadrant 1 Root-Mean-Squared-Prediction-Errors (RMSPE)

Synth Models	RMSPE
*Outcome lags only*
1980, 1990, 1998	2.3624
*Predictors with lags*
TFR, Stunting, HIV + 3 lags	0.9203
TFR, Stunting, HIV, DPT + 3 lags	0.8477
Clean water and Sanitation + 3 lags	1.0187
Logged GDP+Urbanization+ODApc + 3 lags	1.7597
Polity score + 3 lags	2.0387
Logged GDP+Urbanization+ODApc + Polity score+ 3 lags	2.2416
*Predictors without lags*
TFR, Stunting, HIV, DPT	4.5781
Clean water and Sanitation	3.9344
Logged GDP+Urbanization+ODApc	26.7628
*Full model*
All 10 predictors + 3 lags	0.5969
All 10 predictors + 3 lags (without Chad)	0.6034
*Predictor year range*
1985-1998	0.6219
1990-1998	0.6438
*Quadrant 3*
All 10 predictors + 3 lags	0.0618
*Non-nested*
All 10 predictors + 3 lags	1.1046
With the exception "Predictor year range", all RSMPE were generated with pretreatment period 1980-1998

Table 3: Predictor means between synthetic and test case in pre-intervention period

- Quadrant 1 Countries.

Variables	Real	Synthetic
TFR	6.56	6.77
Stunting	49.22	48.45
HIV	5.10	1.32
DPT	47.73	30.42
Sanitation	17.10	17.03
Clean water	40.30	44.20
Logged GDP	6.08	6.49
Urbanization	23.36	23.36
ODA per capita	41.06	41.10
Polity score	-2.90	-2.91
Under-5 mortality (1998)	167.31	167.31
Under-5 mortality (1990)	189.46	189.43
Under-5 mortality (1980)	219.89	219.85

*Ordering of these predictors is important in the nested analysis and the results can be replicated if the variables are entered in Stata syntax in the above manner.

Table 4: Country weights in synthetic Quadrant 1.

Percent composition of synthetic control.

Country	Weight
Chad	0.590
Eritrea	0.031
Gambia	0.068
Mongolia	0.040
Namibia	0.076
Niger	0.077
Guinea-Bissau	0.040
Swaziland	0.078

Table 5: Post-treatment results: Effects and their p-values. Effect size is in units of U5MR. p-values are exact, empirical p-values based on placebo testing. Standardization involves dividing the effect size by RMSPE.

		2-sided	Standardized	1-sided	Standardized
Year	Estimates	p-values	2-sided p-values	p-values	1-sided p-values
1999	-2.156	0.326	0.130	0.130	0.086
2000	-5.528	0.130	0.001	0.043	0.001
2001	-9.991	0.065	0.001	0.001	0.001
2002	-14.917	0.043	0.001	0.001	0.001
2003	-19.921	0.043	0.001	0.001	0.001
2004	-24.590	0.043	0.001	0.001	0.001
2005	-28.302	0.065	0.001	0.001	0.001
2006	-30.641	0.065	0.022	0.001	0.001
2007	-32.603	0.065	0.022	0.001	0.001
2008	-34.145	0.065	0.022	0.001	0.001
2009	-35.084	0.065	0.022	0.001	0.001
2010	-35.352	0.065	0.022	0.001	0.001
2011	-35.457	0.043	0.043	0.001	0.001
2012	-36.681	0.022	0.043	0.001	0.001
2013	-37.445	0.022	0.043	0.001	0.001
2014	-38.004	0.022	0.043	0.001	0.001
2015	-38.566	0.022	0.043	0.001	0.001
2016	-38.496	0.022	0.043	0.001	0.001

Table 6: Post-treatment results: Effects and their p-values excluding Chad from donor pool.

		2-sided	Standardized	1-sided	Standardized
Year	Estimates	p-values	2-sided p-values	p-values	1-sided p-values
1999	-3.559	0.261	0.109	0.130	0.087
2000	-7.426	0.087	0.022	0.022	0.001
2001	-11.899	0.065	0.001	0.001	0.001
2002	-16.354	0.043	0.022	0.001	0.001
2003	-20.462	0.043	0.022	0.001	0.001
2004	-23.904	0.043	0.022	0.001	0.001
2005	-26.239	0.065	0.022	0.001	0.001
2006	-25.630	0.065	0.043	0.001	0.001
2007	-25.301	0.065	0.065	0.001	0.001
2008	-25.522	0.065	0.065	0.001	0.001
2009	-24.284	0.065	0.065	0.001	0.001
2010	-21.926	0.087	0.130	0.001	0.001
2011	-18.043	0.152	0.152	0.043	0.001
2012	-18.149	0.130	0.152	0.022	0.001
2013	-17.884	0.130	0.130	0.022	0.001
2014	-17.431	0.130	0.130	0.022	0.001
2015	-17.499	0.130	0.130	0.022	0.001
2016	-17.845	0.130	0.109	0.022	0.001

PDF

Version 1

posted 09 Jun, 2020

You are reading this latest preprint version

Estimating the impact of donor programs on child mortality in low- and middle-income countries: a synthetic control analysis of child health programs funded by the United States Agency for International Development

Status:

Version 1

Abstract

Figures

Background

Estimating the impact of United States Agency for International Development child health programs

Methods

Synthetic Control Analysis Method

Treatment Unit

Donor Pool

Time Period

Dependent Variable

Predictor Variables

Assumptions

Model Optimization

Procedure

Statistical Analysis and Inference

Sensitivity Analysis

Individual Country Analysis

Results

An optimized model

Quantifying the impact of USAID investments to reduce child mortality in high-investment countries

Placebo Testing and Inference

Sensitivity Analysis

Discussion

Limitations

Conclusions

Availability Of Data And Materials

Abbreviations

References

Declarations

Tables

Supplementary Files

Status:

Version 1