Study design
We used a combination of existing datasets collected for programmatic purposes by humanitarian and government actors (see below) to develop and evaluate country-specific models to predict various anthropometric indicators at the resolution of one month and a single administrative level 2 unit (district in Somalia, county in South Sudan), hereafter referred to as a ‘stratum’.
Drawing from an a priori causal framework of factors leading to wasting (Additional file 1, Figure S5), we identified potential predictor variables collected at the desired resolution, and merged these with individual child-level data from SMART surveys designed to be representative of single strata. We fitted various candidate models to a training data subset, and evaluated their predictive accuracy on a validation data subset, as well as on cross-validation.
Study population and timeframe
For Somalia (including Somaliland and Puntland), we sourced predictor and anthropometric survey data from January 2014 to December 2018 inclusive. During this period, Somalia’s population rose from about 12.8M to 14.5M [10]. Surveys were done in 22 (29%) of Somalia’s 75 districts. For South Sudan, the analysis spanned January 2015 to April 2018, and featured surveys from 63 (80%) of the country’s 79 counties, as per 2013 administrative borders. South Sudan’s population declined from 10.2M to 9.7M during the period, reflecting refugee movements to neighbouring countries [11].
Data sources
Anthropometric surveys. We accessed reports and raw datasets of 177 SMART surveys from South Sudan (two were excluded due to very unusual values, leaving 175 analysis-eligible), and 167 from Somalia (82 were excluded: 76, mainly done before 2016, were representative of livelihood zones rather than districts, and thus could not be coupled with predictor data; five appeared to have followed a non-representative sampling design; one had no available dataset, leaving 85 analysis-eligible). For each survey, we inspected the report to identify any possible bias sources and, in particular, any reported restriction of the effective sampling frame due to insecurity or inaccessibility (e.g. if a report stated that two out of 12 boma, South Sudan’s administrative level 3 unit, could not be included in the sample, we approximated the sampling coverage as 2/12 ≈ 83%). We also rescaled the ENA software-reported quality score for the survey (a composite of several indicators including proportion of outlier values, digit preference and properties of the distribution of observed values, ranging from 0% = best to 50% = worst [12]) to a 0-100% range, where best = 100%. We reanalysed all surveys by converting the raw anthropometric readings (weight, height or length, age, middle-upper arm circumference or MUAC) into z-score indices as per the World Health Organization 2006 standardised anthropometric distributions using the anthro package in R, flagging and excluding all observations with missing values, <> 5 z-scores from the mean and/or outside the allowed age range (6-59mo). Lastly, we classified all children into severe wasting or wasting according to two alternative definitions: (i) bilateral oedema and/or weight-for-height (WHZ) <3Z (severe wasting) or <2Z (wasting); (ii) bilateral oedema and/or MUAC < 115mm (severe wasting) or < 125mm (wasting) [13]. We fitted generalised linear models (binomial for severe wasting and wasting, gaussian otherwise) with standard errors adjusted for cluster design to verify concordance with point estimates and 95% confidence intervals (CI) contained in the survey reports.
Predictors. We developed a causal framework of wasting (Additional file 1, Figure S5) based on existing evidence and plausibility reasoning. We used this framework to identify factors potentially predicting the outcomes of interest. We searched for candidate predictor data representing these factors online and through contacts with humanitarian actors in both Somalia and South Sudan, the main desirable characteristics of datasets being stratification by stratum and month, and that data be generated routinely for programmatic purposes, i.e. realistically available without further primary data collection. Most datasets had already been sourced as part of similar projects to retrospectively estimate mortality in both countries [10, 11]. Candidate predictors for both Somalia and South Sudan are detailed in Table 1 and Table 2, respectively. Each predictor dataset was subjected to data cleaning to remove obvious errors. We excluded predictors that were missing for ≥30% of strata or ≥30% of months. Remaining completeness problems were resolved through interpolation (humanitarian presence), manual imputation (missing market data points were attributed a weighted average of the geographically nearest market’s value and the mean of all other non-missing markets, with 0.7 and 0.3 weights respectively) and automatic imputation using the mice R package [14] (water price, severe wasting and wasting treatment quality). To reduce stochastic noise in the time series, we computed three-month window rolling means for all time-varying predictors, and applied moderate local spline smoothing to terms of trade or market price variables. Where appropriate, we computed per-population rates using stratum-month population figures previously estimated as part of mortality estimation projects for each country. Briefly, these combine available base estimates (census projections in South Sudan; quality-weighted averages of four alternative sources in Somalia), natural growth assumptions and data on refugee as well as internal displacement to and from each stratum, by month.
Table 1
Candidate predictor datasets, Somalia.
Predictor | Variable(s) | Domain | Time span of availability | Source(s) | Notes and assumptions |
Administrative level | Administrative entity within Somalia | (various) | n/a (static variable) | n/a | Somaliland, Puntland, south-central Somalia |
Rainfall | Total rainfall (mm) | Climate | 2013 to 2018 | Climate Engine (https://clim-engine-development.appspot.com/fewsNet) [15] | |
Mean of Standard Precipitation Index | 2016 to 2018 | Compares current rainfall with historical averages. |
Vegetation density | Normalised Difference Vegetation Index | Climate | 2013 to 2018 | Food Security and Nutrition Analysis Unit - Somalia (FSNAU) | |
Incidence of armed conflict events | events per 100,000 population deaths per 100,000 population | Exposure to armed conflict / insecurity | 2010 to 2018 | Armed Conflict Location & Event Data Project (ACLED, https://www.acleddata.com/) [16] | Meta-data on individual armed conflict events based on extensive review of multi-language media sources and other public information. |
Incidence of attacks against aid workers | deaths per 100,000 population injuries per 100,000 population | Exposure to armed conflict / insecurity | 2010 to 2018 | Aid Worker Security Database (AWSD, https://aidworkersecurity.org/incidents) | Data on various types of attacks to aid workers, capturing information from media sources, aid organisations and security actors. |
Proportion of IDPs | proportion of IDPs among total district population | Forced displacement | 2016 to 2018 | Estimated by authors as part of a separate mortality study [10]. | |
Main local livelihood type | Pastoral, agropastoral, riverine and urban | Food security and livelihoods | n/a (static variable) | FSNAU | Assumed to be constant over time. |
Water price | Price of 200L drum of water in Somali Shillings | Food insecurity and livelihoods | 2013 to 2018 | FSNAU | |
Terms of trade purchasing power index | Kcal equivalent of local cereals that an average local-quality goat can be exchanged for | Food insecurity and livelihoods | 2013 to 2018 | Calculated by the authors based on FSNAU price data from 100 sentinel markets. | See Annex. |
Kcal equivalent of local cereals that can be purchased with an average daily labourer wage |
Incidence of admission to nutritional therapeutic services | cases of severe wasting admitted to treatment services per 100,000 population | Nutritional status | 2011 to 2018 | Nutrition Cluster, Somalia | Unpublished data. |
cases of wasting admitted to treatment services per 100,000 population | 2013 to 2018 |
Cholera incidence | cases per 100,000 population | Disease burden (epidemic) | 2013 to 2018 | FSNAU | Suspected and confirmed cases. |
Measles incidence | cases per 100,000 population | Disease burden (epidemic) | 2013 to 2018 | FSNAU | Suspected and confirmed cases. |
Malaria incidence | cases per 100,000 population | Disease burden (endemic) | 2013 to 2018 | FSNAU | Suspected and confirmed cases. |
Humanitarian actor presence | Ongoing humanitarian projects per 100,000 population (all sectors) | Humanitarian (public health) service functionality | 2010 to 2018 | United Nations Office for Coordination of Humanitarian Affairs | Proxy of intensity of humanitarian response. Unpublished data. |
Ongoing projects per 100,000 population (health, nutrition and water, hygiene and sanitation) |
Food security humanitarian services | Proportion of the population that are a beneficiary of any food security service | Humanitarian (public health) service coverage | Jan 2013 to Apr 2018 | Food Security Cluster, Somalia | Unpublished data. |
Proportion of the population that are a beneficiary of cash-based food security services | Humanitarian (public health) service coverage |
Proportion of the population that are a beneficiary of food distributions | Humanitarian (public health) service coverage |
Quality of severe wasting treatment | Proportion of severe wasting admissions that exit the treatment programme cured | Humanitarian (public health) service quality | 2011 to 2018 | Nutrition Cluster, Somalia | Unpublished data. |
Table 2
Candidate predictor datasets, South Sudan.
Variable | Value(s) | Domain | Time span of availability | Source(s) | Notes and assumptions |
Administrative level | Broad region within South Sudan | (various) | n/a (static variable) | n/a | northeast, northwest, southern |
Rainfall | Difference between current rainfall and 10y historical average (mm) | Climate | 2014 to 2018 | United Nations World Food Programme Food Security Analysis data site ( http://dataviz.vam.wfp.org/seasonal_explorer/rainfall_vegetation/visualizations) | |
Incidence of armed conflict events | events per 100,000 population deaths per 100,000 population | Exposure to armed conflict / insecurity | 2010 to 2018 | Armed Conflict Location & Event Data Project (ACLE, https://www.acleddata.com/) [16] | Meta-data on individual armed conflict events based on extensive review of multi-language media sources and other public information. |
Incidence of attacks against aid workers | deaths per 100,000 population injuries per 100,000 population | Exposure to armed conflict / insecurity | 2010 to 2018 | Aid Worker Security Database (AWSD, https://aidworkersecurity.org/incidents) | Data on various types of attacks to aid workers, capturing information from media sources, aid organisations and security actors. |
Proportion of IDPs | proportion | Forced displacement | 2012 to 2018 | Estimated by authors as part of a separate mortality study [11]. | |
Main local livelihood type | agriculturalist, agropastoral, pastoralist, displaced (Protection of Civilians camps only) | Food security and livelihoods | n/a (static variable) | Famine Early Warning Systems Network (FEWS NET) [17] | Assumed to be constant over time. |
Terms of trade purchasing power index | Kg of white wheat flour that an average medium goat can be exchanged for | Food insecurity and livelihoods | 2011 to 2018 | CLiMIS portal (http://climis-southsudan.org/) | |
Food distributions | metric tonnes per 100,000 population | Food insecurity and livelihoods | 2013 to 2018 | United Nations World Food Programme | Unpublished data. |
Incidence of admission to nutritional therapeutic services | cases of severe wasting admitted to treatment services per 100,000 population | Nutritional status | 2015 to 2018 | Nutrition Cluster, South Sudan | Unpublished data. |
cases of wasting admitted to treatment services per 100,000 population |
Cholera incidence | cases per 100,000 population | Disease burden (epidemic) | 2012 to 2018 | World Health Organization | Suspected and confirmed cases. No cases reported before 2014. Unpublished data. |
Measles incidence | cases per 100,000 population | Disease burden (epidemic) | 2012 to 2018 | World Health Organization | Suspected and confirmed cases. Unpublished data. |
Humanitarian actor presence | actors per 100,000 population (all sectors; health, nutrition and water, hygiene & sanitation; health only) | Humanitarian (public health) service functionality | 2014 to 2018 | United Nations Office for Coordination of Humanitarian Affairs | Proxy of intensity of humanitarian response. Unpublished data. |
Acute flaccid paralysis incidence | cases per 100,000 population | Humanitarian (public health) service functionality | 2012 to 2018 | World Health Organization | Proxy of functionality of public health surveillance. |
Uptake of measles routine vaccination | doses given per 100,000 population | Humanitarian (public health) service coverage | 2012 to 2018 | World Health Organization | Assume no value = no routine vaccination taking place. |
Quality of severe wasting treatment | Proportion of severe wasting admissions that exit the treatment programme cured | Humanitarian (public health) service quality | 2015 to 2018 | Nutrition Cluster, Somalia | Unpublished data. |
Quality of wasting treatment | Proportion of wasting admissions that exit the treatment programme cured | Humanitarian (public health) service quality | 2015 to 2018 | Nutrition Cluster, Somalia | Unpublished data. |
While for both countries data on food security and nutritional therapeutic services were available (Table 1, Table 2) and moderately predictive (data not shown), we ultimately decided to exclude them as candidate predictors for two reasons: (i) we considered that improved prediction could plausibly result in better targeting of these humanitarian services, which in turn would result in improved nutrition, a reverse-causal effect whose future size the model might fail to predict; and (ii) we assumed that end-users would benefit from a model that could be used to predict malnutrition burden even where none of these services were available, e.g. due to access constraints.
Predictive models
We explored two prediction approaches, as follows.
Generalised linear modelling. We first split the data by period into a training set (consisting of approximately the chronologically first 70% of the data) and a ‘holdout’ (i.e. validation) set (the most recent 30%). For each anthropometric indicator, we fitted generalised linear models (GLM) to individual child observations in the training dataset, with robust standard errors to account for the cluster sampling design of most surveys, a quasi-binomial distribution for binary outcomes (severe wasting, wasting) and a gaussian distribution for continuous outcomes (WHZ, MUAC), which we did not transform as they were normally distributed. We specified model weights as the product of survey quality score and survey sample coverage.
After visual inspection, we categorised continuous predictors, and selected categorical versus continuous versions of these based on linearity of the association and the smallest-possible Chi-square (for binary outcomes) or F-test (continuous outcomes) p-value testing whether the univariate model provided better fit than a null model. We also used this p-value to select among candidate lags for each predictor; however, we modelled climate variables (rainfall, Normalised Difference Vegetation Index or NDVI) as either the means of the two trimesters, or the mean over the semester prior to each survey observation. We then fitted models consisting of all possible combinations of predictors, and shortlisted the best 10% based on predictive accuracy (lowest mean square error, MSE) of model predictions at stratum-month level, relative to observations in the holdout dataset. We manually selected the best fixed effects model among these based on relative accuracy on holdout data, accuracy on external data simulated through leave-one-out cross-validation (LOOCV) [18], the plausibility of observed associations, and model parsimony (while the latter characteristic is relatively unimportant for prediction, in practice we wished to avoid users of the model having to collect a large amount of predictor data). Lastly, we explored plausible two-way interactions.
We also fitted mixed models (with stratum as a random effect, given that in both countries surveys were repeated in many districts / counties). The latter, however, offered inconsistent accuracy advantages over fixed effects models on either cross-validation or holdout datasets. Furthermore, we assumed that end users would be most interested in predicting malnutrition prevalence in hard-to-survey districts / counties, i.e. where no a priori random effects would be estimable. For these reasons, we discarded mixed models altogether.
Machine learning. After splitting data as above, we used the ranger package [19] to grow random forest (RF) regression models on the training dataset, aggregated at stratum-month level: this approach makes minimal assumptions about data structure; briefly, it partitions the data according to various randomly generated ‘trees’, where each node is defined by a particular value of one of the predictor variables, with branches being the resulting split in the data; the ‘depth’ of each tree is defined by the number of variables that are used to create nodes; randomness is introduced by the choice of variables to build any given tree, values at which splits occur, and the order of variables in the tree structure. The distribution of the outcome arising from the partitions in each tree is compared to the observed data to determine accuracy. RF averages predictions across a large ensemble of trees. We grew RFs with 1000 trees, using all candidate predictors as above, and computed prediction CIs using a jack-knife estimator [20].
Performance evaluation
For both the GLM and RF approach, we present various metrics of predictive accuracy, for estimation: (i) effective coverage, defined here as the proportion of stratum-months for which the predicted point estimate fell within the 95% or 80%CIs of the observed data; (ii) relative bias, defined as \(\frac{1}{n}\sum _{i=1}^{i=n}\frac{{\widehat{y}}_{i}-{y}_{i}}{{y}_{i}}\), where \(n\) is the number of stratum-months, \({\widehat{y}}_{i}\) the prediction and \({y}_{i}\)the observation for stratum-month \(i\); and (iii) relative precision, namely the mean ratio of predicted stratum-month one-sided 95%CIs to point estimate; and for classification: (iv) sensitivity and (v) specificity of predictions against severe wasting or wasting prevalence thresholds commonly used in humanitarian response, and adopting observed point estimates as the gold standard. For brevity we present only best models for ‘now-casting’ (i.e. prediction of malnutrition based on data collected up to the present). We also explored models for forecasting malnutrition 3 months into the future (i.e. prediction based on data collected up to 3 months previously), but found that these had low performance (data not shown). All analysis was done using R software [21] through the RStudio [22] platform.