Measuring Governance: Developing a Novel Metric for Assessing the Translation of Policies into Action in Nepal

The Nutrition Governance Indicator (NGI) defines a first standardized approach to quantifying the ‘quality of governance’ in relation to national plans of action to accelerate improvements in nutrition. It was created in response to growing demand for evidence-based measures that reveal opportunities and challenges as nutrition-related policies on paper are translated into outcomes on the ground. Past efforts to measure ‘governance’, most notably the WHO’s Nutrition Governance Indicator and Hunger and Nutrition Commitment Index (HANCI) lack granularity below the national level, and failure to capture pinch points relating to necessary cross-sectoral actions. The NGI addresses these caveats. This paper introduces the approach that underpins the NGI, discusses its methodology and demonstrates its application in the context of Nepal.


3
The NGI passed validity tests with the exception of two fit indices. Two of the six domains passed the reliability test (Understanding nutrition and Collaboration), the Financial resources domain marginally passed at 0.70 criteria of Nunally, while the remaining domains need re-assessment to improve their reliability. The Nepal Health sector showed the most improvement in mean NGI score over a two-year period followed by the Agricultural sector. About half of all health sector representatives were more likely to have a high score compared to other sectors (OR 0.52, 95% CI 0.36 -0.74).

Conclusion
Using two panels of data collected in Nepal, this paper has shown that policy-toaction translation can be quantified and applied to sub-national institutions. These findings underscore the practical utility of the index and its importance to the nutrition policy realm.

Background
Child malnutrition continues to be a significant global public health concern.
Roughly 155 million preschool children were recorded in 2016 as being stunted (too short for their age against internal standards), which represents growth failure at some point in their life from conception through to 5 years of age. (1) While the trend is downward at a global level, progress in resolving malnutrition is patchy geographically (with Africa and South Asia lagging behind the rest of the world) and too slow. As a result, generations of children in mostly low-income countries continue to face the many health risks, impaired psychosocial development, impeded educational attainment and longer-term economic hurdles associated with malnutrition. (2) 4 The international community has responded to this challenge by agreeing to a United Nations-endorsed Decade of Action for Nutrition (2016)(2017)(2018)(2019)(2020)(2021)(2022)(2023)(2024)(2025).(3) Among the targets set for this decade was a 40% reduction in the number of children who are stunted by 2025, to be achieved by an effective consolidation and alignment of international actions and actors. (2),(4). A set of targeted interventions have been agreed, such as accelerating the promotion of exclusive breastfeeding, greater coverage of ante-natal care services, and increasing the access of vulnerable consumers to foods fortified with key vitamins and minerals. (5) However, while there is broad agreement on many of the technical components of these kinds of programmatic actions for nutrition, it is also widely agreed that successful interventions are ones that are implemented in a conducive or 'enabling' policy environment. That is, individual programs are generally cost-effective and sustained if supported by appropriate "structures and policies amenable to project goals."(6), (7), (8) That is because interventions do not operate in a vacuum; they are implemented in a context that has institutional and individual (human capacity) characteristics that shape how professionals, and non-government stakeholder partners, are able to carry out their responsibilities. This is where "governance" comes in. The ability of countries to effectively translate policies on paper into desired outcomes on the ground is key to the achievement of national goals. Weak governance has been repeatedly identified as a threat to achieving national and global nutrition goals.(9), For example, a recent review of 75 studies on the drivers of effective action by governments found that an inability to implement even welldesigned policies for nutrition was often linked to" the absence of institutional ownership for nutrition, and institutional failure." (10) Unfortunately, the study of governance (successes or failures) has been hampered 5 by the lack of agreed metrics for empirically measuring the processes involved.
Most of the literature pertaining to nutrition governance is based in qualitative interviews with key informants and/or desk reviews of the presence/absence of key policy documents and legislation that would be supportive of national nutrition goals. There have been few attempts to establish more quantifiable metrics that could be compared across country situations or monitored over time.(11),(12), (13) In their review of the state of evidence on processes that underpin political and policy successes for nutrition, Gillespie et al. (2013) concluded that "analyses about how to shape and sustain enabling environments is essential", and that "the collection of credible metrics …is desperately needed in this area." (6) This paper seeks to address that important gap by introducing a novel metric for assessing the effectiveness of nutrition governance at all levels of administrative responsibility, from central level line ministry down to grassroots services providers on the ground. Using empirical data collected in Nepal, we used a principal component analysis (PCA) approach to calculate a nutrition governance index based on participants' weighted responses on many different facets of governance. We then validated the tool using a confirmatory factor analysis (CFA). The goal of this work is to demonstrate that this new metric offers insights of potential value to both researchers and policymakers keen to gain an improved understanding of how complex multi-institutional policies on paper are translated into practice.

Measuring Nutrition Governance
Defining metrics of governance is a challenge.(6) Decision making within and across governments is typically opaque, decisions usually emerge over time rather than appear fully formed as discrete events, and there are many hurdles to "accessing the many different, geographically widespread actors, individuals, groups and 6 networks involved in policy processes." (14) As a result, most attempts at standardizing measures of nutrition governance have used national level benchmarks based on available data, such as the presence or absence of certain kinds of policy documents, budgetary allocations, and staffing levels. (15), (16) The three recent and widely cited approaches are i) the World Health Organization's Nutrition Governance Index, ii) the Hunger and Nutrition Commitment Index (HANCI), and iii) the Political Commitment (for Nutrition) Rapid Assessment Tool.
Each is reviewed briefly below.

Nutrition Governance Index
In its global analysis of governments' readiness to accelerate actions on nutrition, the World Health Organization chose to focus on the concepts of 'commitment' (willingness to act) and 'capacity' (readiness to act). just outcomes) to allow for improved nutrition governance. (17) A total of 11 indicators were pulled together for 36 low-and middle-income countries with the highest burden of child stunting; the indicators were equally weighted, to generate a simple index from 0 to 11. (18) The "strength of nutrition governance" was arbitrarily classified as 'weak' for countries scoring from 0 to 6.9, 'medium' for those scoring from 7 to 9.9, and 'strong' for those scoring from 10 to 11.0. Using 2007/08 data, 10 countries were classified as having 'strong' nutrition governance, including Peru, Malawi and Vietnam (all countries that have made great strides in reducing the prevalence of stunting since the 1990s), while 12 had 'weak' nutrition governance, including Cambodia, Mali and Pakistan. Countries facing the persistent challenges of armed conflict were all in the weak category, including Afghanistan, Iraq, Yemen and the Democratic Republic of Congo (DRC). (18) While the ranking of countries is intuitive, this approach to determining effective governance is limited by its lack of granularity; that is, its inability to differentiate across sectors of government activity or below the national (central) level of government. It is also unable to determine which facets of governance appear to be more, or less, related to policy-driven actions. For example, analysts cannot differentiate among potentially conflicting processes since countries like Cambodia and Ghana identified as having "weak nutrition governance" were still rated as "on track" to meet global nutrition goals, while a state like Burkina Faso was reported as making "no progress" towards such goals despite a 'strong' rating for nutrition governance. Furthermore, being based on official data collated at national level there is limited change across years.

Hunger and Nutrition Commitment Index (HANCI)
The HANCI (sometimes shortened in the literature to the Nutrition Commitment 8 Index, or NCI) was developed to rank governments on their political commitment to tackling undernutrition while seeking to measure what governments achieve and where they fail. This index ranked 45 countries according to 22 indicators grouped under 3 themes: public expenditure, policies and programmes, and legislative agendas. Like the WHO metric of nutrition governance, HANCI decouples its indicators from outcomes; that is, it focuses on measures of 'input' (such as spending on nutrition) rather than on 'outcomes' (such as levels of stunting). Rather than allocating equal weight to all indicators, the authors give greater weights (subjectively) to aspects of governance that appear to have more relevance to nutrition than others -such as granting implementation into law of the International Code of Marketing of Breastmilk Substitutes a higher weight than simply having a dedicated line for nutrition in the national budget. (15) The 2014 version of HANCI rated Peru, Malawi and Guatemala as the nations with greatest commitment to nutrition; Cambodia, Pakistan, Afghanistan, Yemen and the DRC all ranked poorly (as with the WHO index) as having "very low commitment." (19) This shows close accord with the WHO rankings, but again suffers the problem of lack of granularity at sub-national level. As the authors themselves have noted, while such metrics offer insight into "the general quality of public administration in a country", this offers little that is specific to nutrition or to the quality of the process of implementation of policies where they exist. (15) Political Commitment (for Nutrition) Rapid Assessment Tool The 'tool' developed by Fox et al. (2015) was designed not to rank countries, but to offer deeper insight into individual countries' depth of political commitment to food security and nutrition.(20) The authors emphasize that "political commitment for food and nutrition is rarely adequately defined or empirically measured", leading to 9 recommended questions that could be used to assess national commitment to nutrition. These cover not just political level commitment, but also policies, organizational structures, funding for programmes, and the legal and regulatory environment. The draft tool was piloted in 10 low income countries in 2012, and revised in 2013. Some questions are yes/no, with a yes worth a score of 1; other questions allow for a range of relative responses from 1 through 10. For the latter questions (on a scale), a response of 7 or higher was allocated a 1, and for questions relating to budgets (with a 0 to 3), a response of 3 was assigned a 1. The total score possible is 51.
In the test with 10 countries, the Philippines and Columbia were among the highest rated, while Vietnam and Bangladesh fell towards the bottom of the ranking. (20) Unlike the previous two example metrics, however, this approach allows for discrimination among the various elements that make up the final score, making it more useful to any analysis of the elements of governance that are stronger or weaker. Nevertheless, it still relies mainly on an assessment of information that exists at national level that may or may not relate directly to a government's ability to implement pro-nutrition policies and programmes.
The alternative approach proposed in this paper focuses on an empirical compilation of information derived through in-person surveys with professionals and other stakeholders holding positions and responsibilities for achieving specific nutrition goals. In this sense, the nutrition governance score is more directly based on what people know, think and do in relation to their responsibilities, from sub-national level down to field level. The next section elaborates on the approach used.

10
The study was conducted as part of an ongoing nationally representative prospective longitudinal study across 21 districts and three agro ecological zones of Nepal. A total of 520 participants, who included government and non-government officials, were selected purposively. The questionnaire comprised of a set of multiple interrelated Likert-scale items that were used to gather data on selfperceived governing practices within respondents' scope of roles and responsibilities. Each participant answered one of four options: "Strongly Agree", "Agree", "Disagree", and "Strongly disagree" and for some questions, "Don't Know" or "Not Applicable". The scale was reconstructed to "Strongly Agree", "Agree" and "Disagree", the latter indicating the respondent selected either "Strongly disagree" or "Disagree". The study was approved by Tufts Social Behavioral and Educational Research Board (SBER) and Nepal Health Research Council (NHRC).
We identified six domains from the data using principal components analysis (PCA).
PCA is a data reduction procedure for transforming a large number of variables into a smaller number of uncorrelated (orthogonal) factors, the principal components. The standardized NGI score ranges from 0 to 100 with higher scores indicating better nutrition governance.

Validity and Internal consistency
Cronbach's alpha coefficient was calculated to determine internal consistency.
Reliability tests assume that the unidimensionality assumption holds therefore these tests were conducted by their domains as opposed to across all domains.
Internal consistency describes the extent to which all items in a test measure the same underlying trait (22). These estimates range from -1 to +1, with higher absolute values indicating higher reliability. It's also a test for measurement error: as reliability increases, measurement error decreases. A criterion of 0.7 and above is generally considered acceptable. (23) Applying the same, only two domains (Domain 1 and 2) are reliable, domains 3 and 4 are borderline reliable and 5 and 6 are unreliable (Table 1). Low alpha values 13 imply that the domains require re-assessment to either increase the number of items or replace them altogether.

Construct validity
The construct validity for the NGI was examined using Confirmatory Factor Analysis (CFA). This technique was used to verify the factor structure imposed on the data, that is, whether the six domains obtained were a good fit. Table 2  is recognized as the cut-off for a good fit (24). Overall, the model is a good fit for the data. Whereas the model didn't pass the chisquare and Bentler's comparative fit index, it passed all the other indices thus confirming the factor structure imposed. Nevertheless, based on relatively poor reliability for some domains and failure to meet some fit indices, we recommend reassessing the items in domains with low alpha values to improve overall model fit.  Table 3 shows percentage distribution of respondents categorized by NGI rank.

Ranking Sub-national sectors
The Health sector showed the most improvement in mean NGI score followed by the Agriculture sector. About fifty percent (50%) of respondents from the health sector were more likely to have a high score (4 or 5) compared to other sectors (OR 0.52, 95% CI 0.36 -0.74). This is an important finding because the Health sector is at the forefront of all health related and some agriculturally relevant activities including operationalizing and advocating for maternal and child nutrition programs.

Ranking the Multi-Sector Nutrition Plan (MSNP)
The Multi-Sector Nutrition Plan (MSNP) is an initiative implemented by the Nepal government in collaboration with various international partners including UNICEF, USAID and the World Bank. They altogether aim to improve the nutritional status in Nepal, especially in mothers and children. Two districts, Jumla and Nawalparasi, were in the MSNP catchment area at the time of data collection. Only 31% of respondents exposed to MSNP initiatives were ranked in the highest scoring categories (4 and 5) compared to 41% not exposed (OR 1.57 95% CI 0.844 -2.90). Furthermore, across time, the mean NGI score was lower for respondents from these districts. This finding serves 1) to show MSNP policy makers that there are areas of improvement to be explored and 2) to avail concerned parties a tool against which they can measure MSNP performance.

Ranking staff trained in nutrition related activities
Building local capacity by putting an emphasis on attending nutrition courses and training improved nutrition governance (OR 1.40, 95% CI 0.79-2.48). Training at the implementation level raises more awareness of the nutrition issues. Trained staff have a higher likelihood to understand the effectiveness of inter-sectoral and crosssectoral co-ordination. Similarly, the more extensive the work experience, the higher the likelihood of performing better on the job and the greater the score on the NGI.

Discussion and Limitations of the Nutrition Governance Index
The newly created index is unique in that it assesses nutrition governance at the sub-national level, which is of great potential value to both researchers and policymakers that are keen on gaining an improved understanding of how complex multi-institutional policies on paper are translated into practice. The index assesses six different domains of nutrition governance; i) understanding nutrition and responsibilities, ii) collaboration within and across offices, iii) access to financial 16 resources, iv) nutrition leadership, v) capacity and vi) coordination and support across and between sectors.
The tool allows for assessment of sub-national groups and individuals on how they perceive implementation of nutrition policies within their official duties. A high score on the NGI may not necessarily mean a high score on all of the domains it constitutes thus sub-national levels should be reviewed by NGI domain in order to gain better insight into their performance.
The same questionnaire was used for each respondent regardless of management roles and administrative level. Whereas respondents differed in their responsibilities, the items were broad enough to be relevant across levels yet specific enough to apply to singular roles. Table 4 below shows percentage distribution of responses at the four management levels.
An analysis of the response pattern showed that there was agreement across all administrative levels with the exception of Item 15, which inquired about access to budgetary resources. The disproportionately low percentage of respondents who strongly disagreed at the lower levels of management could be explained in two ways; 1) Their roles do not involve handling budgets so the question is irrelevant 2) They genuinely strongly disagree with the item. All the 13 respondents who opted for either the "Don't Know" or "Not applicable" categories were from the Ward level which points us to the first explanation. A similar analysis of missing value patterns showed that Item 3 and Item 16 may not be applicable to lower levels of management, therefore rating them using these items reflects an unfair sense of judgement in their overall NGI score. Results therefore should be interpreted with caution.
The sampling strategy used for these analyses was purposive in design. Although effective in achieving meaningful results (when well executed), this strategy can be problematic when making inferences. Inferential statistics allow for generalization of results to a much larger population using sampled data. Whereas we sampled from across the three major agro-ecological regions in Nepal, participants were sampled non-probabilistically on a smaller scale within each zone thus there would have been a selection bias if perceptions of governing practices captured in this study differ from participants' not included in the study. Consequently, the results obtained in this study may not be applicable nationwide or on a much greater scope due to limited representation.
The process of item selection was iterative in practice. The first round of data collection was qualitative because we took an exploratory approach that allowed us to gain a broad understanding of governing practices on the ground. At each subsequent round, items were refined using PCA, which resulted in replacing uncorrelated items with new, relevant items under each domain. We used the fourth round of data collection in this paper, to construct the nutrition governance indicator which poses the following limitations; 1) some of the items were not asked in previous rounds therefore the outcome of the NGI is dependent on the round of data collection 2) it may be problematic to compare trends in NGI across rounds.
As aforementioned, the first principal component accounts for the most variation may not exist. However, the aim was to obtain a single index representing each domain therefore this approach was necessary. In addition, since we only considered the factor with the highest variation, there won't be significant benefit in adding higher-order factors as they tend to explain much smaller proportions of variation.

Conclusions
The findings from the multi-year PoSHAN policy survey showed that different groups/institutions performed differently against the index. The health sector, in particular, did well compared to six other sectors. Additional groups that performed well include respondents that were trained in nutrition-related activities and education, respondents that had over 2 years of work experience and officials from the district administrative level. We identified MSNP exposure as a target for improvement. This may be regarded as a significant finding to improve service delivery in these districts.

Availability of data and material
The datasets generated and/or analyzed during the current study are not publicly available yet but are available from the corresponding author on reasonable request.  *0=Disagree 0.75 = Agree 1 =Strongly agree

Tables
We attributed an arbitrary penalty of 0.25 to "Agree" responses in order to account for the loss of certainty. This ensured that these responses carried less weight in the PCA procedure compared to "Strongly agree" responses. The lesser degree in agreement was represented in the factor scores created resulting in more robust principal components. A larger penalty (0.5) would insinuate that a slight shift from strong agreement reduces the positive effect by half -an assumption too strong to 28 make Figures Figure 1 Median Nutrition Governance Index score by respondent groups