This assessment has confirmed that the HSS program can be a vehicle for building UNICEF graduates capabilities to apply HSS actions to meet UNICEFs strategic goals of ending preventable maternal, newborn and child deaths and promoting the health and development of all children and reducing inequities in health.
At an operational level, the HSS program has clear strengths including: program content, structure and mode of delivery that provides a mix of theory and practice, engaging with diverse visuals, audios and presenters.
While HSS program graduates reported increased confidence with HSS actions and changes in work practices, the HSS program faces key contextual challenges: graduate workload and work foci; limited opportunity to apply HSS approaches; limited HSS examples; and varied supportive workplace environments. These challenges require us to reflect upon the fit between the HSS program learnings and UNICEF’s enabling or not enabling workplace environment for its graduates to apply HSS actions. These findings are in line with the formative evaluation findings of UNICEFs programming in HSS (ITAD, 2019).
The following section contextualises and discusses the assessment findings with regard to two key domains: 1) Health Systems Strengthening policy context; 2) Transformative professional development transfer and effects and UNICEF’s Workplace Environment; and 3) the hypothesised Pilot HSS Program logic model. Reflections upon the assessment strengths and limtation are also discussed.
Health Systems Strenghtening Policy Context
The Assessment has revealed that the HSS program has contributed to increased UNICEF graduates’ HSS knowledge, skills, confidence and capabilities to contribute to HSS in their contexts. While these findings are indicators of the HSS program effectiveness, and provide evidence to suggest that a HSS paradigm shift is occuring across UNICEF and its graduates, it cannot be attributed solely to the HSS program. Over the 2017 HSS program period UNICEF initiated multiple other initiatives to advance HSS which also involved UNICEF gradutaes, thus potentially contributing to increased awareness and understandings about HSS approaches and actions. The Assessment of the HSS program needs to view its graduates within an ecosystem framework – where the organisation and interaction of the HSS initiatives in the system are as important as the initiatives themselves.
Transformative professional development transfer and effects and UNICEF’s Work Environment
The Assessment findings revealed that graduates Senior Manager interviews had mixed impressions of graduates applying HSS actions upon their return to their workplaces. These findings require reflection for several reasons.
Firstly, professional development programs are recognised as a means and not an endpoint and part of a transformative change process, involving a process of ongoing change, adoption, implementation, dissemination, and sustainability of the innovation into practice (Frenk et al., 2010)
Secondly, based upon the reported challenges to applying HSS actions and the findings that UNICEF graduates requested more ongoing support, experiences and opportunities to applying HSS actions, it is important to reflect upon evidence about factors that influence learning transfer and effects.
Learning transfer is recognised as a complex phenomenon (Thalheimer, 2018) which presupposes a series of stages and is based upon:
- Attendance- UNICEF graduates have attended the HSS program
- Completion- UNICEF graduates having completed the HSS program
- Knowledge recitation and retention – UNICEF graduates can recite and have retained HSS knowledge
- Decision-making competence – UNICEF graduates know what to do with HSS actions
- Task competence – UNICEF graduates use HSS knowledge
- Transfer – UNICEF graduates use HSS actions; and
- Effects of Transfer - training influences outcomes of UNICEF graduates work practices.
While a substantial literature exists (Burke and Hutchins, 2007; Grossman and Salas, 2011; Tonhäuser and Büker L. 2016) from multiple disciplines (management, training, professional development, adult learning and psychology) on learning program transfer and its effects, three key conceptual factors influence learning transfer: 1) Graduate/individual characteristics; 2) Training program design and delivery; and 3) Work environment. These three learning and effects factors will now be briefly discussed in relation to the HSS program.
UNICEF graduates/individual characteristics: related to HSS program the following issues need reflection: graduates needs and goals related to HSS in their work; graduates competency to apply HSS; graduates motivations (intrinsic vs extrinsic) for applying HSS approaches; graduates career aspirations; graduates personality traits (e.g., openness to new experiences, conscientiousness), and graduates perceived value of HSS training.
HSS learning program design- related to HSS program the following issues need reflection: HSS training content, relevance and alignment to graduates work practices; HSS training presenter quality and engagement; provision of instructional strategies and methods to facilitate transfer of HSS approaches into practice (e.g., providing opportunities for adequate practice, feedback, reinforcement of applying HSS; opportunities for repeated practice of HSS approaches in “real” work practices at country level; active learning – involving trainees in course materials delivery; provision of self-management strategies to equip graduates to transfer HSS learning into practice.
UNICEF workplace environment– related to HSS program the following issues need reflection: the strategic link of the HSS training to UNICEF graduates work practices; the extent to which the learning transfer climate is supportive of HSS via management, supervisory and peer support and accountability system; extent of opportunities to use HSS knowledge and skills in work practices.
Hypothesised Pilot HSS Program logic Model
The hypothesised program logic (Box 4) underpinning the HSS program was largely confirmed. The hypothesised contextual factors were confirmed and expanded upon and remain key issues needing to be recognised including: UNICEF staff workload; limited opportunities to apply HSS actions; limited HSS examples; and variable support to apply HSS actions.
The hypothesised inputs and activities necessary for the HSS Training Program to work were also largely confirmed – given the positive comments on the program content, structure and mode of delivery.
The assessment has expanded the hypothesised HSS Program outputs, as UNICEF graduates reported that the HSS training had not only improved their knowledge, skills and confidence taking HSS actions and using HSS tools, but changed their thinking and work practices (e.g., increased systems thinking and using of health system-based approaches).
The short assessment time frame limits our ability to comment on intermediate and long-term Outcomes achieved so far. However, the assessment has revealed that the HSS Training Program is contributing to transformative changes (change in HSS mindsets and practices) at the individual and organizational level.
Assessment Strengths and Limitations
The evaluator recognised that the HSS program developers (Nossal Institute for Global Health and UNICEF) and funded by UNICEF were the primary intended users of the assessment and hence required useful, specific and contextualised evidence to inform future HSS programs. A participatory and realist assessment approach guided the assessment methods, analyses and interpretations. The assessment findings reveal that the assessment approach facilitated a working partnership between the HSS program developers and the evaluator, leading to increased assessment utilisation and fostering a culture of learning between all.
We acknowledge that despite the assessment approaches, the assessment had limitations. The sample sizes in the assessment activities were small and not representative of the entire UNICEF HSS graduate cohort (n=83) nor of UNICEF graduates who participated in other assessment activities by UNICEF and / or HSS program developers- whose perceptions of the HSS program were more positive. Despite inviting via email all graduates to complete online surveys, participate in interviews, the evaluator was unable to involve all trainees in the assessment, notwithstanding many invitations and email reminder attempts. We also recognise that from the outset the assessment had a particular focus upon UNICEF graduates applying HSS actions, while the learning program had a broader HSS focus.
We also recognise that being a pilot the HSS program has evolved significantly since 2017 and the assessment is therefore limited to the 2017 graduates not those who have completed the course in subsequent years.