Facilitators and Successes to Implementation
Data collected during the pilot phase of Youth FORWARD’s scale-out into GIZ’s youth employment scheme demonstrated the feasibility and appropriateness of using a CTA to deliver an evidence-based mental health intervention like the YRI. Scale-out study data will prioritize analyses on YRI clinical effectiveness as well as the added benefit of the YRI when combined with the ENTR; process components including reach and access of the intervention, organizational culture and leadership, and fidelity to evidence-based practice; and a costing analysis to examine cost-effectiveness and return on investment of YRI and ENTR integration.
Impact
Pilot data demonstrated that the YRI was considered culturally appropriate and relevant (manuscript submitted for publication, Desrosiers et. al.). The YRI underwent rigorous adaptation in-country to ensure cultural fit and appropriateness through the use of local parables, relevant language, and skills useful to the Sierra Leonean context.23 After the pilot, facilitators spoke to the inner context of the YRI: training and supervision, embeddedness in local communities, and their perceptions regarding fit, relevance, and compatibility of the YRI+ENTR. For the scale-out, surveys measured appropriateness to determine YRI fit with cultural and personal values and its perceived relevance and effectiveness in addressing problems at all stakeholder levels, along with professional fit at the provider and organizational levels. The quantitative assessment battery for youth participants followed the WHO process of translation and adaptation of instruments to ensure accurate cultural translation and adaptation.24
The uptake of the YRI by participants was well-documented in qualitative interviews with YRI interventionists after the pilot. During interviews, facilitators provided many examples of youth using YRI activities and strategies to solve everyday problems, providing evidence for the youth’s engagement with the YRI and the success of the intervention. Many YRI facilitators were also using the intervention to improve their own lives (manuscript submitted for publication, Desrosiers et. al.). For the scale-out, adoption will be further measured via surveys with youth, facilitators, and agency leaders about the uptake of the YRI and whether stakeholders discuss, recommend and refer others to become involved.
Effectiveness data on the broader outcomes of the Hybrid Type II Effectiveness Implementation trial design is still being analyzed, but initial impact of the intervention is documented in the outcomes of the feasibility pilot which included a randomization feature. That study indicated that YRI results demonstrated significant improvements in mental health outcomes among males in the pilot. Compared with the control group, males who received the YRI in addition to the ENTR reported significant improvements in post-traumatic stress symptoms and emotion regulation. These improvements in daily functioning may enhance effectiveness for youth in employment settings (manuscript submitted for publication, Desrosiers et. al.). The results for female participants were less clear. Initial findings from the pilot also suggested that youth receiving both YRI and ENTR were able to shift their wage-earning hours from self-employment to more steady wage-paying work, indicating that YRI may be associated with greater labor market impact (manuscript in preparation, Akram et. al.). Although trends in the pilot showed improvement for youth, the fully-powered effectiveness trial will further measure and examine these results.
Local District Youth Councils
Youth councils established by the Sierra Leonean government in 2012 are present in each district, with councils spanning from the district to the village level. These councils are positioned to work across relevant stakeholder groups to amplify youth presence in the policy making process, elevate youth perspectives, and illuminate challenges facing youth in Sierra Leone.25 As part of its youth employment platform, GIZ works closely with youth councils to support program coordination and implementation. Leveraging this established relationship was an asset for the YRI team given a limited presence in the more rural study districts. As a result, YRI experts and study research assistants relied heavily on youth council leadership to engage, identify and contact youth. These councils represent an important innovation factor that supported YRI and ENTR implementation with the potential to influence long-term sustainability.26
Collaborative Team Approach
The CTA provided a bridging factor that allowed researchers from Boston College and the University of Georgia to learn from local stakeholders with significant programming expertise who are embedded in Sierra Leonean communities. In addition, the CTA provided opportunities for Sierra Leonean agency members to learn from each other’s experience in mental health and youth programming. One seed team member appreciated how the CTA provided specific implementation guidance: “(The CTA) gives the team a road map to best practice. Clarity within the organization leaves less for assumption and allows all partners involved…to make the best decisions and strengthen the core of the program” (Male seed team member).
Two interviewed agency leaders expressed that the CTA allowed for sharing of knowledge and benefitting from the expertise of each member. As one agency leader described, “one organization might be an expert in one thing and the other might be an expert in the other thing. We're able to kind of meet and discuss things and strengthen that model that we want to use for greater achievement in the communities. And that seems to be working really well” (Senior Program Manager, Restless Development). During regular meetings, both agency leaders appreciated how members across teams would work together to problem solve as issues arose during implementation. Communication and mission alignment improved throughout the process. Both agency leaders explained how multiple budgets and modes of operation were challenging at first, but ultimately led to increased collaboration. According to one agency leader, “it's (the CTA) has given us a lot more insight to be able to have more team players in a small pitch. So that we, we coordinate more and, and we share a lot more information...which has been very good” (Program Manager, Caritas).
Seed Team Supervision
In supervision sessions, seed team members provided guidance to facilitators in a manner that empowered self-reflection and self-monitoring. One supervisor described how he would begin supervision sessions by asking the facilitator what his own perceptions were on his delivery of the YRI. This approach allowed for critical thinking and growth while maintaining fidelity of the intervention. YRI sessions were audiotaped for review using a fidelity monitoring checklist which guided supervision sessions with facilitators. When possible, a seed team member directly observed YRI sessions, using the fidelity guide to lead structured supervision with facilitators immediately after session delivery.
All interviewed facilitators expressed appreciation for supervision and provided examples of how group and individual meetings with supervisors assisted with problem solving and content challenges. One facilitator described her experience with supervision:
It is very helpful. My co-facilitator and I will be busy with other things in the manual, and maybe one session is not well explained. (Our supervisor) will be there to observe and she will tell us that you have to explain this area…she will tell us to probe very well so that the participants will get a better understanding. Supervision empowers you to be a good worker…it will empower you to become very self-sufficient in your job (female facilitator).
Scale-Out Data
A return-on-investment analysis and a cost-effectiveness analysis will be calculated with the goal of informing program policymakers in Sierra Leone and similar LMICs what it would cost to deliver a mental health intervention through an employment program platform. The analysis will be performed for the total cost and benefits across the scale-out study time period to show the projected value of scaling up the YRI in relation to the cost of doing so. Benefit estimates will come from surveys that capture mental health and income data from participants. Cost data will be collected through a dedicated cost data capture procedure to provide effectiveness estimates of the YRI including cost-per-participant and cost-per-outcome.
Surveys administered at all stakeholder levels collect information on barriers and facilitators of reach and access with a focus on acceptability, appropriateness, and feasibility. Surveys with YRI facilitators and agency leaders contain additional domains on leadership, organizational culture and sustainability. Questions prioritized personal feelings at work and the perceived work environment. To measure important characteristics for facilitating change processes in organizations, YRI facilitators and agency leaders provided perceptions on the degree to which implementation leadership is proactive, knowledgeable, supportive, perseverant, and skillful.
The implementation impact evaluation delivered in the scale-out measures the CTA as a strategy for sustaining the YRI in terms of delivery (e.g. feasibility, fidelity), context (e.g. leadership, climate), capacity (e.g. funding, workforce), and process (e.g. decision making, partnerships).27 Mixed methods approaches, which include fidelity data, surveys, key informant interviews, and focus groups, will identify barriers and facilitators to effective implementation and sustainment. Additionally, qualitative interviews will provide insight into the extent to which the YRI is maintained or institutionalized within GIZ as part of ongoing programming.
Finally, the scale-out study will measure the degree to which youth are able to access YRI sessions (measured by session attendance and completion rates) and derive clinical benefit from the YRI. Ongoing monitoring of the intervention throughout the scale-out will assess fidelity to evidence-based YRI practices. All YRI facilitators completed a session information sheet to document youth attendance, length of the session, and any issues with the delivery or the session.
Challenges in Implementation
The outer context, including contextual limitations, macro-level regulation, and intervention logistics and fidelity, were identified as challenges and barriers to YRI delivery and scale-out. Sierra Leone grapples with a health system overtaxed by war, disease outbreaks and natural disasters; weak governance structures and limited policy supports; a fractured funding environment; prolonged elections that threaten project implementation; and, a fragile context that contributes to reticence from donors.
Contextual Limitations
A major challenge arose in 2017 when the elections resulted in a new government, creating uncertainty with government engagement and a smooth transition of power. As the incoming administration appointed new leadership, relationships initially built when designing the study had to be rebuilt. Another challenge arose when the World Bank, Youth FORWARD’s original partner, changed leadership and set back their Youth Employment Program two years for re-design. As a result, PIs and the Scale-Out Study Team had to identify an alternative implementation partner. The team networked across multiple sectors and eventually partnered with GIZ. While GIZ’s programming provides a strong platform from which to deliver the YRI, a partnership with GIZ presented new challenges. Other adaptations were necessary as GIZ rolled out their employment programming, including the contracting delays and unanticipated acceleration of GIZ’s timeline during study scale-out. Overall, understanding and incorporating the Youth FORWARD research activities within GIZ’s more business-oriented style of operating has required ongoing negotiation and adaptations with implications for implementation research.
Macro-Level Regulation
Youth FORWARD is part of a network of hubs delivering interventions in low-resource settings. Tremendous challenges lie in complying with regulatory processes, sometimes at odds with field realities. Current project oversight mimicked regulatory processes utilized in drug trials and relied on an external and independent Data and Safety Monitoring Board with a fixed meeting schedule and routine study monitoring from a contracted clinical research associate. This level of oversight was in place to ensure study compliance and participant safety but led to study delays given the fixed nature of the review and approval processes. As a result, the need to align study procedures across several ethical review bodies resulted in increased study costs and implementation delays.
Intervention Logistics and Fidelity
Facilitator surveys revealed a number of implementation challenges while delivering the YRI during the pilot (See Table 4). For example, facilitators described issues with providing youth with the travel allowance upfront “because if you give them all the money, they [participants] would just go and never come back” (male facilitator). Facilitators also explained an additional consequence of giving the allowance upfront; non-participants often showed up posing as participants, only to accept the travel allowance and disappear. Others faced delays due to participants arriving late or not coming at all. Many requested more time dedicated to training and to delivering each individual module. Finally, facilitators described language and educational barriers for youth, as a result of operating in rural, remote parts of Sierra Leone (manuscript submitted for publication, Desrosiers et. al.).
Worsening intervention fidelity over time is another obstacle to overall effectiveness. Traditionally, implementation models rely heavily on remote expertise and do not prioritize building local expertise. Organizations in Sierra Leone experience high levels of staff turnover without consistent funding to provide employees with long-term employment contracts. One facilitator noted: “My organization told us that [the job is a] five-month contract. And when we came… all of a sudden they said six weeks so I was not okay” (male facilitator). This culture is not conducive to maintaining institutional knowledge or sustaining intervention delivery over time. Agency leaders at local organizations confirmed challenges in retaining newly trained staff when projects were short-term, citing funding structures as the chief barrier (Program Manager, Caritas). When delivering a program like Youth FORWARD that utilizes a CTA, considerable time and resources must be spent developing strategies to overcome this challenge.