This multiple case mixed-methods study will further refine and test the prototype Agenda Gap intervention with additional groups of youth, aged 15-18, across three sites in British Columbia and Alberta, Canada. Study sites have been selected based on existing community-based partnerships and to ensure the representation of diverse youth identities including urban indigenous youth, racialized and newcomer youth, and youth who experience structural vulnerabilities. This protocol has been written using the Standard Protocol Items: Recommendations for Interventional Trials guidelines (SPIRIT, see Additional File 1 for SPIRIT checklist).
Study Methods
This study is grounded in a realist evaluation approach and depicted in Figure 1. Informed by realist philosophy, this theory-driven methodology supports the identification of key context-mechanism-outcome configurations (CMOs) underpinning intervention activities to inform comprehensive insights into how the intervention works, for whom, and in what context. In realist evaluation, Context refers to the conditions (encompassing populations and social/structural conditions) that influence intervention outcomes. Generative mechanisms describe the processes (e.g., social/psychological drivers) that contribute to observed outcomes. The outcomes (intended and unintended) result from the interaction between contexts and mechanisms. This evaluation approach will improve our understanding of how the Agenda Gap intervention operates, thereby, supporting the testing and refinement of theories undergirding our intervention and enabling us to make meaningful contributions to youth mental health research agendas at multiple levels (e.g., increased evaluation/research capacity among mentors and youth collaborator participants; new insights on a methodological approach that can inform broader evaluation practice; increased understanding of functional mechanisms and relevant concepts that can be applied to other programs/contexts; identification of the necessary contextual features for effective program functioning) [41].
Realist Evaluation Stages
Realist evaluation is comprised of five stages [42], which are detailed for our study below (Fig. 2).
Stage 1 – Initial Intervention Theory
As per realist evaluation methodology, our research process began with a review – in our case, this took the form of an environmental scan to identify existing resources and evidence informing practices to equip youth for meaningful policy engagement [10]. This review, along with theories of youth development, mental health promotion, and liberation psychology identified above, have informed our “Initial Intervention Theory” (IIT) (Fig.3). Guided by the realist evaluation approach, this IIT will be tested and refined though our study activities. Specifically, the IIT as well as the various elements of the intervention itself will be described through a series of “program theories”, or “if/then” statements that are intended to explain the causal process believed to underpin the intervention activity effects. For example, we hypothesize that financial compensation for our youth collaborators is a key element that will contribute to intervention effects. As such, we have developed and will test this as a program theory framed as, “if youth who have historically felt that their voices are not heard [Context] are financially compensated for their collaboration in the Agenda Gap intervention [Mechanism], then they will feel valued, committed and motivated. This will lead to reduced barriers to participation, sense of co-ownership and sustained involvement [Outcome].”
Our IIT builds on the Theory of Socio-political Development [27], which describes youth policy engagement as a product of social awareness and action, moderated by perceived agency and existing opportunity. We have advanced this theory and suggest that our Agenda Gap intervention will build critical consciousness – or the ability to recognize and act on issues of inequities. In turn, we hypothesize that this action will generate an enhanced sense of agency amongst participants, contributing to further opportunities for meaningful engagement within policy contexts – with mental health promoting impacts across the socioecological domains. For example, this intervention holds the potential to influence study participants directly, by equipping them with new knowledge and skills pertaining to mental health and policy. Further, through the engagement process, we anticipate impacts on youth, policymakers and others living in communities where policy change efforts are initiated.
Stage 2 – Study Design and Recruitment
We are using a multiple case study design across three sites, with one case consisting of each site’s team (e.g., mentors, youth collaborators, site organization) and context. Each case will enable cross-case comparisons to identify common and distinct CMOs and their configurations [43-45]. This will allow us to test the intervention and program theories across diverse contexts and youth populations, with respect to mechanisms theorized to produce expected outcomes.
Mentor Recruitment and Training.
This study will begin with the identification and training of youth and adult mentors across three sites. Mentors will be identified through our partner networks (i.e., community organizations, non-profits, health authorities, School Boards) and bring experience in working with youth and of engaging in policy change initiatives. In future iterations, additional partnerships will be developed to support intervention scale-up. Our train-the-trainer model will involve approximately 12 hours of training delivered by a Community Engagement Specialist. This training will equip mentors to facilitate engagement with youth collaborators, while also building leadership skills and growing local capacity for intervention sustainability. Mentor training will include how to:
- promote social and cultural safety among youth from diverse contexts and backgrounds
- model or bring lived expertise of policy engagement or advocacy
- enable shared decision making with youth for meaningful youth engagement
- incorporate principles of trauma and violence informed practice
- be inclusive of diverse learning needs
To ensure mentors have the needed level of substantive knowledge to support implementation of the intervention, a Facilitator’s Guide has been developed and contains step-by-step details of the curriculum content and strategies. This approach was selected over a manualized approach to intervention delivery to promote flexibility, while still providing consistent guidance across sites. As mentors will be an important data source, all mentors will provide written informed consent prior to participating in study activities.
Youth Collaborator Recruitment.
A particularly novel element of our approach is that we are committed to engaging a diversity of youth, aged 15-18 years, from structurally vulnerable communities. Policy participation for such youth has remained largely inaccessible due to intersections of health and social inequities that operate to exclude certain groups from contributing (i.e., youth who have experience with mental health services, live in poverty, are in the care of the child welfare system, or who are Indigenous, newcomer/immigrants, or LGBTQ2+). Our equity-oriented approach and strong community and policy partnerships will enable us to engage our youth participants to better reflect the lived experiences of youth who have some of the greatest mental health needs [46, 47]. By working with youth who have historically not had opportunities for engagement and leveraging our partner networks to influence policy, we can contribute to more equitable contexts, which are supportive of improved mental health outcomes for all. Recruitment will be facilitated by our community partners, with interested youth encouraged to contact the research team. All applicants will be interviewed to assess eligibility and interest in becoming a youth collaborator on the study as well as to ensure the inclusion of diverse youth identities (e.g., age, gender, social location). A total of 10-15 youth will be selected (after obtaining informed consent) as collaborators for each of the three sites. This position will be a paid position to appropriately acknowledge the expertise and contributions that these youth will make to the intervention and research process.
Our project also includes ongoing partnerships with adults who interact with and provide services to youth (e.g., youth workers, policymakers, educators, and health and social service providers). This engagement builds relationships through which knowledge and perspectives are shared; and understandings, beliefs, and behaviors shifted to contribute to more equitable and mentally healthy communities [20]. For example, one of the critiques in the youth engagement literature is that youth tend to be the sole intervention target. Instead, efforts need to include strategies to support relevant adults in appreciating and recognizing the various ways that youth are already “engaging” (which may differ from adults’ preconceived notions) and shift systems to be more responsive to diverse forms of engagement [48].
Youth and their communities, adult and youth mentors, and policymakers, will be involved in all phases of intervention refinement, implementation and testing through direct engagement in project activities and membership on Advisory Committees that guide project processes, partner expansion, recruitment, curriculum refinements, youth mentorship, youth meetings with policymakers, and other youth identified activities.
Stage 3 – Data Collection Methods
A mixed methods approach to data collection will allow for comprehensive measurement/tracking of inputs (i.e., suggested intervention refinements), implementation process and intervention outcomes. Quantitative and qualitative data will support the characterization of CMOs and community perceptions of intervention impact.
Quantitative surveys will be used to gather data pre- and post-intervention to examine the impact of the intervention at multiple levels. Specifically, these surveys will include demographic questions to explore how different factors may be related to outcome patterns, an assessment of knowledge gains, standardized tools measuring key hypothesized outcomes (i.e., resilience, self-efficacy, and critical consciousness) and will collect indicators or metrics of policy engagement and change.
Standardized Quantitative Tools include:
- Child and Youth Resilience Measure (CYRM-12). The CYRM-12 is a brief self-report scale that measures youth resilience. It has been used with youth in a variety of contexts and has good content validity and reliability (Cronbach’s alpha=0.840) [49].
- General Self-Efficacy (GSE). This 10- item scale has been used widely among youth and adult populations to assess self-beliefs for dealing with various life challenges. GSE has been demonstrated to measure a unitary concept with the good psychometric properties [50, 51].
- Critical Consciousness Scale (CCS). The CCS is a validated, 22-item scale that measures the capacity of structurally vulnerable youth to reflect on and analyze their social conditions and engage in action for change [52].
In addition to standardized tools, we will develop items to quantitatively assess mental health promotion literacy and indicators of positive mental health.
Given the complexity of policy influence and the challenge of linking intervention activities with policy change, in addition to the ideal “hoped-for” policy impacts, we will collect data across a continuum of policy influence processes (e.g., precursors to policy change) [53]. Additionally, data on the economic inputs or costs associated with intervention delivery will be tracked for future feasibility and sustainability considerations.
Qualitative realist interview methods will be used post-intervention to collect varying perspectives from youth, mentors, community partners and other stakeholders about the intervention, which will complement structured surveys in verifying and refining the program theories. Qualitative realist interviews are a key source of theory refinement and discovery of underlying mechanisms within realist evaluation. They are also a valuable tool for surfacing youth and community voices and for assessing, from participant perspectives, the nuanced ways in which the intervention contributes to policy change and risk and protective factors, at the individual through population levels [55]. Project logs will also be maintained by the research team to capture observations of community resources and dynamics, including those unspoken during intervention sessions, and to inform intervention refinements.
Stage 4 – Data Analyses
Quantitative survey data will be entered into SPSS to facilitate statistical analyses. Descriptive statistics will be calculated for each question to determine the effectiveness of the intervention for addressing each of the constructs represented in the survey. Paired t-tests will be used to assess pre- and post-intervention impacts (e.g., does resilience increase post-intervention?). When sample sizes permit, ANOVA will be employed to examine mean scores and change across sites to gain an understanding of impact in varying contexts.
All qualitative data (i.e., individual interviews and project log data) will be transcribed and uploaded to NVivo 12 to facilitate thematic analyses that will generate high-level themes, representing findings within and across transcripts [55]. During later intervention engagement sessions, youth collaborators will participate in collaborative analysis, an iterative process with the research team, who will produce summaries of identified themes and engage youth collaborators to reflect and extend interpretations. Triangulation of data sources (quantitative, qualitative and cross-site) will be undertaken to: a) gain a comprehensive understanding of the impact of the intervention; b) illuminate CMOs; c) examine regularities or areas of contradiction between data sources; and d) as an approach to analytic and procedural rigor to support the validity of findings [56]. Analyses will be concurrent and ongoing throughout the study process to support rich and detailed analyses that are responsive to emerging understandings [57].
Additionally, an intersectionality lens will inform analyses to assess outcomes by contextual and identity factors. This is consistent with realist evaluation, which seeks to represent the multiple and intersecting ways that sex, gender, socioeconomic status, health status, ethnicity, and other factors shape participants’ experiences with mental health and responses to the Agenda Gap intervention. Data monitoring and management will be conducted by the core research team. At least two researchers or research manager will ensure the data quality.
Stage 5 – Synthesis
Final in-case analyses and comparisons with original IIT and program theory predictions will help to determine CMO configurations with the greatest potential [58]. During this stage, final conclusions will be drawn about what works, for whom, and in what context (e.g., hypotheses about how various contexts facilitated or inhibited generative mechanisms). Results will refine the program theories, which, in turn, will guide intervention refinement and contribute to the broader mental health promotion research and practice agenda.
Sustainability planning
The research team will work with youth collaborators and policy partners to negotiate an appropriate mechanism to support sustained youth policy engagement (e.g., Community of Practice; youth champions; dissemination plans for Agenda Gap intervention). A formal sustainability plan will be developed in future project phases. Policy partners will also be enlisted to help to disseminate project findings and outputs to build awareness to support meaningful youth engagement in the policy setting.