Evaluating Mental Health First Aid: an Evidence Synthesis Review of Published Empirical Studies.

Background: Mental Health First Aid (MHFA) has received substantial international attention since its founding in the late 1990s, with a growing evidence base relating to its nature and impact across a variety of settings. Aims: To identify the effectiveness of MHFA upon a range of outcomes, recipients, its cost-effectiveness, and the mechanisms of its effect. Method: A systematic evidence synthesis Results: Data from 65 studies show MHFA education improves trainees’ mental health literacy, their perceived condence in helping people living with mental distress and their intentions to help such people. MHFA also raises employees’ knowledge of mental illnesses in the workplace. There was also evidence of MHFA trainees using aspects of the ve-stage ALGEE MHFA approach in their helping behaviour. The quality of the studies in this review varied across different types of studies. No published studies to date have evaluated MHFA’s impact on recipient outcomes, articulated the mechanisms of its effect, its cost-effectiveness, or societal impact. Conclusions: MHFA remains popular, but evidence of its effectiveness upon those receiving it remains unknown. It is urgent to undertake studies testing the effectiveness and cost-effectiveness of MHFA upon recipients, as well as identifying, empirically, how MHFA works, for whom, under what conditions, and barriers to its implementation. Given that the enthusiasm and acceptance of MHFA appears widespread, systematic evaluations of its social impact are warranted. Registration: The review protocol has been submitted to the Open Science Framework (View-only link: https://osf.io/rj4uh/? view_only=d1f9f2ed73724b3f8075c0c4581d0d87).

1. Empirical studies assessing the impact of MHFA and (or) MHFA Training.
2. Reviews of empirical studies (meta-analytic, meta-synthesis or narrative). 3. Qualitative and quantitative studies and descriptive empirical studies of MHFA. 4. Empirical studies that have used any other version of MHFA (Youth MHFA, MHFA-Armed Forces, Teen MHFA as a mental health literacy intervention).

Exclusion criteria
1. Empirical studies that have used MHFA combined with other interventions where it was impossible to detect the effect of MHFA alone.
2. Literature not available in the English language.
3. Studies covered in the systematic reviews and meta-analyses. 4. All pilot or feasibility studies of MHFA. 5. All studies that have evaluated MHFA as part of a range of mental health literacy interventions (i.e., all other interventions that have used MHFA as a foundation against the standard courses of Adult MHFA 12hr, YMHFA 8hr) Screening Titles and abstracts were screened by one reviewer (OA) for initial eligibility. Full texts were then reviewed by one reviewer (OA) to con rm eligibility. Where articles fell outside the inclusion criteria, the papers were additionally reviewed by two independent reviewers (PC) and (KVW), and the decision was taken to either include or exclude through consensus.

Data extraction
Following the guidelines suggested by the Centre for Reviews and Dissemination (CRD) (2009), one reviewer (OA) extracted data using a tailored version of a standardized form. All extracted data were subject to cross checking by an independent reviewer (PC or EV).
The following data were extracted for each included study: Author, year of publication and country All quantitative, qualitative, and mixed method studies were quality-assessed using a modi ed tool appropriate for mixed-studies review (Mixed Methods Appraisal Tool; MMAT) (Hong et al., 2018). The systematic reviews were quality checked against the PRISMA statement and checklist; the Delphi studies were quality-assessed based on the quality criteria proposed by Diamond et al. (2014). Details of the MMAT tool are shown in table 1. Details of the quality assessment of all studies included have been well-documented in the appendix.

Data Synthesis
Due to the variety of study designs in the included studies in this review, the data extracted were descriptively summarized. We have organized the information based on each methodological category, extracting the outcomes, and linking them to the mechanisms and the impact of the various contexts in which MHFA is used.

Results
Of the 226 papers assessed, following removal of duplicates, 66 met the inclusion criteria under the following categories: Review, Systematic Reviews & Meta-Analyses (n=4), Qualitative studies (n=9), Quantitative Non-Randomized studies (n=20), Quantitative Randomized studies (n= 3), Mixed method studies (n=8), Delphi Studies (n=20), and other studies (n=2). Details of search results are included in the supplementary le of this review.

Systematic reviews and Meta-analyses
Two meta-analyses (Hadlaczky et. al., 2014) (Maslowski et. al., 2019), a combined systematic review & meta-analysis ) and a review  were included in the review. Using the PRISMA checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), the details of each review's compliance are detailed in table 2.
Hadlaczky et al. (2014) assessed 15 studies evaluating MHFA training delivered over a 9 or 12-hour period using meta-analysis. The samples typically consisted of a self-nominated population. As reported in this review, MHFA training improved trainees' mental health knowledge and reduced participants' negative attitudes with a combined mean effect size of 0.56 (95% CI= 0.38 -0.74; p < 0.001) for the identi cation of mental health problems and trainees' the knowledge about effective treatment. The observed differences in negative attitudes pre & post measures was reported as moderate (effect size = 0.28 (95% CI= 0.22 -0.35; p < 0.001). While MHFA training was also reported to be effective in increasing trainees' 'help-providing' behavior with a reported small effect size (0.25, 95% CI = 0.12 -0.38; p < 0.001) this was based on the opportunities that trainees had to provide help, rather than actual help. Morgan et al. (2018) identi ed 18 studies (4 cluster-RCTs, 10 RCTs, and 4 controlled trials), six were included in the previous meta-analysis by Hadlaczky and colleagues (2014) which were 4 controlled trials and 2 RCTs, the others were excluded because they were only single-group pre/post studies. Like the studies included in Hadlaczky's meta-analysis, most were conducted in Australia (n=8), while others were conducted across North America (n=5), Europe/UK (n=4) and Hong Kong (n=1) In line with Hadlaczky et al., (2014), Morgan & colleagues (2018) reported an improvement in trainees' knowledge, the study interpreted their effect sizes based on Cohen's work (Cohen, 1992) and they have reported a high effect size at post-intervention (d = 0.72 (95% CI = 0.59, 0.86; p < 0.001), which was smaller (moderate based on Cohen's guideline) at 6-month follow-up (d = 0.54 (95% CI = 0.43,0.64; p < 0.001) and 12 month follow-up (d = 0.31(95% CI = 0.09,0.53; p < 0.006). Small effects were reported for stigmatizing attitudes across all post intervention periods. Further analyses were conducted exploring the type of stigma, most included measures for social distance and personal stigma. Additionally, the perceived con dence levels of mental health rst aiders have been been considered for workplace settings. They included the following study design; 13 single studies, 3 reviews, 3 single study protocol and 3 Delphi studies. In line with the previously reported systematic review & meta-analyses, the Bell et al review reported consistent evidence that MHFA raises employees' knowledge of mental illnesses. MHFA trainees were more aware of where to nd information and professional support and have shown increased con dence to render help to individuals experiencing a mental health crisis. The review also highlighted the lack of evidence from the published studies about the impact of introduction of MHFA in workplaces on the support provided by those trained and the sustainability of such actions of support. In addition, Bell et al reported limited evidence to on the adaptability of MHFA to different workplaces.
In sum, all the review studies included in this section reported on trainee related outcomes which showed an improvement of moderate or high at different time-points. Most signi cant trainee outcomes reported were knowledge of mental health, con dence of trainees to help someone struggling with a mental health crisis, trainees' attitudes was also reported with closer attention placed on different forms of stigma and the trainees psychological distress as a result of the MHFA training. Whereas the effect on the so-called recipients ranged from none too small with the reviews reporting recipient related outcomes around changes in the psychological distress which were all indirectly observed via trainees. However, none of the reviews found studies that gathered data from actual recipients of MHFA.

Quantitative Non-Randomized & Randomized studies
Twenty-three studies are reviewed in this section, the detailed characteristics of the studies & their quality checks are shown in Table 3 Fifteen (65%) of the included studies were non-randomized cohort studies that adopted the use of either a pre & post study or time series design (pre, post, and follow-up design). Sample sizes range from n = 29 (Borrill, 2010) to 606,941 (El-Amin et. al., 2018). The remaining included studies comprised 2 (9%) quantitative non-randomized controlled trials, 2 (9%) quantitative non-randomized cross-sectional studies, 1 (4%) quantitative non-randomized descriptive study, and 3 (13%) randomised control trials. Improvement in the perceived con dence levels of participants following MHFA training was generally reported. Boukouvalas  (2018) observed changes in con dence levels using simulated patient case studies amongst university pharmacy students and the use of vignettes assessed with an ALGEE-based assessment rubic. These vignettes were reported to satisfy diagnostic criteria and correct recognition has been validated by a group of mental health professionals in Australia (Morgan et. al., 2013). Hart et al. (2018) utilized a cluster-randomized crossover trial to assess the e cacy of the teen version of the MHFA intervention used in schools to encourage students' supportive behaviours towards their peers. The study also assessed, the quality of rst aid intentions, recognition of mental health, beliefs about the helpfulness of adults and their measures of stigmatizing beliefs. Trainees were assessed against individuals who were also trained within the same study on a physical rst aid course (PFA). All outcomes except the measures of stigmatizing beliefs were observed using a vignette with characters, John & Jeanie presenting with suicidal ideation/depression and social anxiety/phobia, respectively. The vignette was developed based on teaching in the teenMHFA training.
The study reported a medium effect size favouring teenMHFA (ds = 0.50-0.58) based on a group-by-time interaction for the primary outcome. The primary outcomes, 'helpful rst aid intentions' towards John/Jeanie, showed signi cant group-by-time interactions with medium effect sizes favoring tMHFA. In comparison to the PFA group, teenMHFA also reported a medium effect size and improvement in trainees' con dence supporting a peer (ds = 0.22-0.37) and there was a medium effect size but small improvement in the number of adults rated as helpful (ds = 0.45-0.46). However, reductions in effect sizes were  Table 5.
All included qualitative studies reported that part of the aim of conducting the studies was to explore the bene t of MHFA training for trainees. Improved con dence in relation to helping someone with a mental health condition was a common theme in the descriptive bene ts of MHFA training across the studies. One Australian study , identi ed a positive perceived impact both on an intra-personal level emphasizing MHFA trainees' perceived empathy and con dence levels in providing help to others, and on an inter-personal level, focusing on how trainees manage crises.
A Swedish study (Svensson et al., 2015) reported participants referred to MHFA as a "toolbox" (p500) that increased their self-reported con dence to help someone with suspected mental distress. An Australian study using a case-study approach  involving 14 participants reported more help-seeking behavior among workers which they reported as being associated with having MHFA trainees in the workplace, over and above that provided by training to be offered to all staff in a bid to encourage a more acceptable atmosphere to have a mental health discussion within the workplace.
In sum, qualitative studies seek to describe, and explain participants' experiences of the issue's researchers' study, using themes 'emerging' from participants' narratives, and often drawing upon theoretical and conceptual insights as explanatory tools. But the data reported in the studies included here are largely impressionistic and provide little evidence about the impact of MHFA training on trainees' actual skills, especially in the application and impact of MHFA in 'real-world' settings where their skills are sought. The perceived gains in trainees' con dence, whether assessed by trainees or instructors, are arguably redundant in the absence of the application of the supposed con dence in practice, whether tested on the training, or in practice post-training. None of the included studies explored participants' impressions of the long-term effects of MHFA training on participants' con dence, the different ways participants implemented the interventions in their various settings, or possibilities and pitfalls implementing MHFA with recipients.

Mixed Method Studies
Eight studies were included three Australian, two American, two British, and one Canadian study. Details of the studies included are in Table 6 alongside details of their quality checks.

DELPHI STUDIES
The Delphi technique is known as a method of eliciting and re ning group judgments (Dalkey, 1969). Studies using this methodology have been included in this review to understand how MHFA is administered to achieve its intended outcomes.
Following the development of MHFA, several problem-speci c guidelines designed to enable mental health rst aiders to better use the ALGEE approach were developed using Delphi-consensus methods. Fifteen studies are included here. The detailed characteristics of the studies & their quality checks are shown in Table 7. While all included studies were conducted in Australia, the process of consensus was mostly international, with panels consisting of experts based in the USA, Ireland, UK, Canada, and New-Zealand.
This section has highlighted a range of guidelines on how to provide MHFA interventions to address speci c issues ranging from mental health problems to other within different groups and communities. The development of these guidelines has been based on the main elements of MHFA (ALGEE) on how to support someone struggling with any form of mental health problem. However, the ALGEE steps have no scienti cally established theoretical basis as they cannot be easily mapped against any evidence-based behaviour change taxonomies such as those developed by Michie et al. (2013). Furthermore, there is no empirical evidence to show the impact of the developed guidelines on their intended recipients. The impact of the guidelines on recipient outcomes remains a serious gap in the literature.

Other studies
A recently completed UK NIHR funded study -Wellbeing for Secondary Education (WISE), investigated two aspects of MHFA: the impact on teacher wellbeing, including effects on teachers' reports of depression, their sickness absence, and under-performance at work, and the impact on pupils' mental health and wellbeing. At the time of writing, there has been no published data from the main study, however, data from the pilot study shared by the authors reported an increase in con dence, knowledge, and self-reported helping behaviour amongst teachers, (Kidger et al., 2016), consistent with previous studies that have reported trainees outcomes (e.g.: Morgan et al., 2018) Discussion Following the introduction of MHFA in the late 1990s, it has been subject to a large body of empirical research investigating its application in varied settings.
The purpose of this systematic evidence synthesis was to identify the effectiveness of MHFA upon its target audience, recipients, its cost-effectiveness, and its mechanisms of effect.
Context -The vast majority of the included studies were carried out in Australia, while others were spread across other countries with well-developed mental health services such as the UK, Canada, Wales, China, and Sweden amongst others. Evaluation studies have also developed alongside the growth of the intervention across a range of different contexts, with most of the studies carried out among various communities. Some studies have reported a 'call out' for volunteers as an approach to recruiting participants for their studies. This self-volunteering approach has seen more female participants than males being recruited onto courses. Another common characteristic across the participants are individuals with previous mental health knowledge or experiences. More voluntary or other help. The latter two of these elements involve behaviour change on the part of mental health rst aiders. Evidence shows that the most effective 'psychological interventions' are those underpinned by robust behaviour change theories (Michie et al., 2005). Currently, based on empirical evidence, MHFA, as de ned by the ALGEE approach fails in this regard.
Outcomes -The reviews have emphasized quite a range of improvements in outcomes following MHFA, but these are focused largely on trainee-related outcomes such as mental health literacy, which includes knowledge and recognition of mental health problems. Other improvements include the perceived raising of the con dence levels of the trainees in rendering help to someone in distress or a mental health crisis and reducing stigmatizing attitudes towards people experiencing mental distress. Conducting a process evaluation which addresses a logic model to explain how Interventions like MHFA work is vital. Explaining these mechanisms could also help address critiques of MHFA (Defehr, 2016) as reproducing "psychocentrism" -the pathologizing of human problems that often arise from social inequalities rather than something people have.

Strengths and limitations
This is the rst evidence synthesis investigating the effectiveness of MHFA across all of levels of involvement. Using rapid realist review methodology, we have included studies from a sizeable body of mostly published work which makes our ndings are better than those we would have gotten from a conventional review of the effectiveness of MHFA. One limitation of this review is short time frame in which a rapid realist review of this sort are typically conducted which can make it di cult to fully theorize the mechanisms as well the linkages between context, mechanisms and outcomes as typically done in a realist review. This may consequently limit the generalizability and potency of ndings. We could also attribute that limitation to the small amount of literature available to draw the proposed relationships.

Future work
There is a pressing need for more and better-designed studies that evaluate MHFA against recipient outcomes. A better test of this surely would be reliably capturing the actual skills developed and used in real-world settings. The qualitative studies in this review has not helped in this regard. Evaluations of MHFA would be more informative and reliable if data are collected from a representative group of end-users. This would include end users from a variety of settings, for example: among the wider population in community studies, workplaces, among particular communities such as young people, older adults, the armed forces and include data on age, gender, and ethnicity, level of education, marital status, accommodation status, and nature and severity of presenting problems.

Conclusions
MHFA remains popular, but evidence of its effectiveness upon those receiving it remains unknown. It is urgent to undertake studies testing the effectiveness and cost-effectiveness of MHFA upon recipients, as well as identifying, empirically, how MHFA works, for whom, under what conditions, and barriers to its implementation. Given that the enthusiasm and acceptance of MHFA appears widespread, systematic evaluations of its social impact are warranted. We have prior to this submission, submitted a review protocol for purpose of integrity documenting the process we have undertaken for this review unto the Open Science Framework (OSF). (View-only link: https://osf.io/rj4uh/?view_only=d1f9f2ed73724b3f8075c0c4581d0d87 ).

Competing interests
The authors declare they have no competing interests Funding No funding applicable.
Authors' contributions PC, PR, EV, KVW and OA conceptualized the idea, OA drafted the manuscript. PC, PR, EV, KVW,TC provided the critical revision to the manuscripts. OA wrote the paper with input from all co-authors . PC & KVW contributed to the inclusion and exclusion criteria. GD, SDB, SW and NOS reviewed the entire manuscripts and provided substantial comments for editing. All authors read and approved the nal manuscript.