Development and validation of the Espaijove.net mental health literacy (EMHL) test for Spanish adolescents

doi:10.21203/rs.2.13057/v1

Download PDF

Research article

Development and validation of the Espaijove.net mental health literacy (EMHL) test for Spanish adolescents

https://doi.org/10.21203/rs.2.13057/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 20 Dec, 2019

Read the published version in International Journal of Environmental Research and Public Health →

Version 1

posted

You are reading this latest preprint version

Background There is evidence of the effectiveness of implementing Mental Health Literacy (MHL) programs in improving mental health knowledge and reducing the stigma. However, there are substantial limitations in the instruments of measurement of mental health literacy. This study aimed to develop and validate the EspaiJove.net MHL test (EMHL) for Spanish adolescents assessing its psychometric properties.Methods The development of the EMHL as a maximum performance test was conducted using item pool generation and pilot study. Content generation was assessed according item relevance by mental health professionals’ and comprehensive and non-offensiveness by adolescents’ focus groups. A convenience sample of high school students aged 13-15y (n=355) participated in the validity study. Reliability was assessed with internal consistency and test-retest. Convergent validity was evaluated comparing effect size among known groups with different levels of mental health knowledge, correlation with mental health-related instruments, and item discrimination index.Results A final version of 35-item EMHL test was obtained with two parts: (i) 1st part consist of binary choice format (yes/no) for the identification of mental disorders; (ii) The part 2 has multiple choice questions with four possible answer options based on the thematic contents of the EspaiJove.net program. Internal consistency was acceptable in the 1st part (Cronbach’s alpha=0.744; Guttman’s lambda 2=0.773) and almost acceptable in the 2nd part (Cronbach’s alpha=0.615; Guttman’s lambda 2=0.643). The test-retest evaluation supported the stability of the test (1st part, ICC= 0.578; 2nd part, ICC= 0.422), no ceiling and floor effects were found. The EMHL test scores discriminated known groups with different levels of mental health knowledge, it is associated with a reduction of related-stigma, emotional symptoms, conduct problems and bullying behaviours and anxiety/depression and self-care quality of life (p<0.05), and it shows a strong discrimination index in almost all items (D≥0.40).Conclusions The EMHL test is a relevant measure for mental health prevention and promotion adapted to Spanish context taking into account the opinion of adolescents, using a non-offensive and adolescent-adapted vocabulary with acceptable validity and stability for assessing MHL levels in adolescents.

Health Economics & Outcomes Research

Health Policy

Infectious Diseases

Mental Health Literacy

EspaiJove.net program

Adolescence

Measurement

tool development

It is estimated that 75% of total people suffering from a mental disorder experienced the onset before age 25 years [1,2], and 50% during adolescence, between 12 and 18 years old [3]. Given the increased risk of onset of a mental disorder in children and adolescent population, efforts are needed for promoting a healthy biological, psychological and social development at these ages [4].

Health and education are closely related, and early education for mental health is recognized as a key promotion strategy [5]. Promoting mental health to prevent mental disorders and their consequences is one of the main goals in public health [6], and improving health literacy reduces health inequities [7]. The lack of mental health literacy is associated with mental illness and the delay of help-seeking, so increasing the community’s mental health literacy (MHL) is needed to empower community for improving mental health [8,9].

Mental Health Literacy (MHL) is defined as “a set of knowledge and beliefs about mental disorders that aid to the recognition, management or prevention” [8]. MHL involves: a) the ability to recognize the development of mental disorders to facilitate early help-seeking behaviors; b) the knowledge and beliefs about risk factors, the causes of mental disorders and how to prevent them; c) the knowledge of how to get professional help, effective treatments available, and effective self-help strategies; d) the attitudes that facilitate the recognition and adequate search for help; and e) the knowledge and skills to provide first aid in mental health and support to other people [8].

Some MHL interventions for adolescents and young people have been developed in recent years and in several countries, such as Australia, United Kingdom, and Portugal [10–14]. These interventions suggest an improvement in mental health knowledge, in facilitating monitoring in help-seeking, an increase in the self-recognition of mental disorders, and an improvement and reducing mental health-related stigma after the intervention. In Spain, the MHL program "EspaiJove.net: a space for mental health”((EspaiJove.net)[15] has been developed in Barcelona in secondary schools.

A systematic review about measurement properties of tools measuring MHL [16] showed that there are only two specific instruments targeting youths, one of them was only specific for depression [17] and another one for improving beliefs and attitudes about mental health [18]. This highlights the need for the development, evaluation and validation of tools addressing mental health knowledge specifically for adolescents and youths who are vulnerable of developing mental illness. So, there is a validation gap in measuring MHL categories and related psychometrics properties of these instruments. Furthermore, no validated MHL measures addressed knowledge of positive/good mental health have been developed [19]. Hence, the existing scales incompletely cover the contents of the MHL and assess it uncertainly. Developing an assessment in test format to comprehensively evaluate the main contents of the MHL (negative and positive mental health, and help seeking) would allow a more specific and rigorous assessment of MHL levels in adolescents, and effectiveness of MHL interventions.

To our knowledge, no specific MHL questionnaire has been validated properly. The aim of this study is to develop and validate the EspaiJove.net Mental Health Literacy (EMHL) test for the assessment of MHL in Spanish, and delivered from the EspaiJove.net intervention thematic contents, allowing the assessment of MHL knowledge and the development and evaluation of interventions aimed at promoting mental health in adolescents into Spaniards. The EspaiJove.net intervention consists in a universal MHL intervention which aims the promotion of mental health, the prevention of mental disorders, and the facilitation of help-seeking behaviors among secondary students into the Spanish context. The EMHL test emerged from the need to assess the efficacy of a program to promote the MHL in adolescents, based on the following aspects a) the age of the program’s target population; b) assessing MHL from a broad perspective of mental health, including positive mental health, not only focusing on restricted number of mental disorders; and c) the inclusion of biased and erroneous statements by adolescents as incorrect response options in order to be able to improve their knowledge.

The EMHL test is a maximum performance test (Criterion-Referenced Test - CRT) based on the thematic contents of the EspaiJove.net. The item pool generation of the EspaiJove.net from the thematic module contents are: 1) Concepts of mental health and mental disorders; 2) Mental health multidisciplinary team network and use of health services; 3) Healthy and risk behaviours in mental health; 4) Social skills and antisocial behaviour, bullying and cyber-bullying; 5) Anxiety; 6) Depression; 7) Self-harm and suicidal behaviours; 8) Eating disorders; 9) Alcohol and substance use; and 10) Psychotic disorders.

The EMHL test development process involved two phases, as shown in Figure 1:

[ Figure 1]

The study adheres to CONSORT guidelines (see additional file 1 for checklist). Trial registration NCT03215654.

Phase 1: Questionnaire content development.

The development of EMHL test was based on: 1) a literature review of MHL measures for adolescents; 2) content analysis of the discourse emerging 6 focus groups held by trained child and adolescent mental health professionals (psychiatrists, psychologists and nurses) and high school students of 14/15y.

The questionnaire content development phase included 5 steps:

Step 1: Item pool generation. The first version of the EMHL test was developed from the thematic content of each module of the EspaiJove.net program resulting on 60-items (1^stversion) (1^st part: 15 items; 2^nd part: 45 items). The experimental versions of the EMHL test consisted of two parts with two response formats: (i) the part 1 consist of binary choice format (yes/no) for the recognition of mental disorders from a list of 15 different diseases (15 items). Mental disorders are based on Diagnostic and statistical manual of mental disorders (5th ed.) (DSM–5) [20]. (ii) The 2^nd part has multiple choice questions with four possible answer options, in which only one is correct. Incorrect answers were considered as distracting items and it were based on stereotypes, prejudices and erroneous affirmations about mental health. The distracting items were selected from focus groups held by adolescents (n = 39) and, then, developed by EspaiJove.net researchers and reviewed by mental health professionals.

Step 2: Then, we conducted 6 focus groups with a total of 29 mental health professionals (expert panel) from four public child and juvenile mental health centers to explore: (i) clinical relevance; (ii) mistakes in wording (question and answers of each item); (iii) comprehensiveness and offensiveness. Between four and seven participants were involved in each group. Semi-structured cognitive interviews were implemented to guide the discussions and recorded. As result, an initially selection of 45 items (2^ndversion) (1^st part: 15 items; 2^nd part: 30 items) from the preliminary 1^stversion was developed excluding the less clinical relevant items of each module considered by skilled mental health professionals.

Step 3: The EMHL test 45-item 2^nd version (1^st part: 15 items; 2^nd part: 30 items) was administered to five high school classrooms (n = 141): two from 4^th grade (n = 69), two from 3^rd grade (n = 50), and one to 2^nd grade (n = 22) in three different public schools. The objective was to examine (i) item relevance by adolescents, (ii) comprehensiveness and level of difficulty, (iii) offensiveness, and (iv) feasibility of the test.

Step 4: Finally, the EMHL test was administered to 3^rd grade high school students (n = 50) and a focus group (n = 5) with the purpose of improving comprehensiveness and vocabulary. The last 35-items test (4^th version) (1^st part: 15 items; 2^nd part: 20 items) were selected for pilot study.

Step 5: Pilot testing study. The final EMHL test 35-item version (1^st part: 15 items; 2^nd part: 20 items) was piloted at a 3^rd grade high school students (n = 23) to assess comprehensiveness and vocabulary. Validation process was performed from this EMHL test version (5^th version).

From the analyses of the results we finally selected 35-items (3^th version) (1^st part: 15 items; 2^nd part: 20 items) deleting each questions that presented two of three following criteria: (i) Psychometric properties (<0.20 in item-total score correlations), (ii) Knowledge level (>50% of positive answer); (iii) Relevance (less relevant items by mental health professionals).

To obtain the EMHL test total score of each part of the test, the formula (A-E)/(n–1) is used, where A: is the number of correct answers, E: the number of errors (including missing values), and n: the number of options for each item. Then, 1^st part of the EMHL test, the formula is (A-E)/(2–1), and for the 2^nd part is (A-E)/(4–1), where each correct answer adds one point to the total score, and each incorrect answer zero points (Uncorrected total score) [21]. To facilitate the interpretation of results, both sections were converted as deciles from 0 to 10 (transformed scores). Higher score means higher mental health knowledge.

Phase 2: Validation of the psychometric properties of the EMHL test

Sample

The validation process was performed through the administration and analysis of the final version of the EMHL test (5^th version) to a convenience nonrandomized sample of high school students of 14/15y (N = 355) in 6 schools of Barcelona, Spain, and signed the informed consent by both adolescents and parents. Exclusion criteria were:1) Special education school; (2) Students with special educational needs and/or with cognitive problems; and (3) not understanding Spanish or Catalan language. Nurses and psychologists, members of the EspaiJove.net team, informed the participants about the contents of the study and administered the EMHL test.

Main validity measures

We hypothesized specific variables to associated with the level of mental health literacy, with a varying degree of strength.

Stigma. Stigma was measured with two questionnaires: (1) the Scaling Community Attitudes toward the Mentally Ill (CAMI) Spanish version [22] is an instrument for the systematic description of the attitudes of the community towards mentally ill people [23], which consists of 40 items divided into four dimensions (Authoritarianism; Benevolence; Community mental health ideology and Social restrictiveness). We only administered the Authoritarism dimension (10 items). For the Social restrictiveness dimension, the 4 questions in the future of the RIBS were chosen, since both works on the same concepts. We choose these two dimensions since they contain items related to the treating and caring for people with mental illness. The score for each subscale is the sum of the positive items, and the reverse of the negative items. All items from Authoritarism dimension are scored on an ordinal scale (5–1), respectively, ranging from 10 to 50. Higher scores mean greater agreement with engaging in the stated attitude; (2) Reported and Intended Behaviour Scale (RIBS) is used to assess reported and intended behavioural discrimination among the general public against people with mental health problems. The RIBS consists in 8 items; the first four items of the RIBS are designed to assess prevalence (past and current) of behaviour in each of the four contexts (1. living with; 2. working with; 3. living nearby; and 4. being in a relationship with someone with a mental health problem) while items 5–8 ask about intended (future) behaviour within the same contexts [24]. We selected four items from 5 to 8 (future behavior). It uses an ordinal Likert scale with five response options: “totally agree”, “somewhat agree”, “neither agree nor disagree”, “somewhat disagree”, “strongly disagree” from 5 to 1 point, respectively. The total score of future behaviours is obtained from a sum of the total answers ranging from 4 to 20. Higher scores indicate greater agreement with engaging the stated behaviour. We hypothesized that higher mental health knowledge would have lower stigma-related.

Mental health. The Strengths and Difficulties Scale (SDQ) was used [25]. The SDQ consists of 25 items which generate scores along five dimensions: emotional symptoms, conduct problems, hyperactivity-inattention, peer problems, and prosocial behaviour (positive mental health). We hypothesized that adolescents with higher emotional symptoms and peer and conduct problems would have lower mental health knowledge, and more prosocial behaviours would have higher mental health knowledge. Non a priori relationship will be found for hyperactivity-inattention because there is no item in the EMHL test about this construct. Each item is rated 0, 1 or 2 points in accordance with being “absolutely true”, “somewhat true” or “not true”. The score is inverted in those items whose presence indicates positive features. The total score ranges from 0 to 40 for each dimension. The SDQ has been validated for use with adolescents aged 11–16.

Health-related quality of life (HRQoL). The 5-level EQ–5D is a brief, multi-attribute, generic, preference-based health status measure [26,27]. The EQ–5D covers five dimensions of health (mobility, self-care, usual activities, pain or discomfort, and anxiety or depression) with five levels of severity in each dimension EQ–5D–5L. We used the Spanish version of EQ–5D–5L and time trade-off preference values from the Catalan general population [28]. EQ–5D–5L scores range from negative values to 1, higher scores indicating better health status, and 0 is equal to death. The single-item EQ–5D visual analogue scale (EQ–5D-VAS) (range 0–100) was also used. We hypothesized that lower HRQoL of anxiety/depression would have lower mental health knowledge but not in other dimensions.

Bullying and cyberbullying. We developed a 4-item scale to assess bully victims and bully behaviors of perpetrators specifically for this study. Two items assess whether an adolescent has been bullied or ciberbullied victim and two items has bully or cyberbully behaviors. Option answers were “Yes” which scores 1 and “No” which scores 0. Total scores was the sum of each item in both dimensions. We hypothesized that adolescents who has been bullied or bulliers would have lower mental health knowledge.

Known-groups validity assessment

We recruited high school teachers (n = 43), nursing and psychology university students (n = 57); primary care physicians and nurses (n = 61); and mental health professionals (psychiatrists, psychologists and nurses) (n = 52). We hypothesized that some groups, in particular health professionals and teachers, would have significant higher mental health knowledge than high school students.

Statistical analysis

Reliability

Missing values were assessed. The distribution of the item responses from complete responders was analyzed in order to detect highly skewed distributions and floor or ceiling effects of correct answers. Internal consistency index for CRTs was calculated using the phi (lambda) coefficient [29,30] as an estimation of the consistency. This coefficient is specific for CRTs and is interpreted as an alpha’s Cronbach coefficient [31], obtaining values between 0 and 1. One month test–retest reliability was assessed with the Intraclass Correlation Coefficient (ICC) two-way random model, testing for absolute agreement between the first and second administration of the scale. Values below 0.4 were considered poor, between 0.40 and 0.59 fair, between 0.60 and 0.74 good, and over 0.75 excellent [32].

Convergent validity

The ability of the EMHL test uncorrected total score to distinguish among different groups was assessed. Differences across known groups were assessed with ANOVA parametric test. The magnitude of the association was estimated with effect size (ES) to compare average differences in MHL mean between subgroups in categorical variables. The cutoffs and the interpretation of ES were low (|0.20| ≤ ES ≥ |0.50|), moderate (|0.50| < ES ≥ |0.80|), and high (ES >|0.80|)(33,34). In the case of continuous measures, the magnitude of the association was assessed by cut-offs for Pearson correlation coefficients: very weak (<0.20), weak (≥0.20-<0.40), moderate (≥0.40-<0.60), strong (≥0.60-<0.80), very strong (≥0.80)(35). Significance tests were all evaluated at the 0.05 level.

Item discrimination index was also assessed and it evaluates how well an individual question sorts the sample who has mastered the material from students who have not. It is based on the comparison of the performance of the extreme groups (low and high) in the test scores. The number of participants who have been successful in the high group compared to the low proficiency group is compared. We selected the 36% of the sample. The discrimination capacity of each item was assessed by these cut-offs: items that must be deleted (≤0.0), inadequate (>0.0-<0.20), low (≥0.20-<0.30), acceptable (≥0.30-<0.40), strong (D≥0.40) discrimination.

Statistical analyses were conducted using the Statistical Package for the Social Science (SPSS) version 22.0 [36] and Excel (Microsoft Office).

Among the 355 high school students, 178 (50.1%) were women (4 subjects were missing), and 56 (15.8%) had non-Spanish nationality, mean(SD) age was 14.5(0.66).

Reliability

No-item category of the EMHL test was missing. Visual inspection of item response frequencies showed very little of skewed distributions in both parts of the EMHL test. Nevertheless, Kolmogorov-Smirnov normality test showed significant for uncorrected and corrected total scores (p<0.001) in both total scores, likely due to the large sample size studied. We assumed that both parts of the EMHL test have a normal distribution of total scores (see Figure 2). Median (25–75 percentile) corrected total scores for the 1^st part was 3(2–7) and for the 2^nd part was 5(2–7) and mean (SD) was 4.2(2.8) for the 1^st part and 4.4(2.8) for the 2^nd part. The 1^st part of the items showed varying difficulty with a large range of correct responses (28.9% to 97.7%), and 2^nd part items were more demanding, ranging from 11.9% to 83.0% of correct responses. Thus, only 5.9% of the total sample scored the maximum possible (floor effect) score of the 1^st part and 0.6% the 2^nd part of EMHL test and 0.6%. For the ceiling effect, the proportion of the total sample scoring the minimum possible score was 0.3% for 1^st part and 0.3% for 2^nd part of EMHL test. The EMHL test showed internal consistency values above ≥0.70 in the 1^st part of 0.744 of alpha’s Cronbach and 0.773 of Guttman’s lambda 2. However, in the 2^nd part was below <0.70, 0.615 of alpha’s Cronbach and 0.643 of Guttman’s lambda 2, respectively. Corrected item-to-total correlations in each item ranged from 0.162 to 0.526 for the 1^st part, and ranged from –0.206 to 0.411 for the 2^nd part (Table 1). Score test–retest reliability using uncorrected scores and measured with the ICC were fair for both parts of the EMHL test (1^st part, ICC = 0.578; p<0.001; 2^nd part, ICC = 0.422; p = 0.012).

[ Figure 2]

[ Table 1]

Convergent validity

Table 2 shows the EMHL test uncorrected total scores by known groups. Results shows significant differences across subgroups (F = 140.459; p<0.001). For 1^st part of the EMHL test, all groups had significant and moderate magnitude of association comparing with high school students (range ES, 0.466 to 0.640; p<0.001). In the 2^nd part of the EMHL test, teachers, university students and primary care physicians and nurses had significant and moderate magnitude of association comparing with high school students (range ES, 0.485 to 0.692; p<0.001), and significant and high magnitude of association when we compared with mental health professionals (ES, 0.812; p<0.001) (see Table 2).

[ Table 2]

Table 3 shows correlations between several scales and EMHL test uncorrected scores. In the 1^st part of the EMHL test, high school students with higher uncorrected scores of MHL had weak strength and significant correlation of CAMI’s Authoritarism dimension (r = –0.246; p<0.01), and very weak but significant correlation of HRQoL of anxiety/depression domain of EQ–5D–5L (r = –0.122; p<0.05), and Emotional Symptoms domain of SDQ (r = 0.140; p = 0.009).

In the 2^nd part of the EMHL test, high school students with higher uncorrected scores of MHL had a weak strength and significant correlation of CAMI’s Authoritharism dimension (r = –0.222; p<0.01) and RIBS’ future behaviour (r = <0.201; p<0.01), and very weak and significant correlation of Conduct Problems domain of SDQ (r = –0.121; p<0.05), HRQoL of self-care domain of EQ–5D–5L and bully behaviors (r = –0.130; p<0.05). Other domains and scales were not significantly related with both parts of total scores of EMHL test.

[ Table 3]

All items of the 1^st part and almost all items of the 2^nd part of the EMLH test show a powerful discrimination index. In the 2^nd part of the test, only the item 18 (Delusions) has acceptable level of discrimination (D = 0.33), and the item 4 (healthy behaviors) has low levels of discrimination (D = 0.24).

[ Table 4]

This study described the development of the EMHL test and its psychometric properties in a sample of Spanish adolescents, showing good validity properties and stability over time. The EMHL test is a measure created to assess the effectiveness of EspaiJove.net intervention, and more generally, to evaluate the constructs of help-seeking, the most relevant mental disorders in adolescence and positive mental health. Accordingly with the developmental process, the EMHL test has been developed to deliver a clinically relevant measure adapted to Spanish context, supported by extensive recommendations by mental health experts and taking into account the opinion of adolescents, using a non-offensiveness and adolescent-adapted vocabulary which was delivered from the contents of the EspaiJove.net universal MHL intervention. Regarding to validity results, the EMHL test showed to be capable of distinguishing known-groups with different levels of mental health knowledge, showed to be correlated with some variables related with mental health, related-behaviours and HRQoL, and it has the capacity to discriminate those adolescents who has mastered the materials of the EspaiJove.net who not. So, these results suggest that this test will be appropriate for the assessment of its efficacy after the intervention. However, some reliability measures showed slightly low scores in the EMHL test.

Reliability of the EMHL test

Score distribution for the EMHL test showed that it has an appropriate ceiling and floor effect and stability over time, only 5.9% and 0.6% of the adolescents answered all the items correctly in the 1^st part and the 2^nd part of EMHL test, respectively. Regarding reliability, these results suggest that this test can be administered over time for the assessment of the efficacy of interventions aimed to increase MHL among Spanish adolescents, such as EspaiJove.net intervention, and adult population with an estimated higher level of MHL. However, internal consistency showed low results for the 2^nd part of the EMHL test, although almost reaching ≥0.70 value which is considered as acceptable. Internal consistency ranges from poor to fair in both parts of the EMHL test, same results has been showed in corrected item-to-total correlations. According to a previous systematic review about the quality of developed MHL instruments [16], from fifteen developed MHL instruments that assessed internal consistency or reliability, five of them showed poor quality properties, two in adolescent population for assessing mental disorders and depression, respectively [17,18], and six for general population [37–42], one for schizophrenia, two for depression, and three for mental disorders, respectively. One hypothesis about these low values is because the EMHL test broadly covers a wide range of mental health constructs. In fact, this instrument asks about mental health services, healthy and risk behaviours, conduct problems and antisocial behaviours among peers, and several mental disorders and problems such as anxiety, mood, eating and behavioural, substance use and psychotic disorders, and self-harm and suicidal behaviours. So, it is probably that the capacity of the EMHL test to cover main clinically relevant questions to promote mental health and to prevent mental disorders may considerably have decreased its internal consistency. Nevertheless, during the development process skilled mental health professionals and high school students were asked about the items that they considered more relevant to be administered to adolescent for improving mental health. So, as a conclusion, although some reliability values showed poor properties, we consider that the opinion of skilled professionals and targeted adolescent population prevails over statistical analyses for the final inclusion of the items in the EMHL test.

Convergent validity of the EMHL test

Both parts discriminated between wide ranges of groups (adolescents, postgraduate students, high school teachers, primary care professionals and mental health professionals) with different levels of mental health knowledge. Discriminative capacity of the EMHL test between these groups makes possible to take studies in youth, even adult population and to take studies to assess the effectiveness of MHL interventions aimed to increase the level of mental health knowledge until mental health professionals which would be the gold standard of MHL and expertise. Furthermore, the EMHL test also demonstrated significant correlations between other mental health-related constructs, as our a priori hypotheses regarding the pattern of correlations between MHL and mental health stigma-related, emotional symptoms, anxiety and depression HRQoL, conduct problems and bullying behaviours were generally satisfied. As predicted, adolescents with higher MHL levels had less mental health related-stigma, anxiety and depression HRQoL, conduct problems and bullying behaviours, and higher emotional symptoms. Unexpectedly, higher self-care HRQoL is correlated with higher MHL levels. This unexpectedly correlation is probably related with positive mental health items of the EMHL test. These results suggest that higher scores of MHL using the EMHL test have an impact on these constructs.

Strengths and limitations

Our study has several strengths, including a large sample size, as well as the fact that the EMHL test addresses a gap in the assessment of adolescent MHL. Methodologically, this study proposes an evaluation mode of the MHL scarcely explored so far. The CRTs are generally used in education to find out to what extent each subject dominates the criterion of interest. In this case, the selection of the items is not based on the individual differences and the variability of the answers but on the purpose of the test: identify those students who manage the domain of those who do not. Therefore, the CRTs’ decisions are based on acceptable performance rather than relative position [43].

On the other hand, our study has some limitations that are worth noting. Although this instrument has been developed for the Spanish context identifying the most relevant items for preventing mental health problems, the EMHL test has been developed with adolescents living in Barcelona, so a further validation should be assessed in other cities, regions and settings, such as rural areas. Second, the broad scope of the EMHL test could be diminishing its reliability. However, we consider that adolescents who know key questions about mental health prevention and they considered as useful are more important and clinically relevant that the consistency of the test assessed statistically.

Recent systematic reviews and meta-analyses have shown that there is insufficient evidence about the positive impact of the school-based MHL interventions, and therefore more studies with better methodological designs are needed [44,45]. According to that, this study provides a new valid instrument for the evaluation of MHL interventions. Although the EMHL test is only used for the EspaiJove.net intervention, it could be a promising based tool to inspire others MHL measurements.

CAMI, Community Attitudes toward the Mentally Ill; cRCT, clustered Randomised Controlled Trial; HRQoL, Health-related quality of life; MD, Mental Disorders; MH, Mental Health; MHL, Mental Health Literacy; EMHL, EspaiJove.net Mental Health Literacy; RIBS, Reported and Intended Behaviour Scale; SDQ, Strengths and Difficulties Scale.

Ethics approval and consent to participate

The Independent Ethics Committee at the Fundació Unio Catalana Hospitals in Barcelona, Spain has approved the trial study protocol. The trial has also been registered on the Clinical.Trial.gov register NCT03215654 (date registration July 12, 2017).

Written informed consent of all adults participating (teachers, university students, primary care physicians and nurses and mental health professionals) and high school students and parents of all students participating in the study was requested.

Consent for publish

Not applicable

Availability of data and materials

Not applicable

Competing interest

The authors declare that they have no competing interests

Funding

This study was funded by the Carlos III Institute of Health, Ministry of Economy and Competitiveness (Spain). This was awarded on the 2015 call for tenders under the Health Strategy Action 2013–2016, within the National Research Programme aimed at Societal Challenges, within the Technical, Scientific and Innovation Research National Plan 2013–2016, with reference PI15/01613, and was co-funded by European Union ERDF funds (European Regional Development Fund).

The funding covered data collection, analysis and interpretation of data. The funding body did not participate in the study design and manuscript writing.

Authors’ contributions

All authors have contributed to the design and content of the study. LLL and RC are responsible for study coordination and obtaining funding. More specifically, RC, VMA and PC contributed to the development of research methodology, data collection, performed the quantitative and qualitative data collection and writing the initial manuscript draft. JG and MT informed the participants about the study, administered the EMHL test and conducted the focus group with professionals and adolescents. RC, PC and DR prepared the data for analysis. CG and DR performed the data analysis and participated in the manuscript writing and final version. JA supervised the study and participated in the manuscript revision. All authors read and approved the final version of the manuscript.

Acknowledgments

The research team would like to thanks all schools, educational and health professionals and students that participated in the study. Finally, we would like to thanks the collaboration of David John Macfarlane of the Institut Hospital del Mar d’Investigacions Mèdiques (IMIM) and Inés Pérez González-Anguiano of the Health Protection Agency-Health Department of the Catalunya Generalitat.

Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005 Jun;62(6):593–602.
Jones PB. Adult mental health disorders and their age at onset. Br J Psychiatry Suppl. 2013 Jan;54:s5–10.
Kessler RC, Amminger GP, Aguilar-Gaxiola S, Alonso J, Lee S, Ustun TB. Age of onset of mental disorders: a review of recent literature. Curr Opin Psychiatry. 2007 Jul;20(4):359–64.
Coughlan H, Cannon M, Shiers D, Power P, Barry C, Bates T, et al. Towards a new paradigm of care: The International Declaration on Youth Mental Health. Early Interv Psychiatry. 2013;7(2):103–8.
World Health Organization. Education and health go hand in hand [Internet]. 2010. Available from: https://www.who.int/dg/speeches/2010/educationandhealth_20100920/en/
World Health Organization. Adolescent mental health: mapping actions of nongovernmental organizations and other international development organizations. [Internet]. WHO Press. Geneva (Switzerland); 2012. Available from: http://www.who.int/iris/handle/10665/44875
Public Health England, UCL Institute of Health Equity. Improving health literacy to reduce health inequalities. Public Heal Engl. 2015;(September):1–16.
Jorm AF. Mental health literacy: empowering the community to take action for better mental health. Am Psychol. 2012 Apr;67(3):231–43.
Zachrisson HD, Rodje K, Mykletun A. Utilization of health services in relation to mental health problems in adolescents: a population based survey. BMC Public Health. 2006 Feb;6:34.
Birchwood M, Singh SP. Mental health services for young people: matching the service to the need. Vol. 54, The British journal of psychiatry. Supplement. England; 2013. p. s1–2.
Chisholm K, Patterson P, Torgerson C, Turner E, Jenkinson D, Birchwood M. Impact of contact on adolescents’ mental health literacy and stigma: the SchoolSpace cluster randomised controlled trial. BMJ Open. 2016 Feb;6(2):e009435.
Hodges CA, O’Brien MS, McGorry PD. Headspace: National Youth Mental Health Foundation: making headway with rural young people and their mental health. Aust J Rural Health. 2007 Apr;15(2):77–80.
Schulze B, Richter-Werling M, Matschinger H, Angermeyer MC. Crazy? So what! Effects of a school project on students’ attitudes towards people with schizophrenia. Acta Psychiatr Scand. 2003 Feb;107(2):142–50.
Campos L, Dias P, Duarte A, Veiga E, Dias CC, Palha F. Is It Possible to “Find Space for Mental Health” in Young People? Effectiveness of a School-Based Mental Health Literacy Promotion Program. Int J Environ Res Public Health. 2018 Jul;15(7).
Casanas R, Arfuch V-M, Castellvi P, Gil J-J, Torres M, Pujol A, et al. “EspaiJove.net”- a school-based intervention programme to promote mental health and eradicate stigma in the adolescent population: study protocol for a cluster randomised controlled trial. BMC Public Health. 2018 Jul;18(1):939.
Wei Y, McGrath PJ, Hayden J, Kutcher S. Measurement properties of tools measuring mental health knowledge: a systematic review. BMC Psychiatry. 2016 Aug;16:297.
Hart SR, Kastelic EA, Wilcox HC, Beaudry MB, Musci RJ, Heley KM, et al. Achieving Depression Literacy: The Adolescent Depression Knowledge Questionnaire (ADKQ). School Ment Health. 2014;6(3):213–23.
Serra M, Lai A, Buizza C, Pioli R, Preti A, Masala C, et al. Beliefs and attitudes among Italian high school students toward people with severe mental disorders. J Nerv Ment Dis. 2013 Apr;201(4):311–8.
Wei Y, McGrath PJ, Hayden J, Kutcher S. Mental health literacy measures evaluating knowledge, attitudes and help-seeking: a scoping review. BMC Psychiatry. 2015 Nov;15:291.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). American Psychiatric Publishing, editor. Washington, DC: American Psychiatric Publishing; 2013.
Abad FJ, García C, Gil B, Vicente-Ponsoda, Julio Olea Revuelta JJ. Introducción a la Psicometría Teoría Clásica de los Tests y Teoría de la Respuesta al Ítem. Universidad autónoma de Madrid, editor. Madrid; 2016.
Ochoa S, Martinez-Zambrano F, Vila-Badia R, Arenas O, Casas-Anguera E, Garcia-Morales E, et al. Spanish validation of the social stigma scale: Community Attitudes towards Mental Illness. Rev Psiquiatr Salud Ment. 2016;9(3):150–7.
Taylor SM, Dear MJ. Scaling community attitudes toward the mentally ill. Schizophr Bull. 1981;7(2):225–40.
Evans-Lacko S, Rose D, Little K, Flach C, Rhydderch D, Henderson C, et al. Development and psychometric properties of the reported and intended behaviour scale (RIBS): a stigma-related behaviour measure. Epidemiol Psychiatr Sci. 2011 Sep;20(3):263–71.
Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry. 2001 Nov;40(11):1337–45.
Dolan P. Modeling valuations for EuroQol health states. Med Care. 1997 Nov;35(11):1095–108.
EuroQol group. EuroQol - a new facility for the measurement of health-related quality of life. Health Policy (New York). 1990;16(3):199–208.
Badia X, Schiaffino A, Alonso J, Herdman M. Using the EuroQoI 5-D in the Catalan general population: feasibility and construct validity. Qual Life Res. 1998 May;7(4):311–22.
Brennan R, Lockwood R. A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability Theory. Appl Psychol Meas. 1980;4(2):219–40.
Traub RE. Book Review : A Guide to Criterion-Referenced Test Construction. Berk RA, editor. Vol. 9, Applied Psychological Measurement. Baltimore, Ma: The Johns Hopkins University Press; 2007. 227–231 p.
Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334.
Cicchetti D V. Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instrument in Psychology. Psychol Assess. 1994;6:284–90.
Kazis LE, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Med Care. 1989 Mar;27(3 Suppl):S178–89.
Cohen A. Statistical power for the behavioral sciences. Lawrence Eribaum, editor. Hillsdale, NJ: Lawrence Eribaum; 1988.
Evans JD. Straightforward statistics for the behavioral sciences. Pacific Grove, CA: Brooks/Cole Publishing; 1986.
SPSS 22 version for Windows. SPSS for Windows. Chicago, Ill; 2013.
Daltio CS, Attux C, Ferraz MB. Knowledge in schizophrenia: The Portuguese version of KAST (Knowledge About Schizophrenia Test) and analysis of social-demographic and clinical factors’ influence. Schizophr Res [Internet]. 2015;168(1–2):168–73. Available from: http://dx.doi.org/10.1016/j.schres.2015.07.009
Compton MT, Hankerson-Dyson D, Broussard B. Development, item analysis, and initial reliability and validity of a multiple-choice knowledge of mental illnesses test for lay samples. Psychiatry Res [Internet]. 2011;189(1):141–8. Available from: http://dx.doi.org/10.1016/j.psychres.2011.05.041
Evans-Lacko SE, Little K, Meltzer H, Rose D, Rhydderch D, Henderson C, et al. Mental Health Knowledge Schedule. Can J Psychiatry. 2010;55(7):157–65.
Gabriel A, Violato C. The development of a knowledge test of depression and its treatment for patients suffering from non-psychotic depression: A psychometric assessment. BMC Psychiatry. 2009;9:56.
Gulliver A, Griffiths KM, Christensen H, Mackinnon A, Calear AL, Parsons A, et al. Internet-based interventions to promote mental health help-seeking in elite athletes: An exploratory randomized controlled trial. J Med Internet Res. 2012;14(3).
Wang J, He Y, Jiang Q, Cai J, Wang W, Zeng Q, et al. Mental health literacy among residents in Shanghai. Shanghai Arch psychiatry [Internet]. 2013;25(4):224–35. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid = 4054559&tool = pmcentrez&rendertype = abstract
Berk R. A consumers’ guide to criterion-referenced test reliability. J Educ Meas. 1980;17(4):323–49.
Mellor C. School-based interventions targeting stigma of mental illness: systematic review. Psychiatr Bull. 2014 Aug;38(4):164–71.
Mehta N, Clement S, Marcus E, Stona A-C, Bezborodovs N, Evans-Lacko S, et al. Evidence for effective interventions to reduce mental health-related stigma and discrimination in the medium and long term: systematic review. Br J Psychiatry. 2015 Nov;207(5):377–84.

Table 1. Percentage of floor and ceiling effects and item-total score correlations.

Item	Floor N(%)	Ceiling N(%)	Item-total score correlation
First part of the EMHL test
1 (Schizophrenia)	67(19.0)	286(81.0)	0,309
2 (Diabetes)	12 (3.4)	341 (96.6)	0,337
3 (Phobia)	60 (17.0)	293 (83.0)	0,272
4 (Asthma)	13 (3.7)	340 (96.3)	0,405
5 (Bipolar disorder)	24 (6.8)	329 (93.2)	0,340
6 (Bulimia)	127 (36.0)	226 (64.0)	0,411
7 (Cerebral palsy)	172 (48.7)	181 (51.3)	0,526
8 (Bronchitis)	4 (1.1)	349 (98.9)	0,394
9 (Alzheimer disease)	251 (71.1)	102 (28.9)	0,287
10 (Depression)	38 (10.8)	315 (89.2)	0,374
11 (Chickenpox)	8 (2.3)	345 (97.7)	0,408
12 (Down's Syndrome)	199 (56.4)	154 (43.6)	0,503
13 (Epilepsy)	153 (43.3)	200 (56.7)	0,354
14 (Anorexia nervosa)	79 (22.4)	274 (77.6)	0,437
15 (Substance dependence)	85 (24.1)	268 (75.9)	0,162
Total First Part	2 (0.6)	21 (5.9)	*0,744
			+0,773
Second part of the EMHL test
1 (Definition of mental health)	282 (79.9)	71 (20.1)	0,132
2 (Where to go for help)	87 (24.6)	266 (75.4)	-0,206
3 (Who develops a mental disorder)	72 (20.4)	281 (79.6)	0,249
4 (Healthy Behaviors)	257 (72.8)	96 (27.2)	-0,133
5 (Night rest)	102 (28.9)	251 (71.1)	0,268
6 (Cannabis / alcohol consumption)	308 (87.3)	45 (12.7)	0,401
7 (Bullying)	139 (39.4)	214 (60.3)	0,127
8 (Cyberbullying)	110 (31.2)	242 (68.6)	0,229
9 (Social skills)	191 (54.1)	162 (45.9)	0,236
10 (Suicidal ideation friend / family)	151 (42.8)	202 (57.2)	0,144
11 (Self-injury)	109 (30.9)	244 (69.1)	0,267
12 (Suicide Behavior Alert)	60 (17.0)	293 (83.0)	0,178
13 (Depression)	270(76.5)	83 (23.5)	0,407
14 (Eating Disorders)	294 (83.3)	59 (16.7)	0,256
15 (Start of Eating Disorders)	270 (76.5)	83 (23.5)	0,238
16 (Characteristics of Eating Disorders)	281 (79.6)	72 (20.4)	0,411
17 (Symptoms Schizophrenia)	269 (76.2)	84 (23.8)	0,301
18 (Delusions)	261 (73.9)	92 (26.1)	0,039
19 (Psychotic episode)	311 (88.1)	42 (11.9)	0,396
20 (Alcohol dependence)	243 (68.8)	110 (31.2)	0,376
Total Second Part	1 (0.3)	2 (0.6)	*0,615
			+0,643
EMHL= EspaiJove.net Mental Health Literacy test.
*Cronbach´s alpha
+Guttman´s Lambda 2

Table 2. EspaiJove.net Mental Health Literacy (EMHL) test scores according to “known groups”, using uncorrected total scores.

Subgroups	N	Mean (SD)	Effect size (ES)	p
First part of the EMHL test
High school students (ref.)	355	7.07 (4.96)	---	<0.001
Teachers	43	11.56 (3.42)	0.466	<0.001
University students	57	12.44 (2.32)	0.570	<0.001
Primary care physicians and nurses	61	13.56 (1.90)	0.654	<0.001
Mental health professionals	51	13.55 (2.37)	0.640	<0.001
Second part of the EMHL test
High school students (ref.)	355	-1.06 (1.48)		<0.001
Teachers	43	0.60 (1.52)	0.485	<0.001
University students	57	1.32 (1.34)	0.645	<0.001
Primary care physicians and nurses	61	1.86 (1.56)	0.692	<0.001
Mental health professionals	52	3.13 (1.53)	0.812	<0.001
EMHL test= EspaiJove.net Mental Health Literacy test; ES= effect size; ref.= reference group; SD= standard deviation.

Table 3. Correlations between the EMHL test scale using uncorrected total score and other construct-related scales

		First part of the EMHL test		Second part of the EMHL test
	N	Pearson Correlation	p-value (2-tailed)	Pearson Correlation	p-value (2-tailed)
Stigma-related (CAMI)	344	-0.246^**	<0.001	-0.222^**	<0.001
Stigma-related (RIBS)	352	-0.094	0.079	-0.201^**	<0.001
Emotional Symptoms (SDQ)	352	0.140^**	0.009	0.032	0.553
Conduct Problems (SDQ)	353	-0.095	0.074	-0.121^*	0.023
Hyperactivity/inattention (SDQ)	353	0.044	0.406	-0.028	0.596
Peer problems (SDQ)	353	-0.041	0.442	0.006	0.909
SDQ total difficulties score	352	0.025	0.645	-0.030	0.576
Prosocial (SDQ)	353	0.029	0.588	-0.023	0.664
Bully victim	352	0.016	0.764	0.068	0.201
Bully perpetrator	351	-0.050	0.348	-0.130^*	0.014
Mobility (EQ-5D-5L)	353	-0.045	0.401	0.000	0.995
Self-care (EQ-5D-5L)	353	0.075	0.160	0.130^*	0.014
Usual activities (EQ-5D-5L)	353	-0.057	0.286	0.093	0.081
Pain/discomfort(EQ-5D-5L)	353	-0.021	0.696	-0.051	0.341
Anxiety/depression (EQ-5D-5L)	353	-0.122^*	0.021	-0.062	0.246
EQ-5D-5L total score	353	0.018	0.741	0.070	0.193
EQ-5D-VAS	352	-0.083	0.120	-0.059	0.273
CAMI= Community Attitudes toward Mental Illness; EMHL= EspaiJove.net Mental Health Literacy test; EQ-5D-5L= 5-level EQ-5D version; EQ-5D-VAS= EQ-5D Visual Analogue Scale; RIBS= Reported and Intended Behaviour Scale; SDQ= Strength and Difficulties Questionnaire.
**. Correlation is significant at the 0.01 level (2-tailed).
*. Correlation is significant at the 0.05 level (2-tailed).

Table 4. Item discrimination index of the EMLH test using 36% of the sample

First part of the EMHL test	D
1 (Schizophrenia)	0.96
2 (Diabetes)	0.85
3 (Phobia)	0.92
4 (Asthma)	0.88
5 (Bipolar disorder)	0.88
6 (Bulimia)	1.12
7 (Cerebral palsy)	1.25
8 (Bronchitis)	0.83
9 (Alzheimer disease)	0.78
10 (Depression)	0.93
11 (Chickenpox)	0.86
12 (Down's Syndrome)	1.21
13 (Epilepsy)	1.02
14 (Anorexia nervosa)	1.07
15 (Substance dependence)	0.83
Second part of the EMHL test	D
1 (Definition of mental health)	0.52
2 (Where to go for help)	0.53
3 (Who develops a mental disorder)	1.03
4 (Healthy Behaviors)	0.24
5 (Night rest)	1.05
6 (Cannabis / alcohol consumption)	0.76
7 (Bullying)	0.87
8 (Cyberbullying)	0.99
9 (Social skills)	0.90
10 (Suicidal ideation friend / family)	0.84
11 (Self-injury)	1.06
12 (Suicide Behavior Alert)	1.01
13 (Depression)	0.87
14 (Eating Disorders)	0.50
15 (Start of Eating Disorders)	0.54
16 (Characteristics of Eating Disorders)	0.71
17 (Symptoms Schizophrenia)	0.66
18 (Delusions)	0.33
19 (Psychotic episode)	0.55
20 (Alcohol dependence)	0.95
EMHL= EspaiJove.net Mental Health Literacy test; D= Item Discrimination Index.

CONSORT2010Checklist.doc

Download PDF

Journal Publication

published 20 Dec, 2019

Read the published version in International Journal of Environmental Research and Public Health →

Version 1

posted

You are reading this latest preprint version

Development and validation of the Espaijove.net mental health literacy (EMHL) test for Spanish adolescents

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods

[ Figure 1]

Phase 1: Questionnaire content development.

The questionnaire content development phase included 5 steps:

Phase 2: Validation of the psychometric properties of the EMHL test

Sample

Main validity measures

We hypothesized specific variables to associated with the level of mental health literacy, with a varying degree of strength.

Known-groups validity assessment

Statistical analysis

Reliability

Convergent validity

Results

Reliability

[ Figure 2]

[ Table 1]

Convergent validity

[ Table 2]

[ Table 3]

[ Table 4]

Discussion

Reliability of the EMHL test

Convergent validity of the EMHL test

Strengths and limitations

Conclusions

List of abbreviations

Declarations

Ethics approval and consent to participate

Consent for publish

Availability of data and materials

Competing interest

Funding

Authors’ contributions

Acknowledgments

References

Tables

Supplementary Files

Status:

Journal Publication

Version 1