SLR Stage 1: Defining research questions, coverage and breadth, and inclusion/exclusion criteria.
Defining aims and research questions
Aims and research questions were formulated to reflect the pragmatic aim of the SLR i.e. guiding the operationalisation of key priorities for local CAMHS transformation. These priorities were identified through Delphi-method consultations (22) with stakeholders including children and young people (CYP), parents/carers, clinicians/other professionals working with CYP, and local healthcare commissioners (22). The SLR aimed to: (i) provide a readily available repository of evidence for effectiveness, acceptability and feasibility of implementing the service improvements identified by stakeholders as key priorities; (ii) enable commissioners to make evidence-informed decisions about redesigning currently provided services and implementing new ones; (iii) guide change processes; (iv) measure outcomes. These objectives led to the following, broad research questions:
-
What are the effective ways of delivering services that were prioritised for CAMHS transformation?
-
What ways of delivering services that were prioritised for CAMHS transformation are feasible and acceptable for CAMHS users?
-
What are the effective procedures for implementing services that were prioritised for CAMHS transformation?
-
What are the effective ways of measuring transformation outcomes?
Defining coverage, breadth and inclusion/exclusion criteria
The pragmatic purpose of the SLR determined coverage corresponding with key priorities for local CAMHS transformation identified through the three-stage Delphi study (22). The process is outlined in Fig. 1 below.
Figure 1 here
Delphi phase 1 (qualitative)
Following completion of the first round of the Delphi, JA conducted an “evidence charting” search to understand the amount and type of evidence available for each key priority identified. A systematic search strategy was then developed to reflect key priorities identified through the first round of the Delphi (see Table 3).
Table 3
Coverage and breath of the SLR based of the areas identified through the Delphi study
Areas of service provision | Key priorities identified - Delphi phase 1 (qualitative) | Areas identified through evidence charting search | Final list key priorities to be covered in the SLR – Delphi round 1 |
Prevention and promotion of MH and wellbeing | Role of family, community and schools (prevention and promotion) | Community based prevention School based prevention Suicide prevention Substance abuse prevention | School based prevention Suicide prevention |
Communication about mental health Education about mental health Information about available CAMHS Promotion of CAMHS | Education and rising awareness | Education and raising awareness |
Developing professional skills/staff training | School staff training | |
Using internet and mobile technologies | Web-based interventions | Technology enabled MH interventions |
Screening and identification | Screening/early detection Initial assessment/ early intervention | Early intervention Screening tools | Screening tools |
Role of family, community and schools (identification) | School based screening Screening in healthcare settings | School based identification Screening in healthcare settings |
Using mobile technologies | | Technology enabled identification |
Access to CAMHS | Access: open vs. referral based Access to CAMHS (specialist care) Referrals system | Access/referrals (general) Pathways Barriers for access/referral | Barriers for access and referral |
Access times | Waiting times | Waiting times |
Accessible settings | Improving access | Improving access |
| User experience | User experience |
Provision of CAMHS | Provision of evidenced based practice Provision of individually tailored services / personalization of CAMHS Role of family, community and schools in CAMHS provision Role of healthcare professionals (other than mental health) in CAMHS provision | Service delivery models Advocacy Therapeutic alliance | Service delivery models |
| Continuity of care Multidisciplinary CAMHS Holistic approach | Integrated/comprehensive services | Integrated/comprehensive services |
| | Interventions delivered using mobile technologies | Technology enabled MH interventions |
Service evaluation and improvement | None | Quality indicators/service evaluation | Quality indicators |
| Quality improvement initiatives Redesign/implementation | Service redesign and implementation |
| User experience/satisfaction | User experience/satisfaction |
Outcome monitoring | | Outcome monitoring |
Table 3 here
Relevant Medical Subject Headings (MeSH) terms and key words in titles and abstracts were searched (see Appendix 1, Table 1). Search results for each key priority were sorted according to relevance (using EBSCO Host results sorting tool) and the first 2000 most relevant abstracts were retrieved and screened using preliminary inclusion/exclusion criteria (8) (See Appendix 1, Table 2). Full texts of included abstracts were retrieved and screened, and full text publications that met inclusion criteria were assigned to categories corresponding with key priorities for CAMHS transformation identified through the first round of the Delphi (See Appendix 1, Table 3).
Delphi round 1
Outcomes of the evidence charting searches, together with findings from the second round of the Delphi study guided the research team’s discussion about the breadth and depth of the definitive literature searches, inclusion of any additional areas, and defining the final inclusion/exclusion criteria. The research team compared Delphi statements for which the highest consensus was achieved in regard to their particular importance for local CAMHS transformation, against areas identified through evidence charting searches to define the coverage and depth of definitive literature search. Additionally, the team consulted external experts about the completeness of the list of key priorities for CAMHS transformation and areas important for designing and delivering effective CAMHS that were not featured in the Delphi results, but in their opinion should be included. (See Table 3).
Delphi round 2
Following the third and final round of the Delphi, the research team selected the themes within each prioritised area for data extraction and reporting (See Table 4).
Table 4
Themes identified within each key priority area
Areas of service provision | Key priorities for local CAMHS redesign | Themes identified |
Prevention and promotion of MH and wellbeing | School based prevention | Evaluation of school-based prevention programmes Development of school-based prevention programmes |
Suicide prevention | Evaluation of school-based suicide prevention programs Evaluation of community-based suicide prevention programs Development and description of suicide prevention programs |
Education and raising awareness | Evaluation of school MH education/awareness/anti-stigma programs Assessment of MH literacy Assessment of attitudes towards MH problems |
Screening and identification | School based identification | Development of school-based identification programmes Evaluation of school-based mental health identification programmes |
Screening in healthcare settings | Outcomes of MH screening in healthcare settings MH screening in healthcare setting and subsequent referral/use of MH services Parental attitudes towards MH screening in healthcare settings MH professionals’ attitudes towards MH screening in healthcare settings |
Screening tools | Development and psychometric properties of screening measures Feasibility/ acceptability/ utility of screening measures |
Access to CAMHS | Barriers for access and referral | Organizational and administrative barriers for access to CAMHS Users’ and healthcare professionals’ perspectives on barriers to seeking help/ access to CAMHS/ treatment engagement Demographic and socioeconomic factors associated with seeking help/ access to CAMHS/ treatment engagement |
Wait times and improving access | Interventions to reduce wait times and/ or improve access to CAMHS Improving access through providing MH services in schools/primary care settings Impact of wait times on attendance/ treatment engagement Service/ patient factors associated with wait times |
Provision of CAMHS | Service delivery models | Interagency collaboration Coordination of care School-based MH services |
Integrated/comprehensive services | Evaluation of an integrated care model |
Technology enabled MH interventions | Evaluation of technology enabled MH interventions Attitudes towards technology enabled MH interventions Development and description of technology enabled MH interventions |
Service evaluation and improvement | Service redesign and implementation | Implementation of services Diffusion of innovations Service improvement/redesign |
User experience/satisfaction | Service users’ experience of CAMHS Service users’ satisfaction with CAMHS Development and psychometric properties of users satisfaction with services measures |
Outcome monitoring | Routine outcome monitoring Service outcomes Development and psychometric properties of outcome measures |
Quality indicators | Development and psychometric properties of quality measures Quality assessment Development of quality standards |
Table 4 here
Defining the final inclusion/exclusion criteria
The methodological framework for conducting scoping studies recommends that initial inclusion/exclusion criteria are developed at the onset of the study, however these are reviewed and if necessary revised post hoc, in light of search results and researchers’ increased familiarity with evidence (8, 11). We excluded papers that were not empirical, not reviews of other studies or not policy documents or guidelines (e.g. commentaries, letters, book reviews). To further define inclusion/exclusion criteria we applied Population, Concept and Context criteria (PCC) as suggested by Joanna Briggs Institute guidelines (10). Once again, our inclusion/exclusion criteria reflected the primarily pragmatic purpose of the review. For example, we only include studies conducted in developed countries where the contexts and settings are likely to be similar to the UK. We decided to exclude studies that did not report CYP mental health (MH) or wellbeing outcomes, and to include studies in which the intervention was delivered to adults, as long as the aim was to influence CYP’s MH outcomes. Due to time restrictions and the large number of retrieved publications, we excluded studies focusing on emotional or behavioural symptoms associated with non-mental health disorders (e.g. autism spectrum disorders) and studies focusing solely on single therapeutic approaches (e.g. cognitive behavioural therapy (CBT) for depression). We decided that the focus must include service delivery and be relevant to community care, regardless of persons or organisation delivering the intervention, and severity or duration of mental health condition, given the purpose of the review and the large number of published reviews of the effectiveness of different therapeutic approach in addressing various MH problems (e.g. CBT for depression). We included studies of any design. See Table 5.
Table 5
Scoping literature review final inclusion/exclusion criteria
| Exclude if: |
1 | Not written in English. |
2 | Published before January 1990. |
3 | Not empirical, not evidence based, not reviews of other studies or not a policy document/guideline (exclude commentaries, letters, book reviews). |
4 | Not directly or indirectly focused on mental health service users age 0–25 years (i.e. studies with parents/carers of mental health service users, service providers will be included). |
5 | No focus on mental health or mental disorders. Exclude if symptoms are associated with non-mental health disorder (e.g. behavioural problems associated with ADS) |
6 | Does not report children, adolescents or young people’s mental health or wellbeing outcomes, if intervention or programme targets adults’ mental health. |
7 | No focus beyond treatment type. The focus must include service delivery and be relevant to community care (regardless of persons or organization providing services, and severity or duration of mental health condition). |
8 | Services are not delivered in community settings (e.g. primary care, schools, youth centres). |
9 | Describes children and adolescents mental health services in developing countries (according to World Economic Situation and Prospects 2014). |
Table 5 here
SLR Stage 2: Identifying relevant studies.
To develop search terms for definitive database searches JA re-read papers included after screening publications identified through evidence charting searches (see Appendix, Table 2), and listed key words and index terms used in each priority area. This list guided the development of terms for electronic bibliographic databases searches (see Appendix 1, Table 4). Search strategies for all areas were discussed with a subject librarian and agreed by the research team, and each strategy was trialled to see if it produces accurate results. Databases and other sources searched are listed elsewhere (18). To ensure that the search strategy was replicable, each search was stored exactly as run, together with search set numbers, and the number of records retrieved. For each search we separately recorded search date and the period searched, and any language or publication status restrictions.
SLR Stage 3: Study selection.
As recommended for systematic reviews, prior to commencing the selection process initial inclusion/exclusion criteria were pilot tested by JA on a sample of 20 papers from each identified area. Results of pilot screening were recorded in a table including reference, key priority area, theme, inclusion/exclusion decision, reason for exclusion (if excluded), and comments on paper’s relevance if there were doubts whether it should be included or excluded. Results of pilot screening were discussed within the research team with particular focus on applying refined inclusion/exclusion criteria (Table 5) and relevance. The studies’ selection process comprised the following stages:
-
All search results from both electronic database searches and hand searches were merged using the reference management software (EndNote). Duplicate records of the same report were removed. Documents retrieved in grey literature searches were filed separately in Excel database and separately screened for relevance.
-
Titles and abstracts of all papers and grey literature documents were examined to remove obviously irrelevant papers (e.g. not in English, not concerning MH services).
-
Abstracts/executive summaries of remaining papers were screened against inclusion/exclusion criteria.
-
Full texts of remaining, potentially relevant papers were retrieved and multiple reports of the same study were linked.
-
Full texts were examined for compliance with inclusion/exclusion criteria. After completing abstracts and full text screening another member of the team double screened 10% of papers in each review area. Team discussions allowed disagreements and boundary papers to be included/excluded via consensus. We kept a detailed record of the outcomes of each stage, including rejected papers and reasons for rejection. Additionally, each screening researcher identified and recorded main themes for each paper in the key priority area. (See Table 4).
SLR Stage 4: Data extraction
After completing the last round of the Delphi study, we collated a final list of key priorities for local CAMHS redesign (22). This list informed the final selection of themes within each identified area for detailed extraction and reporting (see Table 4).
Before data extraction the lead researcher developed an extraction table to be trialled by members of the research team. Because the review included a variety of evidence types i.e. papers describing qualitative and quantitative studies, process evaluation studies, theory and framework papers, discussion and opinion papers, policy documents, we developed different forms to extract different type of evidence. This aimed to maximise the amount of meaningful information extracted, and to facilitate further narrative synthesis process. Data for each key priority area identified through the Delphi study were organised in a separate table(s). Extracted data were organised under the key themes identified in the previous stage (See Table 4). Identifying key themes was an iterative process with some themes added, deleted, or replaced by a new, more relevant ones as the analysis progressed.
SLR Stage 5: Collating, summarizing and reporting results
The pragmatic purpose of SLR guided the selection of frameworks to summarise and report results. We needed a ‘fit for purpose’ method of analysing and synthesising findings that would be both robust, systematic and scientifically driven, yet presented in a way that is useful and accessible for commissioners and other decision-makers.
Firstly, we provided a numerical account of findings for each key priority area identified. We reported years of publication, countries where studies were conducted and study designs or type of publications in both numerical and graphical form. This was to enable readers to establish whether evidence sources were up to date and relevant in UK context; for example, in some areas only a small fraction of reported studies were conducted in the UK, or most studies were dated before the year 2000. Although we were not assessing the quality of evidence, we noted that in some areas majority of publications were opinion and discussion papers, rather than primary research or systematic reviews. It was important to highlight, as any conclusions or recommendations made in regard to these priority areas reflected experts’ opinions, rather than research evidence.
Before commencing narrative data analysis, we created a logic map to represent all themes in each key priority area, to illustrate how some themes from the same or different areas overlap or complement each other, and to help us organise the synthesis to provide the most comprehensive picture (see Appendix 2). To summarise available evidence, draw conclusions, make clear recommendations regarding most appropriate, feasible and acceptable ways of delivering services, effective procedures for implementing these services and measuring outcomes, we carried out a narrative synthesis of evidence for each key priority for local service transformation. The narrative synthesis process was broadly guided by the framework proposed by Popay et al (23). This framework, applicable both to effectiveness and implementation reviews, comprises four main phases that are iterative rather than linear. The framework was developed mainly to synthesis data gathered through systematic literature review, thus we adjusted it to synthesis broader and more varied evidence.
Phase I: Developing theory of change. Explaining a theoretical model of how, why and for whom an intervention works is a recommended rather that mandatory stage of narrative synthesis (23). When conducting a systematic review, a theory of change is usually developed in its initial stages to understand the theory behind the intervention and to inform the decision about review questions, and what type of studies to include. The theory of change can guide an interpretation of findings an assessment of its applicability. However, in systematic reviews, review questions are well defined and usually narrow, regarding only interventions with particular focus or design. In contrast, the described SLR had a much broader objective with research questions that reflected different stages of service transformation including development of new services or redesigning existing ones, service implementation and measuring transformation outcomes. The review focused on a number of areas of service delivery identified as key priorities for local CAMHS transformation. In some priority areas (mainly regarding service delivery model, service redesign and implementation), a majority of publications were discussion and opinion papers and case studies, while in other areas (e.g. technology delivered MH interventions, education and rising awareness) we identified a large number of empirical evaluations of the effectiveness of interventions targeting various outcomes. For the latter, we attempted to outline a theory of change post hoc to describe mechanisms and factors that determined intervention effectiveness in achieving intended outcomes, and contributed to its successful or failed implementation.
Phase II: Developing a preliminary synthesis of evidence. Evidence synthesis was conducted separately for each key priority area, however we identified some overlaps between themes in different areas (see Fig. 2). Where themes overlapped, data from different areas were merged to provide a comprehensive summary. To develop a preliminary synthesis of evidence for each priority area we organised evidence on a number of levels, to identify patterns in factors that contributed to the effectiveness of services or interventions. Initially, we grouped papers in each area based on key themes we assigned them to (see Table 4). In some instances it was sufficient to carry out preliminary data synthesis (e.g. development and psychometric properties of outcome measures). If further organising of evidence was required, we grouped the papers as follows, adjusting the grouping (depending on the nature of evidence available in each area reviewed):
Figure 2 here
From the analysis of information included in the extraction tables, we identified some initial, broad themes and patterns to explore further in the next stage. For each priority area we also considered potential limitations resulting from both the nature of available evidence (e.g. lack of UK studies) and review process (e.g. decisions about search strategy), to consider when translating findings from the review into practice.
Phase III: Exploring relationships in data. The aim of this is to identify factors explaining differences in direction and size of effects across included studies, and factors that might explain differences in facilitators and barriers to successful implementation. The SLR aimed to provide an evidence-base to guide the transformation of local CAMHS in each identified priority area. In practice we were aiming to develop a robust, credible, yet concise and user-friendly summary of evidence (See Appendix 3) readily available for commissioners to support them in making decisions about developing, implementing and evaluating services most appropriate in different contexts and for different user groups. We explored relationships between data in each key priority area to determine which ways of delivering services are the most feasible, clinically and cost-effective, and acceptable for users in different contexts, and to identify common factors contributing to effectiveness and acceptability. Additionally, we described factors that may serve as barriers and facilitators for effective implementation. Finally, we explored optimal ways of measuring potential transformation outcomes that would provide rich, reliable data, while minimizing burden on service users and clinicians.
Phase IV: Assessing the robustness and strength of evidence for drawing and generalising conclusions. SLR guidelines explicitly state that scoping studies do not aim to assess the quality of available evidence, but only report extent, range and nature of research activities undertaken in a field of interest [13, 16]. We accordingly provided a narrative description of types of available evidence, identified gaps for each key priority area, and elucidated what impact this may have on the robustness of any recommendations made and the translating findings into practice (See Appendix 3). We have not assessed the quality of evidence since our SLR included grey literature. However, where feasible, we would recommend assessing the quality of included studies to increase the reliability of findings and recommendations for practice.