Effectiveness of the Extension for Community Healthcare Outcomes (ECHO) Model of continuing tele-education and influence of key learning conditions on the development of competencies in healthcare professionals: Protocol for a mixed methods systematic review

doi:10.21203/rs.3.rs-334821/v1

Download PDF

Research Article

Effectiveness of the Extension for Community Healthcare Outcomes (ECHO) Model of continuing tele-education and influence of key learning conditions on the development of competencies in healthcare professionals: Protocol for a mixed methods systematic review

https://doi.org/10.21203/rs.3.rs-334821/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

The Extension for Community Healthcare Outcomes (ECHO) Model of continuing tele-education is an innovative guided-practice model aiming at amplifying healthcare professionals’ competencies in the management of chronic and complex health conditions. While data on the effectiveness of the ECHO Model is increasingly available in the literature, the influence of key learning conditions on the model effectiveness remains unclear. Therefore, the overarching aim of this systematic review is to identify, appraise and synthesize the available quantitative (QUAN) and qualitative (QUAL) evidence regarding the effectiveness of the ECHO Model and the influence of key learning conditions on the development of competencies in healthcare professionals.

Methods

This proposed systematic review will be conducted in accordance with the Joanna Briggs Institute (JBI) methodology for Mixed Methods Systematic Reviews (MMSR) and will follow a convergent segregated approach. A systematic search will be undertaken using QUAN, QUAL and mixed methods (MM) studies of ECHO-affiliated programs identified in five databases. A publication date filter will be applied to find the articles published from 2003 onwards. Sources of unpublished studies and grey literature will be searched as well. Retrieved citations will be screened by two review authors independently. Disagreements will be resolved through discussion until a consensus is reached, or by including a third reviewer. Studies meeting the predefined inclusion criteria will be assessed on methodological quality and the data will be extracted using standardized data extraction forms. Separate QUAN and QUAL synthesis will be performed, and findings will be integrated using a matrix approach for the purpose of comparison and complementarity.

Discussion

This MMSR will fulfill important gaps in the current literature on the ECHO Model, as the first to provide estimates on its effectiveness and consider simultaneously the influence of key learning conditions on the development of competencies in healthcare professionals. As implementations of the ECHO Model greatly vary depending on the context, topic and targeted professional group, a better understanding of the conditions that contribute to competencies’ development in healthcare professionals is crucial to inform the design and implementation of the model.

Systematic review registration:

This MMSR protocol is pending registration in the International Prospective Register of Ongoing Systematic Reviews PROSPERO (submitted October 21, 2020; ID Number: 197579).

Health Economics & Outcomes Research

Competency

Distance learning

Information and communication technologies

Project ECHO

Primary care

Technology-enabled collaborative models

Tele-mentoring

Videoconferencing

Virtual community of practice

Virtual collaboration

Innovation in continuing education: the ECHO Model

In the current of change and uncertainty, healthcare professionals are expected to develop high levels of competencies in order to effectively manage complex health conditions and to respond to populations’ needs [1]. In the health education sciences literature, the definition of competency remains polysemous due to varying conceptions and underlying philosophical assumptions [2]. However, authors mostly agree that a competency: (1) requires the efficient mobilization and orchestration of a cluster of internal resources (e.g., knowledge, attitudes, values, skills, abilities) and external resources (e.g., material, human, organizational) in clinical practice; (2) is constantly contextualized to a specific situation; and (3) evolves throughout a professional’s lifetime [3–8]. Hence, a competency can be understood as complex and systemic knowledge-in-action to solve real-life situations effectively [9, 10]. The development of competencies refers to a dynamic and ongoing process of learning and practice renewal requiring engagement at an individual and collective level [5, 11, 12]. The development of competencies in healthcare professionals is crucial in ensuring they practice within the full scope of their role and in improving patients’ health outcomes [13, 14].

In the healthcare professions, continuing education (CE) is recognized as an essential aspect of competency development [15–17]. CE can be described on a continuum, from informal learning experiences and practices to formal educational interventions held through a diversity of modalities and sources of media, in both academic and clinical practice settings [18]. In all cases, CE focuses on meaningful learning experiences that are conducive to the development of competencies in healthcare professionals.

In recent decades, a number of continuing educational programs using information and communication technologies (ICTs) have been developed to overcome barriers related to healthcare professionals’ participation in CE activities (e.g., staff shortages, cost and travel time) [19–25]. Advantages of ICT-based programs include increased accessibility, lower costs and personalization compared to large-group, in-person instruction [26]. One of these ICT-based programs is the Extension for Community Healthcare Outcomes (ECHO) Model [27], a continuing tele-education program that provides ongoing support and clinical supervision to healthcare professionals in the management of complex and chronic health conditions.

Description and conceptual representation of the ECHO Model

Launched in 2003 at the University of New Mexico Health Center, ECHO aims to facilitate knowledge sharing, capacity building and expanded access to best practice care to reduce treatment disparities in underserved populations. The ECHO Model was first developed under the name Project ECHO (© 2020, The University of New Mexico, Albuquerque, NM, USA; http://www.hsc.unm.edu/echo) to support primary care providers in rural and carceral settings in managing patients infected by the hepatitis C virus [28–33]. Since then, the model has been replicated for dozens of diseases and health conditions and it now operates at more than 100 academic medical health centres across multiple continents [7]. The ECHO Model involves establishing a network between front-line healthcare professionals located in remote areas—i.e., “spokes”— with a multidisciplinary team of specialists at academic medical centres—i.e., a “hub”— using video conference technology. The model typically includes a 6-to-12-month curriculum of weekly “ECHO clinics”, in which a case-based discussion about a real patient situation and a short didactic presentation are held over two hours.

To offer a meaningful understanding of the ECHO Model, we developed a conceptual representation designed to explore the intended function of the model in context—meaning the examination of surface-level components with the conceptual learning conditions they were designed to foster. This conception was built from Cianciolo and Regehr’s Learning Theory and Educational Intervention Framework[34], a layered perspective based on the premise of enabling a rich examination of the interplay between the pedagogic intention of an educational intervention (educational theories and principles) and its adaptation in a specific context (educational methods, contextual and personal conditions). The authors claim that this examination helps to discern whether a given educational intervention “worked” as intended on an anticipated outcome and to draw any plausible conclusions about which components of the intervention this effect can be attributed to. Figure 1 depicts the proposed layered conceptualization of the ECHO Model and is summarized in the paragraphs below.

As shown in Fig. 1, our conception excavates the layers of the ECHO Model: methods at the surface, principles in the middle, and theories at the core. Together, these three layers depict the favorable educational conditions that must be established in a program for learning to occur. This layered classification illustrates that the ECHO Model has a unique identity, which preserves its essence despite being adapted in a specific context. Also, the lack of clear delineations between layers reflects the absence of clear boundaries between them.

At the bottom of Fig. 1, the layer of educational theories indicates that the ECHO Model is based on three educational theories which together constitute the foundational layer or identity of the model: (1) Social Cognitive Theory [35]; (2) Situated Learning Theory [36, 37]; and (3) Theory of Communities of Practice [38]. According to Cianciolo and Regehr [34], this foundational theory layer represents a context-independent and idealized statement of the educational conditions that must hold for a given intervention to be what its designer claims it is—i.e., the pedagogic intention.

The middle layer of educational principles illuminates the general underlying postulates that are engaged throughout an intervention and clarify the structural learning aspects of the ECHO Model—i.e., assumptions of how participants will learn [39]. These principles may be adjusted to context, but nevertheless they reflect relatively stable approaches to learning. The top layer of the figure, the most context-sensitive, comprises educational methods that account for context, allowing each ECHO model implementation to be tailored to a specific local setting (modes of delivery, functioning and characteristics of educational program). The educational methods used in the ECHO Model are summarized in three main categories, as shown in Fig. 1, each of them specifying the type of learning experience a participant might be exposed to—meaning the intervention components. Concrete examples of how these educational methods are delivered in ECHO implemented programs [28] can be found in additional file 1.

Our conceptualization also suggests that implementation comprises a complex and uncontrolled set of conditions that influence the adaptation of a given educational program in a specific context. Hence, the arrows surrounding Fig. 1 imply that personal and contextual conditions, as an overlay to the maintenance of the conceptual educational conditions of the ECHO model, may influence its effectiveness, sometimes in unanticipated ways. Importantly, our conception reflects the fact that personal and contextual conditions are seen as processes of influence on learning. Specifically, personal conditions refers to influences from an individual perspective, while contextual conditions range from the proximal influences (interpersonal) to increasingly distal influences (institutional, organizational, community and sociopolitical) [40, 41]. In summary, the proposed conception supports the contention that a deeper investigation of the ECHO Model’s effectiveness must be undertaken within a holistic and rich examination of the key learning conditions under which an effect is observed (or not observed)—i.e., the educational, contextual and personal conditions influencing competency development.

Current knowledge gaps about the ECHO Model and added value of the proposed systematic review

There is substantial evidence showing the impacts of the ECHO Model on healthcare professionals’ learning outcomes [7, 42]. For instance, increases in healthcare professionals’ perceived knowledge and confidence in their ability to manage complex cases without referring to specialists, as well as improvements in their ability to perform new behaviours have been reported [7, 42]. Moreover, there is evidence in support of the acceptability and feasibility of the model, notably for reducing healthcare professionals’ feelings of isolation and regarding its cost-effectiveness [42]. The relevance of the didactic presentations’ topics to practice, personal interest and motivation to learn, peer-to-peer interactions and positive reinforcement were reported as favorable conditions in extending healthcare professionals’ knowledge and skills [7, 43]. Additionally, mentorship or supervision led by a qualified and competent team of healthcare professionals—the “hub”—in a helping environment has been found to support ECHO’s participants in a difficult situation and allow the opportunity to reflect on their own practice from feedback and positive reinforcement [43].

While Project ECHO has been successful at expanding its scope and scale internationally, there remains a paucity of evidence regarding the effectiveness of the ECHO Model on the development of healthcare professionals’ competencies [44]. Considering that the model has been implemented in diverse settings, each implementation bringing variations to the original Project ECHO design, and that the model is intended for a wide range of clinical conditions and populations, our current understanding of the key learning conditions that may contribute to its success is limited. A second contributor to this knowledge gap is substandard reporting of implemented ECHO-affiliated programs’ components in primary studies (such as number and duration of intended and actual videoconferencing sessions conducted, professions of participants, retention rates, targeted topics in didactic presentations and adaptations in educational methods) [44]. In order to address these gaps, it has been suggested that there is a need to not only assess the effectiveness of the ECHO Model on the development of healthcare professionals’ competencies, but also to examine in which key learning conditions the model is more effective [44].

A preliminary search of MEDLINE, PROSPERO, the Cochrane Database of Systematic Reviews and the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports was conducted to identify potentially relevant reviews. Only two systematic reviews focusing on the impact of the ECHO Model on healthcare professionals’ learning outcomes and patients’ health outcomes were found [7, 42]. To the best of our knowledge, no previous or in-progress reviews have focused on investigating the influence of key learning conditions on the effectiveness of ECHO-affiliated programs on the development of competencies in healthcare professionals.

Aim and research questions

The aim of this mixed methods systematic review (MMSR) is to identify, appraise and synthesize the available quantitative (QUAN) and qualitative (QUAL) evidence regarding the effectiveness of the ECHO Model and the influence of key learning conditions on the development of competencies in healthcare professionals. This systematic review seeks to answer the following three research questions:

What is the effectiveness of the ECHO Model on the development of competencies in healthcare professionals? (QUAN question)
What key learning conditions influence the development of competencies in healthcare professionals participating in ECHO? (QUAL question)
To what extent, and under which key learning conditions, is the ECHO Model most effective in developing competencies in healthcare professionals? (Mixed methods [MM] question)

This systematic review protocol was developed in accordance with the JBI methodology for MMSR [45], and is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) 2015 checklist [46, 47] together with the PRISMA 2020 updated guidance [48] (see additional file 2). The review protocol was submitted to the PROSPERO database on October 21, 2020 (ID Number: 197579) and is pending registration.

Approach

We will use a convergent segregated approach [45, 49] to extract and synthesize data from the QUAN, QUAL and MM included studies (see Fig. 2). With this type of design, the QUAN and the QUAL extracted data will first be analyzed separately using different synthesis methods (QUAN descriptive statistics and synthesis, intervention effect estimates and QUAL thematic synthesis) and then the findings of both syntheses will be merged for the purposes of comparison and complementarity [50].

Eligibility criteria

The eligibility of studies will be screened based on the following criteria structured according to the PICO (Population, Intervention, Comparator, Outcomes) mnemonic for the QUAN component, and the PICo (Phenomenon of Interest, Context) mnemonic for the QUAL component [45].

Population

We will include studies conducted with healthcare professionals, regardless of their profession group (such as community health providers, long-term care providers, pharmacists, physicians, psychologists, primary care physicians, nurse practitioners and social workers) and area of practice (such as family medicine and general practices, geriatric care, addiction and psychiatric services, pain management and pediatric care). We will exclude studies that involve pre-licensure healthcare professional students only, since entry-to-practice competencies may differ from the standards and levels of competencies in clinical practice.

Intervention

For inclusion in the QUAN component of this review we will consider studies of implemented affiliated ECHO programs targeting healthcare professionals, where the effectiveness of the program was investigated. We defined ECHO programs as “a technology-enabled learning model in which a mentor with specialized knowledge and expertise provides interactive and case-based guidance to a group of mentees for the purpose of strengthening their skills and knowledge to provide high-quality healthcare” [7]. The following six inclusion criteria will be used for considering programs as “ECHO-affiliated”: (1) using a technology-enabled platform (videoconferencing sessions); (2) having a health-focused objective; (3) implementing a hub-and-spoke framework with generalists in one or more locations (spokes) and specialists at a different location (hub) with a ratio of more than 1:1; (4) using case-based learning (case presentations and discussions); (5) using interactive mentorship; and (6) including a didactic component. Technology-enabled learning models that are not “ECHO-affiliated” will be excluded. We will include studies of implemented ECHO programs addressing any type of health conditions and topics (e.g., chronic pain, co-occurring disorders of mental health and substance use disorders, delirium, diabetes, infection diseases).

Comparator

We will consider for inclusion studies with all types of comparator(s), including active and passive comparators.

Outcomes

The New World Kirkpatrick Model [51], an evaluation model for educational programs that is frequently used in the healthcare professions, was chosen to operationalize and categorize the outcomes of this systematic review. The model includes the following four levels of outcomes: (1) reaction—i.e., the degree to which participants find the educational program favorable, engaging and relevant to their practice; (2) knowledge—i.e., the degree to which participants acquire the intended knowledge based on their participation in the educational program; (3) behaviour—i.e., the degree to which participants apply what they learned in the educational program in their practice; and (4) results—i.e., the degree to which targeted outcomes occur as a result of the educational program.

Based on this categorization and the focus given to the development of competencies in healthcare professionals, the QUAN component of this review will consider studies that include at least one of the following critical or important [52] outcome measures:

Critical outcome

Competency development (i.e., the efficient mobilization and orchestration of a cluster of internal resources in clinical practice): studies assessing changes in healthcare professionals’ competencies on the basis of their participation in ECHO will be considered for inclusion. This may include objective outcome measures (e.g., simulation, observations in educational settings) or subjective outcome measures (e.g., self-reported competence, perceived competence, self-reported confidence, self-reported abilities, perceived skills, self-efficacy) of competencies (or overall competence) [53].

Important outcomes

Reaction (Kirkpatrick’s Level 1): healthcare professionals’ views and reactions to ECHO including outcome measures of satisfaction (e.g., general satisfaction, program acceptability, perceived usefulness, perceived benefits) and participation (e.g., number of online participations and degree of retention).
Knowledge (Kirkpatrick’s Level 2): healthcare professionals’ knowledge acquisitions on the basis of their participation in ECHO, including objective and subjective outcome measures of knowledge.
Behaviour (Kirkpatrick’s Level 3): application of healthcare professionals’ new knowledge acquisitions in clinical practice on the basis of their participation in ECHO, including objective outcome measures (e.g., interventions or tool utilization, initiation of treatment, patient charts, care performance indicators from administrative database) or subjective outcome measures (e.g., perceived clinical performance, self-reported change in care plan) of clinical performance.
Results (Kirkpatrick’s Level 4): changes in patients’ health due to changes in the practice behaviour of healthcare professionals participating in ECHO, including objective outcome measures (e.g., measures recorded in patient charts or administrative databases and access to care) or subjective outcome measures (e.g., measures from patient self-reports) of health indicators. The health indicators targeted for this review will include outcomes such as health behaviours, health status and wellbeing, including physical and psychological health, social functioning and treatment outcomes.

Phenomena of interest

For the QUAL component of this review, we will consider for inclusion studies in which conditions influencing—positively or negatively—the development of competencies in healthcare professionals participating in ECHO are investigated. In line with our conceptual representation of the ECHO Model, these conditions may include educational (educational intervention components), personal (e.g., sociodemographic, motivation, engagement, ability to use technology, anterior knowledge, satisfaction regarding programs and workplaces, openness to change, attitude and feeling of social recognition) and contextual conditions (interpersonal, institutional, organizational, community and sociopolitical influences) of learning.

It may also include the benefits and challenges of the program (e.g., resources constraints, change in patient health, integrating new knowledge in work practices, challenging clinical situations) in the views of ECHO participants (e.g., experiences, perceptions, perspectives). Studies exploring factors influencing competency development, facilitators and barriers to learning, knowledge transfer in practice (e.g., changes in clinical practices, behaviours or interventions) and lifelong learning processes (e.g., developmental stages, transition, unexpected or critical moments in learning processes) will be considered as well.

Context

For inclusion in this review (QUAN and QUAL components), we will consider studies conducted in any type of clinical setting (e.g., ambulatory clinics, community health centres, home care, hospitals, long-term care facilities, primary care services), geographic location (e.g., rural or remote areas, urban located services and carceral healthcare) or country.

Type of studies

This review will consider QUAN, QUAL and MM studies. Regarding QUAN studies, we will include both experimental studies (randomized controlled trials [RCTs], cluster RCTs, crossover RCTs) and quasi-experimental studies (e.g., non-randomized controlled trials, cluster non-randomized controlled trials, cohort study with control group). QUAL studies will include designs such as phenomenology, grounded theory, ethnography, narrative inquiry, interpretative description, exploratory and action research. All types of QUAL data sources will be included in this review (e.g., individual semi-structured interviews, observations, field notes, focus groups). MM studies will be considered for inclusion if the QUAN or the QUAL or both components meets the inclusion criteria mentioned above. All types of MM designs will be considered for inclusion (e.g., convergent, sequential exploratory, sequential explanatory).

This review will be limited to empirical studies in peer-reviewed journals as well as in the grey literature. Only full-text papers of English or French-language studies will be included. Case reports, study protocols, discussion papers, editorials and knowledge synthesis papers (e.g., MMSR, narrative reviews, rapid reviews, realist reviews, systematic reviews, scoping reviews) will be excluded. We will consider for inclusion studies published from 2003 onwards as the initial pilot-ECHO program was launched in 2003 [32].

Search strategy for identification of studies

A systematic search strategy was developed in consultation with a scientific health librarian (DZ) and was reviewed by a second librarian. The search strategy was built with the objective of locating studies on ECHO-affiliated programs exclusively. Therefore, the search combined specific words and expressions related to the ECHO Model (e.g., Extension for community healthcare outcomes, ECHO, Project ECHO, SCAN-ECHO, TeleECHO). Given the absence of standardized indexing, we exploded the ECHO specific terminology with search term groups covering the following three domains: (1) healthcare professionals (population); (2) technology-enabled learning model (intervention); and (3) hub-and-spoke model linking specialists with general healthcare professionals (context). The ECHO-specific terminology and the search terms used for each domain were developed based on a previous systematic review on the impact of the ECHO Model [7]. A pilot search was executed using the search terminology to ensure its all-inclusiveness and the accuracy of the original articles retrieved.

The following bibliographic databases were searched: CINAHL COMPLETE (from 1937 onwards), All EBM Reviews (from 1991 onwards), EMBASE (from 1974 onwards), MEDLINE (from 1946 onwards) and PsycINFO (from 2002 onwards). The search strategy was translated and adapted for each database using controlled vocabulary (MeSH, EMTREE and others) and free text searching. A publication date filter was applied to find the articles published from 2003 onwards. Additional file 3 presents the complete search strategy.

A forward citation tracking procedure—i.e., search articles that cited the included studies—will also be performed in Google Scholar. Duplicates will be removed with EndNote X9 (© 2020 Clarivate Analytics, Boston, MA, USA; www.endnote.com) using a Bramer method for de-duplication of database search results for systematic reviews [54]. Automated search updates will be set up in each database to ensure the inclusion of the latest publications in the field of the ECHO Model. Sources of unpublished studies and grey literature will include ProQuest Dissertations and Theses and DART Europe E-theses Portal. For QUAN studies only, ClinicalTrials.gov and WHO International Clinical Trials Registry Platform will be searched via the Cochrane Library. The list of references of existing reviews on the ECHO model [7, 42] will be manually scrutinized.

Study selection, appraisal, and data extraction

A three-stage process including study selection, appraisal and data extraction will be followed. Each stage of the process will be conducted by teams of two independent reviewers. Teams will be formed based on the experience of each reviewer in a field (e.g., screening titles and abstracts, conducting a systematic review of effectiveness, assessing heterogeneity, coding studies, using qualitative research software).

During the selection of studies, quality appraisal and data extraction, the study authors will be contacted for additional information regarding eligibility criteria if necessary. Also, any disagreements that arise between the reviewers will be resolved through discussion until consensus is reached. In the event of a persistent disagreement, a third reviewer will be convened to make a decision.

Selection of studies

Following the search, all identified records will be uploaded into the Covidence systematic review software (© 2019 Veritas Health Innovation Ltd, Melbourne, Australia; www.covidence.org). Reviewers will first independently screen titles and abstracts according to eligibility criteria. Then, full-text articles of all studies deemed eligible will be retrieved and assessed in detail against the eligibility criteria by two independent reviewers. The results of the search will be presented in a PRISMA flow diagram [48]. Excluded studies with reasons for exclusion will be reported in table form as well.

Assessment of methodological quality

All included studies will be assessed by two independent reviewers for methodological quality using the Mixed Methods Appraisal Tool (MMAT) version 2018 (© 2019 MMAT, Quebec, Canada; http://mixedmethodsappraisaltoolpublic.pbworks.com) [55–57].

The MMAT was specifically developed to assess the methodological quality of various study designs, including MM studies, and proposes a list of 25 criteria based on five categories of empirical studies. In the current literature on critical appraising tools, authors discourage using an overall numerical score since the latter does not shed light on any methodological issues and provides equal weight to all criteria [52, 56, 58]. The MMAT was revised in accordance with these recommendations and therefore offers guidance on presenting details of the ratings for each criterion of included studies.

All studies, regardless of their methodological quality, will undergo data extraction and synthesis. The results of critical appraisal will be reported in narrative form and will also be taken into consideration when discussing the final integrated findings of the review. A detailed presentation of the ratings for each criterion of the included studies will be reported in table form.

Data extraction and management

Based on the JBI standardized QUAN [59] and QUAL [60] data extraction tools, two separate data extraction forms were developed specifically for this systematic review. These data extraction forms will be iteratively validated by the entire team of reviewers to ensure their completeness and clarity. Before data extraction, the forms will be tested on a total of six randomly selected articles from the search strategy (two studies of QUAN method only, two studies of QUAL method only and two studies of MM) and amended accordingly.

For the QUAN component, data will be extracted from QUAN and MM studies (QUAN component only) included in the review by two independent reviewers and will be managed with the Covidence systematic review software (© 2019 Veritas Health Innovation Ltd, Melbourne, Australia; www.covidence.org). Data extraction will include the following specific details:

First and corresponding author(s) information, publication year and country.
Study funding source(s).
Study objective(s) and design.
Study population, and health care setting.
Time of study, method(s) of data collection.
Planned and actual sample sizes.
Participation and response rate.
Results of significance to the QUAN review objective (outcomes measures of competency development and Levels 1 to 4 of Kirkpatrick’s Model), including details on outcomes (definition, time points measured, missing data) and measurement (name of tool, unit of measurement, scales).

In addition, we will use the Guideline for Reporting Evidence-based practice Educational interventions and Teaching (GREET) 2016 checklist [61] to collect informally reported evidence [62] regarding implemented ECHO programs; thus enabling an in-depth examination of each educational intervention components (e.g., health conditions or topics addressed in the program, learning objectives, duration and frequency of videoconference sessions, instructors’ degree, modes of delivery, materials, incentives). Informal evidence about implemented ECHO programs will be mainly captured from the method and/or discussion sections [62] of studies included in the QUAN component of the review exclusively.

For the QUAL data extraction, full-text of QUAL and MM studies (QUAL component only) included in the review will be uploaded into the MAXQDA Standard software version 2020.1 (© 1995–2020 MAXQDA, distribution by VERBI GmbH, Berlin, Germany; www.maxqda.com). MAXQDA is a convenient software to conduct QUAL data extraction and in-depth analysis from a variety of sources (e.g., PDF, text audio and video files, figures and images) [63].

The QUAL data extracted will include specific details about first and corresponding author(s), study funding source(s), publication year and geographical location. We will also extract data regarding study aim and research question(s), population(s), context, philosophical or theoretical foundations, methodology and method(s) for data collection. Study results that are relevant to answer the QUAL question of this review on key learning conditions will be extracted for further analysis. Where possible and if appropriate, we will extract figures, images or schemas and their related summarized text as well. Data extraction of QUAL studies and MM studies (QUAL component only) included in the review will be performed by two reviewers, with each of them subject to repeated independent readings.

Synthesis and integration of QUAN and QUAL findings

In accordance with the JBI convergent segregated approach to MMSR [45], a fourth-step procedure will be performed at the synthesis and integration stage. This procedure will involve separate QUAN and QUAL synthesis followed by integration of the QUAN findings and QUAL findings. Table 1 summarizes this procedure and is detailed in the sections below.

Table 1

Planned procedures in a convergent segregated approach, adapted from Pluye et al. [50].
REVIEW QUESTION	INPUT	TECHNIQUE	OUTPUT
What is the effectiveness of the ECHO Model of continuing tele-education on competency development in healthcare professionals? (QUAN question)	Informal evidence: ECHO programs’ components (from QUAN studies and QUAN component of MM studies) QUAN results: Critical outcome: Competency development Important outcomes: Corresponding to Kirkpatrick’s four levels of program evaluation (reaction, knowledge, behaviour and results)	Descriptive synthesis and meta-analysis (QUAN synthesis)	Description of ECHO programs’ components Effect sizes of ECHO programs on critical and important outcomes
What key learning conditions influence the development of competencies in healthcare professionals participating in ECHO? (QUAL question)	QUAL results: Educational, personal and contextual conditions influencing competency development in healthcare professionals participating in ECHO (from QUAL studies and QUAL component of MM studies)	Thematic synthesis (QUAL synthesis)	Major themes relating to the key learning conditions that influence the development of competencies in healthcare professionals participating in ECHO
To what extent, and under which key learning conditions is the ECHO Model is most effective in developing competencies in healthcare professionals? (MM question)	Resultant QUAN and QUAL findings (from QUAN, QUAL and MM studies)	Juxtaposition of findings using a matrix approach (MM synthesis) for the purpose of exploring the correspondence between the effectiveness of included ECHO programs’ (that we anticipate obtaining from the QUAN synthesis) and the influence of key learning conditions on competency development in healthcare professionals (that we aimed to gather from the QUAL synthesis)	MM inferences providing an interpretation of the success or failure of the ECHO Model on competency development in healthcare professionals

First step: Descriptive synthesis

Prior to undertaking QUAN and QUAL syntheses, we will summarize all included QUAN, QUAL and MM regarding the characteristics of studies, population, context and settings in table format. Informal evidence on the components of educational programs from included QUAN and MM studies (QUAN component only) will be summarized in a table as well.

Second step: Quantitative synthesis

Summary intervention effects and meta-analyses

To evaluate the effectiveness of the ECHO Model on competency development in healthcare professionals, we will synthesize all intervention effect estimates for each outcome of interest using meta-analyses. Meta-analyses will be undertaken using the Cochrane Collaboration Review Manager RevMan version 5.4.1 (© 2020 The Cochrane Collaboration, London, UK; www.training.cochrane.org). All results will be expressed with 95% confidence intervals (CI). Statistically significant results will be defined with a two-sided alpha of 0.05.

At least two studies will need to contribute to a meta-analysis for it to be conducted. To minimize clinical heterogeneity, we will favor the pooling of studies in which the comparators (active or passive) and the outcomes of interest (objective or subjective outcome measures) are similar. Based on existing systematic reviews on the ECHO Model [7, 42], we currently expect that all outcomes of interest were mostly measured as continuous variables and using different instruments. As such, we will use an inverse variance approach for continuous outcomes and random effect models in all meta-analyses. All results will be expressed as standardized mean differences. We will interpret the significance of effect sizes using Cohen’s classification (< 0.2 = negligible; 0.2—0.49 = small; 0.5—0.8 = moderate; > 0.8 = large) [64].

In each meta-analysis, we will assess statistical heterogeneity, which is the inconsistency in intervention effect estimates between studies that is not due to chance, using the X² test and the I² statistic. A statistically significant p value at the X² test or an I² statistic > 50% will be considered as indicative of high statistical heterogeneity. Where statistical pooling is not possible the findings will be presented in narrative form including tables and figures to aid in data presentation.

Subgroup and sensitivity analysis

We plan to carry out subgroup analyses to investigate potential statistical heterogeneity sources when four or more studies are included in a single meta-analysis (two in each subgroup). If there are a sufficient number of studies, we will explore the following potential effect modifiers:

Intervention: topic(s) or health condition(s) targeted in the program;
Population: professional group(s) participating in the program;
Context: practice setting of participating healthcare professionals.

Sensitivity analyses will also be conducted to exclude studies deemed at high risk of bias.

Assessment of reporting biases

Based on Cochrane recommendations [52], we will assess reporting biases using funnel plots if more than 10 studies are included in a single meta-analysis. We will follow the guidelines regarding funnel plot asymmetry as described in the Cochrane Handbook for Systematic Reviews of Interventions version 6.1 [52].

Third step: Qualitative synthesis

Thirdly, a thematic synthesis [65] of the QUAL findings from the QUAL and MM studies (QUAL component only) will be undertaken by two independent reviewers using the MAXQDA Standard software version 2020.1 (© 1995–2020 MAXQDA, distribution by VERBI GmbH, Berlin, Germany; https://www.maxqda.com). This will assist in understanding what key learning conditions influence the development of competencies in healthcare professionals participating in ECHO.

To ensure consistency and coherence between each reviewer’s coding, we will first use a deductive approach to data reduction (organization of the mass of QUAL data and discarding of irrelevant data) [66]. To achieve this, a set of three conceptual categories that depict the nature of key learning conditions (educational, personal and contextual conditions) [34] will serve as an initial path for organizing the QUAL raw data. However, this stage of the review will remain somehow iterative as these broad categories may be modified in case of similarity and recurrence of data that do not match the conceptual categories.

During this process, each study will be read and reread to enable the reviewer to familiarize themselves with the study results and the methods used. Then, two independent reviewers will scrutinize the results of each study for meaningful units with regards to the QUAL review question. Data will be coded line by line to assign the content of each line or sentence under one of the established conceptual categories. Any changes or differences arising from the coding system will be resolved with two reviewers and we will bring in a third reviewer in case of a persistent disagreement or uncertainty. Irrelevant information will be kept in a in an independent category to ensure that we have access to it later if required, as unexpected findings may call for reexamination of some data previously considered unnecessary.

In the following step, the lead review author (GC) will examine each code to identify recurrence or patterns in the data. Subsequently, a first set of themes and subthemes will be created, through assembling the data and displaying the data into the form of a hierarchy. These initial sets of themes and subthemes will then be subjected to a synthesis by examining the themes within and across each study, based on similarity in meaning. Themes will be refined and renamed with the review authors (JP and GR) until the synthesized findings provide an answer to the QUAL review question. The QUAL findings will be presented in narrative form including each emergent theme with supporting quotes drawn from the included studies.

Fourth step: Integration of quantitative and qualitative evidence

At the final stage, findings of each synthesis will be compared and contrasted to produce an overall configured synthesis and interpretation of the effectiveness of the ECHO Model and the influence of key conditions on competency development in healthcare professionals. This will involve QUAN and QUAL evidence being juxtaposed for the purpose of interrogating simultaneously the extent to what, and under which key learning conditions, the ECHO Model is most effective in developing competencies in healthcare professionals. Integration of both sets of evidence will be attained by performing a comparison integration strategy [67], which will assist in considering where the QUAN and QUAL findings of the review agree (correspondence, similarities), offer complementary information or are in contradiction (disagreement or dissonance).

The comparison integration strategy will be performed using a matrix approach [68, 69]. A matrix will allow us to closely map the findings of the review on a same table and to conduct a side-by-side comparison to identify matches and mismatches [69]. We will organize the matrix in a theme-by-effect size configuration [70]. This juxtaposition will assist the review team in exploring heterogeneity between the QUAN findings (educational components and effect size measures) and in interpreting, on the basis of the QUAL findings (themes), under which key learning conditions some ECHO programs were effective—or more effective—and some were not. Where juxtaposition is not possible, the findings will be presented in narrative form. An example of the planned matrix inspired by Candy et al MMSR [71] can be found in additional file 4.

To support the use of the review findings in informed decision-making, we will apply the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach [72] jointly with the GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative research) approach [73], to assess and transparently communicate how much confidence can be placed in the cumulative evidence.

This protocol outlines the process to be undertaken for a MMSR aiming to gather evidence on the ECHO Model. This review is necessary to establish in which key learning conditions the model is most effective in developing competencies in healthcare professionals. The proposed review was developed in accordance with the current recommendations in the literature on MMSR [45, 49, 68, 69, 74], entailing a rigorously and thoughtfully articulated convergent segregated design. The evidence from QUAN, QUAL and MM studies drawn from the literature will be utilized to illustrate the best ways to implement the ECHO Model as an effective intervention and will be useful in guiding future research and educational practice in this area. These findings may be applied internationally across all disciplines. It is expected that the review findings will be valuable to researchers, academicians and other stakeholders (e.g., policymakers, healthcare professionals, educators) in improving aspects of further implementations of the ECHO Model.

To the best of our knowledge, no review has selected, appraised and synthesized evidence from QUAN (and the QUAN component of MM studies) and QUAL (and the QUAL component of MM studies) studies for the overarching objective of comparison and complementarity between both strands of findings. We therefore believe that the synthesis and integration of evidence from a range of diverse methodologies in a systematic way should help shed new light on the ECHO Model. Furthermore, no review has focused on extracting evidence on how ECHO-affiliated programs are implemented in various contexts, sometimes reflecting the particular health conditions and learning objectives addressed by a given program [44]. This MMSR protocol was developed in order to elucidate these variations of the ECHO Model in practice and to provide a clear understanding of which critical components may lead to better outcomes in healthcare professionals and patients.

This MMSR has some potential limitations. First, the search strategy of this review was built with the objective of including English and French-language empirical studies exclusively, which means that other sources of existing information and language will be excluded from the outset. While other forms of evidence might be an interesting addition to the state of knowledge in terms of comprehensiveness, this decision was made on the grounds that there is a considerable amount of eligible and primary studies. Indeed, based on a previous review that focused on a similar body of literature [7], 52 empirical studies from peer-reviewed articles were included by these authors.

Second, all studies meeting the eligibility criteria will be included in the review, meaning that no studies will be excluded on the basis of low methodological quality. Since risk of bias and lack of rigour are primary concerns when undertaking a MMSR [45], all included studies will be critically appraise using the latest version of the MMAT. To ensure transparency and enhance rigour, a table clearly indicating the ratings for each criterion of all included studies will be developed using the MMAT and will be reported in full.

Third, although this review will be restricted to ECHO-affiliated programs only in order to limit clinical diversity, we anticipate that programs’ characteristics of included QUAN studies will vary in terms of population (targeted professional groups), topics (targeted health condition or disease), participants’ exposure to the intervention (frequency and duration of the program) and context of care delivery (e.g., community services, primary care and hospital). However, the richness of the QUAL findings on learning, individual and contextual conditions that we expect to gather will aid in explaining any potential variation in the program effect on our critical and important QUAN outcomes. Another downside of including studies of ECHO-affiliated programs exclusively is that the findings generated from this review may not be generalizable to other technology-enabled learning models. However, we believe that these models, which differ in their identity and layers, do not meet the aim and scope of this review.

Finally, considering that this review builds on a MM approach, a potential challenge to consider is the complexity associated with the incorporation of evidence derived from a range of research designs into one single synthesis [74]. To address any practical issues during synthesis and integration, the process will follow the available guidance for MMSR, and the overall interpretation of the evidence will be reviewed independently by each member of the review team. Full immersion of the lead author in the entirety of the evidence base, extended reflection with potential explanations in case of divergences between QUAN and QUAL findings, and transparency in the reporting of the integration process will provide greater insight into and understanding of the evidence.

In conclusion, the dissemination plan will include standard and innovative (e.g., website portals, social media, Project ECHO Networks, knowledge exchange events with clinical administrators, healthcare professionals, key stakeholders) means of ensuring that the findings of the review are communicated regionally, nationally and internationally, and that they are accessible to a broad audience. Results of this MMSR—including the QUAN, QUAL and MM findings—will be disseminated through publication in relevant peer-reviewed journals, and will be presented at suitable fora including academic, scientific and professional conferences in the field of ICTs and CE in the health professions. The strengths, limitations, and recommendations to improve the development and evaluation of further implementations of ECHO-affiliated programs will be discussed in the completed review.

CE Continuing Education

CERQual Confidence in the Evidence from Reviews of Qualitative research

CI Confidence Interval

CINAHL Cumulative Index to Nursing & Allied Health Literature

ECHO Extension for Community Health Outcomes

GRADE Grading of Recommendations Assessment, Development, and Evaluation

GREET Guideline for Reporting Evidence-based practice Educational interventions and Teaching

JBI Joanna Briggs Institute

MM Mixed Methods

MMAT Mixed Methods Appraisal Tool

MMSR Mixed Methods Systematic Review

PICO Population – Intervention – Comparator – Outcomes

PICo Phenomenon of Interest - Context

PRISMA-P Preferred Reporting Items for Systematic review and Meta-Analysis Protocols

QUAL Qualitative

QUAN Quantitative

RCT Randomized Controlled Trial

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material

No data are yet available.

Competing interests

Review authors have no competing interests.

Funding

This review received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Authors’ contributions

GC conceptualized the review, designed the review, drafted the article and is the guarantor of the review. JC, GF and MAMC conceptualized the review, designed the review, and drafted the article. All other authors helped design the review, draft and revise the article. All authors read and approved the final manuscript.

Acknowledgments

This MMSR protocol was developed as part of the doctoral studies of the first author (GC) and for which she received scholarships from the following: Université de Montréal Hospital Centre Foundation, Quebec’s Ministry of Education and Higher Education Scholarship Program, Fonds de recherche du Québec – Société et Culture (FRQSC), Research Chair in Innovative Nursing Practices, Center for Innovation in Nursing Education, and Équipe FUTUR (FRQSC).

DJA holds a clinical scientist career award from the Fonds de Recherche du Québec (FRQS) and acknowledges research support from Health Canada and the Quebec’s Ministry of Health and Social Services.

QNH held a postdoctoral fellowship from the Fonds de recherche du Québec – Santé (FRQS).

Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, Fineberg H, Garcia P, Ke Y, Kelley P, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. The Lancet. 2010;376(9756):1923–58.
Fernandez N, Dory V, Ste-Marie LG, Chaput M, Charlin B, Boucher A. Varying conceptions of competence: an analysis of how health sciences educators define competence. Med Educ. 2012;46(4):357–65.
Watson R, Stimpson A, Topping A, Porock D. Clinical competence assessment in nursing: A systematic review of the literature. J Adv Nurs. 2002;39(5):421–31.
Fukada M. Nursing Competency: Definition, Structure and Development. Yonago Acta Med. 2018;61(1):1–7.
Rizany I, Hariyati RTS, Handayani H. Factors that affect the development of nurses' competencies: A systematic review. Enferm Clin. 2018;28(Suppl 1):154–7.
Batt AM, Tavares W, Williams B. The development of competency frameworks in healthcare professions: A scoping review. Adv in Health Sci Educ. 2019;25:913–87.
McBain RK, Sousa JL, Rose AJ, Baxi SM, Faherty LJ, Taplin C, Chappel A, Fischer SH. Impact of Project ECHO Models of Medical Tele-Education: A Systematic Review. J Gen Intern Med. 2019;34(12):2842–57.
Axley L. Competency: A concept analysis. Nurs Forum. 2008;43(4):214–22.
Tardif J. L'évaluation des compétences: Documenter le parcours de développement. Montreal: Chenelière-éducation; 2006.
Charette M, Goudreau J, Alderson M. An evolutionist analysis of competence. Recherche en soins infirmiers. 2014;116(1):28–39.
Cooper E. Creating a culture of professional development: a milestone pathway tool for registered nurses. J Contin Educ Nurs. 2009;40(11):501–8.
Manley K, McCormack B, Wilson V. International practice development in nursing and healthcare. Oxford: Blackwell publishing; 2008.
Meretoja R, Koponen L. A systematic model to compare nurses' optimal and actual competencies in the clinical setting. J Adv Nurs. 2012;68(2):414–22.
Pepin J, Goudreau J, Lavoie P, Belisle M, Blanchet Garneau A, Boyer L, Larue C, Lechasseur K. A nursing education research framework for transformative learning and interdependence of academia and practice. Nurse Educ Today. 2017;52:50–2.
Babenko O, Koppula S, Daniels L, Nadon L, Daniels V. Lifelong learning along the education and career continuum: meta-analysis of studies in health professions. J Adv Med Educ Prof. 2017;5(4):157–63.
Fleet LJ, Kirby F, Cutler S, Dunikowski L, Nasmith L, Shaughnessy R. Continuing professional development and social accountability: A review of the literature. J Interprof Care. 2008;22(Suppl 1):15–29.
Price S. C. R: The Importance of Continuing Professional Development to Career Satisfaction and Patient Care: Meeting the Needs of Novice to Mid- to Late-Career Nurses throughout Their Career Span. Adm Sci. 2017;7(2):17.
Institute of Medicine (US) Committee on Planning a Continuing Health Professional Education Institute: Redesigning Continuing Education in the Health Professions. Summary, vol. 2020. Washington, DC: National Academies Press; 2010.
Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, Davis D, Odgaard-Jensen J, Oxman AD. Continuing education meetings and workshops: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009;2009(2):Cd003030.
Kononowicz AA, Woodham LA, Edelbring S, Stathakarou N, Davies D, Saxena N, Tudor Car L, Carlstedt-Duke J, Car J, Zary N. Virtual Patient Simulations in Health Professions Education: Systematic Review and Meta-Analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019;21(7):e14676.
McLoughlin C, Patel KD, O'Callaghan T, Reeves S. The use of virtual communities of practice to improve interprofessional collaboration and education: Findings from an integrated review. J Interprof Care. 2018;32(2):136–42.
Ranmuthugala G, Plumb JJ, Cunningham FC, Georgiou A, Westbrook JI, Braithwaite J. How and why are communities of practice established in the healthcare sector? A systematic review of the literature. BMC Health Serv Res. 2011;11:273.
Rouleau G, Gagnon M-P, Côté J, Payne-Gagnon J, Hudson E, Dubois C-A, Bouix-Picasso J. Effects of E-Learning in a Continuing Education Context on Nursing Care: Systematic Review of Systematic Qualitative, Quantitative, and Mixed-Studies Reviews. J Med Internet Res. 2019;21(10):e15118–8.
Wang R, DeMaria S Jr, Goldberg A, Katz D. A Systematic Review of Serious Games in Training Health Care Professionals. Simul Healthc. 2016;11(1):41–51.
Rouleau G, Gagnon M-P, Côté J, Richard L, Chicoine G, Pelletier J: Virtual patient simulation to improve nurses’ relational skills in a continuing education context: A convergent mixed methods study. BMC Nurs 2020, PREPRINT (Version 1). Available from: https://www.researchsquare.com/article/rs-43757/v1 [accessed 11.10.2020].
Coventry TH, Maslin-Prothero SE, Smith G. Organizational impact of nurse supply and workload on nurses continuing professional development opportunities: An integrative review. J Adv Nurs. 2015;71(12):2715–27.
Project ECHO. University of New Mexico, School of Medicine [https://echo.unm.edu/ [accessed 28.09.2020]].
Arora S, Kalishman S, Thornton K, Dion D, Murata G, Deming P, Parish B, Brown J, Komaromy M, Colleran K, et al. Expanding access to hepatitis C virus treatment–Extension for Community Healthcare Outcomes (ECHO) project: disruptive innovation in specialty care. Hepatology. 2010;52(3):1124–33.
Arora S, Kalishman S, Dion D, Som D, Thornton K, Bankhurst A, Boyle J, Harkins M, Moseley K, Murata G, et al. Partnering urban academic medical centers and rural primary care clinicians to provide complex chronic disease care. Health Aff. 2011;30(6):1176–84.
Arora S, Kalishman SG, Thornton KA, Komaromy MS, Katzman JG, Struminger BB, Rayburn WF, Bradford AM. Project ECHO: A Telementoring Network Model for Continuing Professional Development. J Contin Educ Health Prof. 2017;37(4):239–44.
Arora S, Murata GH, Thornton K, Parish B, Jenkusky SM, Dunkelberg JC, Hoffman RM, Komaromy M: 380 Project ECHO (Extension for Community Healthcare Outcomes): Knowledge Networks Expand Access to Hepatitis C (HCV) Treatment with Pegylated Interferon and Ribavirin in Rural Areas and Prisons. Care Is As Safe and Effective As a University HCV Clinic. Gastroenterology 2008, 134(4):A-50-A-51.
Arora S, Thornton K, Jenkusky SM, Parish B, Scaletti JV. Project ECHO: Linking university specialists with rural and prison-based clinicians to improve care for people with chronic hepatitis C in New Mexico. Public Health Rep. 2007;122(Suppl 2):74–7.
Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, Parish B, Burke T, Pak W, Dunkelberg J, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199–207.
Cianciolo AT, Regehr G. Learning Theory and Educational Intervention: Producing Meaningful Evidence of Impact Through Layered Analysis. Acad Med. 2019;94(6):789–94.
Bandura A. Self-efficacy: The exercise of control. New York: Freeman; 1997.
Vygotsky LS. Mind in Society: The Development of Higher Psychological Processes. Cambridge: Harvard University Press; 1978.
Lave J, Wenger E. Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press; 1991.
Wenger E. Communities of practice: Learning, meaning, and identity, 1 edn. Cambridge: Cambridge University Press; 1998.
Bordage G. Conceptual frameworks to illuminate and magnify. Med Educ. 2009;43:312–9.
Bronfenbenner U. The ecology of human development: Experiments by nature and design. Cambrigde: Harvard University Press; 1979.
McHale S, Dotterer A, Kim J-Y. An Ecological Perspective on the Media and Youth Development. Am Behav Sci. 2009;52:1186–203.
Zhou C, Crawford A, Serhal E, Kurdyak P, Sockalingam S. The Impact of Project ECHO on Participant and Patient Outcomes: A Systematic Review. Acad Med. 2016;91(10):1439–61.
Socolovsky C, Masi C, Hamlish T, Aduana G, Arora S, Bakris G, Johnson D. Evaluating the role of key learning theories in ECHO: A telehealth educational program for primary care providers. Prog Community Health Partnersh. 2013;7(4):361–8.
Faherty LJ, Rose AJ, Chappel A, Taplin C, Martineau M, Fischer SH. Assessing and Expanding the Evidence Base for Project ECHO and ECHO-Like Models: Findings of a Technical Expert Panel. J Gen Intern Med. 2020;35(3):899–902.
Lizarondo L, Stern C, Carrier J, Rieger CG, Salmond K, Apostolo S, Kirkpatrick J, Loveday P H: Chap. 8: Mixed Methods Systematix Reviews. In: JBI Manual for Evidence Synthesis. Edited by Aromataris E, Munn Z. JBI; 2020. Available from: https://synthesismanual.jbi.global [accessed 19.10.2020].
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. The PRISMA Group: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Sys Rev 2015, 4(1).
Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ. 2015;349:g7647.
Page MJ, Moher D, Bossuyt P, Boutron I, Hoffmann T, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan EP, et al: PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews [Internet]. MetaArXiv 2020. Available from: osf.io/preprints/metaarxiv/gwdhk [accessed 23.10.2020].
Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Rev. 2017;6(1):61.
Pluye P, Hong QN, Vedel I: Toolkit for Mixed Studies Reviews. Version 3: Department of Family Medicine, McGill University, and Quebec-SPOR SUPPORT Unit, Quebec, Canada; 2016. Available from: http://toolkit4mixedstudiesreviews.pbworks.com [accessed 18.06.2020].
Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s for levels of training evaluation. Alexandria: ATD Press; 2016.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Page LT M, editors: Cochrane Handbook for Systematic Reviews of Interventions. Version 6.1 (updated September 2020): The Cochrane Collaboration; 2020. Available from: [accessed 02.10.2020].
Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1–15.
Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. JMLA. 2016;104(3):240–3.
Hong QN, Gonzalez-Reyes A, Pluye P. Improving the usefulness of a tool for appraising the quality of qualitative, quantitative and mixed methods studies, the Mixed Methods Appraisal Tool (MMAT). J Eval Clin Pract. 2018;24(3):459–67.
Hong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, Gagnon MP, Griffiths F, Nicolau B, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. J Clin Epidemiol. 2019;111:49–59.e41.
Pluye P, Gagnon MP, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. Int J Nurs Stud. 2009;46(4):529–46.
Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed. J Clin Epidemiol. 2011;64(1):79–89.
Tufanaru CMZ, Aromataris E, Campbell J, Hopp L: Chap. 3: Systematic reviews of effectiveness. In: JBI Manual for Evidence Synthesis. Edited by Aromataris E MZ: JBI; 2020. Available from: https://synthesismanual.jbi.global [accessed 22.09.2020].
Lockwood CPK, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D: Chap. 2: Systematic reviews of qualitative evidence. In: JBI Manual for Evidence Synthesis. Edited by E A, Z M: JBI; 2020. Available from: https://synthesismanual.jbi.global [accessed 18.10.2020].
Phillips AC, Lewis LK, McEvoy MP, Galipeau J, Glasziou P, Moher D, Tilson JK, Williams MT. Development and validation of the guideline for reporting evidence-based practice educational interventions and teaching (GREET). BMC Med Educ. 2016;16(1):237.
Sutcliffe K, Thomas J, Stokes G, Hinds K, Bangpan M. Intervention Component Analysis (ICA): A pragmatic approach for identifying the critical features of complex interventions. Syst Rev. 2015;4(1):140.
Kuckartz U, Rädiker S: Analyzing Qualitative Data with MAXQDA: Text, Audio and Video: Springer Nature Switzerland AG 2019; 2019. Available from: https://link.springer.com/book/10.1007%2F978-3-030-15671-8 [accessed 03.06.2020].
Cohen J. A power primer. Psychol Bull. 1992;112(1):155–9.
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45.
Miles MB, Huberman AM, Saldaña J. Qualitative data analysis: A methods sourcebook. 4 ed. Los Angeles: SAGE; 2020.
Pluye P, Kaur N, Granikov V, Garcia Bengoechea E, Tang D. Mixing phases, results and data in patient oriented research. International Journal of Multiple Research Approaches. 2018;10(1):41–56.
Harden A, Thomas J, Cargo M, Harris J, Pantoja T, Flemming K, Booth A, Garside R, Hannes K, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance series-paper 5: Methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.
Hong QN, Rees R, Sutcliffe K, Thomas J. Variations of mixed methods reviews approaches: A case study. Res Syn Meth 2020:1–17.
Guetterman TC, Fetters MD, Creswell JW. Integrating Quantitative and Qualitative Results in Health Science Mixed Methods Research Through Joint Displays. Annals Fam Med. 2015;13(6):554–61.
Candy B, King M, Jones L, Oliver S. Using qualitative synthesis to explore heterogeneity of complex interventions. BMC Med Res Methodol. 2011;11(1):124.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924.
Lewin S, Booth A, Glenton C, Munthe-Kaas H, Rashidian A, Wainwright M, Bohren MA, Tunçalp Ö, Colvin CJ, Garside R, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: Introduction to the series. Implement Sci. 2018;13(Suppl 1):2.
Thompson Coon J, Gwernan-Jones R, Garside R, Nunns M, Shaw L, Melendez-Torres GJ, Moore D. Developing methods for the overarching synthesis of quantitative and qualitative evidence: The interweave synthesis approach. Res Syn Meth. 2020;11(4):507–21.