We will use a three-phase approach (see Figure 1 for details) including a critical analysis of existing literature overviews, a systematic review of systematic reviews, and a series of systematic reviews and meta-analyses. We will conduct this project following the recommendations of Cochrane Reviews, and drafted it according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) review guidelines.
Eligibility criteria
In the first two phases of the project, according to a PICOS (Population, Intervention, Comparison, Outcomes, Study design) with knowledge products, studies that meet the following criteria will be considered for inclusion:
Population
The population of interest is individuals or stakeholders participating in healthcare delivery, specifically, healthcare providers, caregivers, and end users (i.e., patients or clients with any conditions).
Intervention
We will consider any type of strategy aiming to implement a knowledge product. The EPOC taxonomy will be used to classify implementation strategies identified (e.g., audit and feedback, educational meetings, interprofessional education) (15).
Knowledge product
As suggested by Pinnock et al. [30], a clear distinction has been made between the implementation strategies and the knowledge products implemented. A knowledge product could be, but not limited to, a decision support tool, a clinical practice guideline, a policy brief, or a decision-making tool, a one-pager (simple, iconographic, infographic), or a health intervention (technological, pharmacological, behavioral, or management). Only implementation studies in which both are reported will be considered in our project.
Comparison
Any comparator will be considered. This includes studies in which two or more implementation strategies for knowledge products are compared (head to head); and studies in which there is no comparator they are with none.
Outcome(s)
In phases 1 and 2, we will mainly consider outcomes related to implementation strategies including, but not limited to, the measures of adherence/fidelity to the use of knowledge products, their acceptability, adoption, appropriateness, feasibility, adaptability, implementation costs, penetration/reach and sustainability [11]. As for phase 3, we will additionally consider the outcomes related to patients (psychosocial, health behavioral, and clinical outcomes) or healthcare professionals (psychosocial, health behavioral, and performance outcomes) that the knowledge products aim to improve.
Study design
This eligibility criterion will be specific to each phase of the project:
- Phase 1: We will consider any review of reviews on implementation strategies for knowledge products (i.e., a knowledge synthesis in which included studies are any literature reviews).
- Phase 2: We will consider only systematic reviews (i.e., any study in which authors performed a narrative and/or quantitative synthesis of experimental studies on implementation strategies for knowledge products using a comprehensive and reproducible approach).
- Phase 3: Only RCTs will be considered.
Setting
Any health domain addressed in primary healthcare will be considered. A review will be included if primary healthcare is covered.
Search strategy
We will perform comprehensive searches within five electronic databases: MEDLINE (Ovid), EMBASE, Web of Science, PsycINFO, CINAHL, and the Cochrane Library from their inception to October 2019. The search strategy will be designed by an information specialist using “knowledge transfer” and “implementation strategies” headings (see Appendix 1). For efficiency considerations, we will not build a new search strategy to identify RCTs, rather we plan in identifying them from the systematic reviews that we will select in our overview. However, to avoid missing any recently published RCTs in the field, we will update our search strategy by looking for primary studies published after the most recent included reviews. Searches will be conducted within the same electronic databases as in earlier phase. Study designs other than RCTs, for example systematic reviews, quasi-experimental studies and observational studies, will be excluded.
There will be neither language nor literature search date restrictions. The references listed in all eligible studies will also be manually searched in order to identify additional relevant ones.
Data management
We will merge the citations identified from five electronic databases mentioned above in EndNote Software. Then, we will identify and remove the duplicates. The unique citations will be considered for the selection process.
Selection of studies
Two assessors will independently contribute to all steps of the selection process. References identified from relevant electronic databases will be merged, and duplicates removed to obtain a database including unique citations for the study selection process. Assessors will discuss the inclusion criteria to ensure mutual understanding, and pilot test on five percent of unique citations identified to confirm that the evaluation process is reliable. The pilot section will be considered conclusive if the kappa statistic referring to the agreement between assessors is greater than 0.7 [31].
Pairs of assessors will then independently screen for titles and abstracts based on inclusion criteria specific to each phase outlined above. In case of doubt, the citations will be included and considered for full-text reading. Assessors will then independently screen the full texts of references retained at the first step. Any disagreement will be resolved by consensus or by involving a third assessor.
Assessors will include overviews for Phase 1 and systematic reviews for Phase 2, based on the eligibility criteria mentioned above. For Phase 3, the following procedure will be adopted for the identification of relevant RCTs from each review considered: first, we will go through the list including primary studies to identify RCTs that are clearly reported as such by the review’s authors. In the case RCTs are not, we will select the complete list of primary articles included. Removal of duplicates and screening based only on the study design will be done by two independent assessors.
Data extraction
For each phase, a pilot data extraction will be done on 5% of selected articles. The data extraction will be pursued only when the pilot is conclusive. This process will be independently performed by two assessors. Discrepancies between them will be resolved by a third author.
Phase 1: The following data will be extracted: the first author’s name; the year of publication; the focus of the overview, based on the scope of PICO elements; the number and name of databases consulted; the period of the literature search; the strategies used to update the literature; the type and number of reviews that are included; the type of reviews (Cochrane, non-Cochrane, both); the strategies to deal with overlap between reviews; the strategies to deal with conflicting results of reviews; the quality assessment of reviews; the type of synthesis performed (narrative, meta-analysis or both); the assessment of the quality of evidence, and the limitations reported by overview authors.
Phase 2: We will extract data concerning the following elements:
- Review details: First author’s name, year of publication, type, objectives, registration information.
- Literature search details: Number and names of electronic databases, search for grey literature, search period, restrictions, design and number of primary studies.
- Characteristics of participants: Name, profile, total number, age (mean, median and/or range), gender (percentage of men or women).
- Characteristics of implementation strategies: Components described according to the EPOC taxonomy, knowledge products implemented (e.g., clinical guidelines, health intervention).
- Characteristics of outcomes: Name, type of measurement tools used (self-administered, objective, both). We will use Proctor et al. taxonomy to guide the classification of outcomes [11].
- Assessment of study quality: Name of the tool used, result.
- Type of synthesis: Narrative, meta-analysis, or both.
- Assessment of publication bias: Method used (funnel plots, statistical tests, both), and treatment of it if present.
- Assessment of the quality of evidence: Name of the tool used, level of evidence.
- Conclusion of reviews: Information reported by authors when they conclude (e.g., main results and limitations).
Phase 3: For each RCT considered in this phase, we will extract data concerning the following elements:
- Characteristics of the study: First author’s name, year of publication, country, language, healthcare domain, setting, study design, sampling, and recruitment method.
- Characteristics of participants: Size of eligible population for both healthcare providers and end users, total sample size, sample sizes in implementation strategy and control groups, unit of allocation, mean age or range, sex, race/ethnicity (percentage of Caucasians), socioeconomic level of the sampled population and profile.
- Characteristics of implementation strategies (description of the intervention): Components (according to the EPOC taxonomy), underlying framework, content, number of participating centers, mode of delivery (e.g., in person, online, etc.), actual duration (e.g., number of hours), actual frequency (e.g., number of sessions), methods used (e.g., interactive or didactic techniques, web applications), course materials used (e.g., web capsules, textbooks), adherence/fidelity, and content of the intervention offered to control group.
- Characteristics of knowledge products implemented: Name, format and content.
- Characteristics of outcomes and analysis methods used: names of interest outcomes, measurement tools, range of scale and analysis methods used, unit of analysis, intention to treat analysis or not, retention rates, and effect measures of implementation strategies.
- Effects of implementation strategies: Any information useful to estimate the effect size and its 95% confidence interval will be extracted. For continuous outcomes, we will consider the standardized mean difference as effect size requiring the sample size, mean score and standard deviation in each group studied. For dichotomous outcomes, we will consider the odds ratio as effect size requiring the sample size and number of events in each group studied. We will also collect the value of effect size (crude and adjusted) and its confidence interval estimated in the studies. Information about confounding factors will as well be extracted.
When necessary, corresponding or first authors will be contacted to obtain information about missing data in their studies.
Quality assessment of studies
The quality of overviews retrieved in Phase 1 will not be assessed since there is no methodological tool designed for this purpose. Moreover, the objective of this phase is not to draw any conclusions on the effectiveness of implementation strategies from these overviews.
For Phase 2, we will assess the methodological quality of selected systematic reviews using the AMSTAR 2 tool [32]. Those meeting the criteria of good or medium quality will be retained for the rest of the process, i.e., will be considered for the next steps of the review of reviews and for the identification of RCTs in Phase 3.
As for Phase 3, RCTs extracted from systematic reviews will be assessed with the Cochrane Collaboration’s tool for assessing risk of bias in RCTs [33].
Both the quality assessment of systematic reviews as the assessing risk of bias for RCTs will be performed independently by two authors. In a pilot assessment of two or four studies included, they will agree on a common understanding of the definitions, criteria, and guidelines provided in the tool to achieve a more objective assessment. The assessment of all studies included will do once the pilot test is conclusive. When needed, a third reviewer will be invited to help reach consensus.
Data synthesis
Phase 1: We will perform a descriptive analysis using counts (number and percentage) to summarize data collected followed by carrying out a critical analysis on the latter considering the following methodological elements: literature search, methodological limitations reported in overviews, overlap between reviews included in overviews, types of data synthesis, quality of reviews included in overviews, and level of evidence.
Phase 2: We will produce a descriptive Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of the selection process for all included reviews [34]. We will then describe characteristics of the latter, populations, implementation strategies, and outcomes assessed. We will also analyze the data on publication bias and quality of evidence assessment. Finally, evaluation data of included reviews from AMSTAR 2 tool [32] will be presented using a graph and table.
Phase 3: First, we will classify RCTs according to the type of knowledge product being implemented. These may include but not limited to: clinical guidelines, decision support tools, research summaries, one-pagers (simple, iconographic, infographic), or other health interventions (technological, pharmacological, behavioral, or management).
Second, we will present a descriptive flowchart of the RCTs included in each meta-analysis, according to the PRISMA guidelines [34]. We will describe by frequencies (number and percentage) the characteristics of studies, populations, implementation strategies and outcomes. Risk of bias levels will also be described for all included RCTs.
Finally, for each outcome and knowledge product, we will determine if there are sufficient data to perform a meta-analysis. If not, we will conduct a narrative synthesis on the effects of implementation strategies, and include in tables and figures to aid in data presentation. If so, we will do a meta-analysis. For each outcome and knowledge product, the random effect model will be used to estimate the pooled effect size of an implementation strategy and its 95% confidence interval, as we anticipate heterogeneity among the RCTs concerning the types of implementation intervention and population [35, 36]. For dichotomous data, effect size will be expressed as a risk ratio or odds ratio; for continuous data, it will be provided as standardized mean difference if different measures are used for the same outcome. For cluster RCTs, the analysis will be adjusted for clustering to avoid unit-of-analysis errors [36]. The influence of clinical and methodological heterogeneity on the observed effects will be discussed. Subgroup analyses will be carried out if necessary according to characteristics of the studies, the populations and the implementation interventions mentioned above. Statistical heterogeneity will be assessed using the Higgins’ I square statistic [37, 38]. A funnel plot will be generated to assess publication bias if 10 or more studies are included in the meta-analysis [39]. Statistical tests for funnel plot asymmetry (e.g., Egger’s regression, Begg’s test, and Harbord’s test) will be performed where appropriate [36, 39].
Sensitivity analyses will be carried out excluding the RCTs with high risk of bias from pooled effect size estimates. For each outcome and knowledge product, we will also explore the influence of each RCT by removing its individual effect size from the pooled estimation. These different analyses will allow us to evaluate the robustness of our results.
Finally, if data consistency permits, we also plan to do a network meta-analysis to rank the different implementation strategies according to their effectiveness of amplitudes, while obtaining more accurate effect size estimates through indirect comparisons [40, 41].
Assessment of the quality of evidence
The quality of evidence for each outcome and knowledge product will be assessed with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) to reduce the misinterpretation of our review’s results [42]. The GRADE tool is based on five criteria for each individual study, namely risk of bias, indirectness of evidence, data heterogeneity, imprecision of effect size estimates, and risk of publication bias [42]. Overall assessment will be rated very low, low, moderate, and high for each outcome [42].