2.1 Design and protocol
Our methods protocol was previously published in BMJ Open [23], and in the Open Science Framework (https://osf.io/rju4f/). The study follows systematic review methods guidance for searching, study selection, data extraction, and critical appraisal. As this is a methods study, no relevant research reporting checklists exist. Formal ethical approval was not required as primary data were not collected. Our raw data files have been uploaded to the repository Open Science Framework (https://osf.io/8rxnp/ with DOI 10.17605/OSF.IO/8RXNP).
2.2 Search
CPGs were retrieved from the Turning Research Into Practice (TRIP) and Epistemonikos databases over a two-year period (January 1, 2017 to December 31, 2018). This time period was selected for current relevance and to limit the number of CPGs screened. In Epistemonikos, we selected the "Broad syntheses" filter for retrieval of CPGs (Appendix 1). Epistemonikos includes citations retrieved from the following databases: Cochrane Database of Systematic Reviews; PubMed; Embase; CINAHL (The Cumulative Index to Nursing and Allied Health Literature); PsycINFO; LILACS (Literatura Latinoamericana y del Caribe en Ciencias de la Salud); DARE (Database of Abstracts of Reviews of Effects); the Campbell Library; the JBI Database of Systematic Reviews and Implementation Reports; and the EPPI-Centre Evidence Library. As the TRIP database only contains CPGs, we downloaded all records without restricting by study type. TRIP retrieves guidelines from over 289 journal publications and has recently migrated all content from AHRQ’s Clinical Guidelines Clearinghouse (www.guidelines.gov), which lost funding on July 16, 2018 (Jon Brassey, personal communication, April 10, 2018).
2.3. Study selection
References retrieved from TRIP and Epistemonikos were imported into a single EndNote file and de-duplicated. Subsequently, the retrieved citations were randomly sorted using Microsoft Excel’s RAND function and, screened at the full text level, using a form designed in Microsoft Excel (2013).
Screening to identify citations meeting our inclusion criteria was conducted independently by two authors, starting with the lowest random number, until 50 guidelines were included. This sample size was chosen to be large enough to include a variety of clinical conditions. All authors involved in study selection screened ten studies as a calibration exercise to establish agreement in definitions of eligibility criteria. Discrepant decisions were resolved by discussion with a senior author.
2.4 Eligibility criteria
In addition to a requirement for publication between January 1, 2017 and December 31, 2018, we defined clinical practice guidelines according to the following inclusion criteria:
- Pertain to the management or treatment of any clinical condition. CPG recommendations for management may include, for example, recommendations for lifestyle modifications, when to implement or adjust therapy, choice of therapy including treatment combinations, and ways to prevent harms associated with therapy.
- Produced by a group or organisation (i.e. not authored by one person).
- Contain at least two explicit recommendations for treatment or management of a condition.
- Contain a description of their methodology within the guideline or in supporting documents (e.g. inclusion/exclusion criteria, key terms used to search, number of databases searched, number of authors used to select studies, methods used to create recommendations, or quality/risk of bias assessment).
- Contain a reference list (i.e. a bibliography).
If more than one publication from the same organisation or author group was identified, we included the most recent version of the guideline.
Exclusion criteria:
CPGs without recommendations or that focus solely on screening or diagnosis were excluded. CPGs were also excluded for the following reasons:
- The full text is unavailable.
- It is designed for local use only (e.g. in a single health facility or single regional health service).
- It is designed for use restricted to hospitalized patients or patients in long-term care facilities.
- It aims to provide recommendations for patterns of use of medications (e.g. guidance about adherence to medications) but not treatment choice.
The eligibility criteria were piloted by all data extractors (CL, DS, BM, CR, TL, SG) independently on a sample of ten guidelines retrieved from the search to ensure consistent application. Once the guidelines were screened and included, we attempted to retrieve any supplementary files, methods documents, published systematic reviews, or any other documentation supplementary to the guideline.
2.5 Definitions of evidence syntheses
We classified approaches to evidence synthesis according to the following definitions.
Literature reviews or non-systematic reviews are summaries of the literature on a particular topic that are not developed systematically. A literature review may identify a search strategy but does not fulfill the minimum requirements of a systematic process as listed below.
Systematic review. A review of evidence is considered systematic if it reports, at a minimum:
- A research question or questions formatted using PICOS (participants, interventions, comparisons, outcomes, and study design);
- Eligibility criteria for all included study types;
- Full search strategy for at least one database (i.e. keywords reported and a full search strategy reported in an appendix);
- Search strategy in the main body of the manuscript (i.e. not only in the abstract) using 2 or more electronic databases; and
- Process for selecting/screening studies (e.g. number of authors; independent process).
Systematic or non-systematic reviews may contain a pairwise meta-analysis or network meta-analysis. A pairwise meta-analysis is a traditional meta-analysis in which the effect estimates of two interventions are compared directly, following a judgment that the included studies are sufficiently similar to warrant pooling. A network meta-analysis compares multiple interventions using both direct comparisons of interventions within randomised trials and indirect comparisons across trials based on a common comparator [9].
An overview of reviews aims to primarily identify, include, and synthesise the results of secondary analyses (reviews, systematic reviews, guidelines, or health technology assessments) and may or may not have used systematic methods as outlined above [10-12].
2.6 Outcomes
The study’s primary outcome consisted of the numbers and proportions of recommendations within CPGs that were based on the following types of evidence syntheses:
1. Systematic reviews without meta-analysis
2. Systematic reviews with pairwise meta-analysis
3. Systematic reviews with network meta-analyses
4. Overviews of systematic reviews
We also evaluated the number of CPGs using either a systematic or non-systematic process to gather, assess, and synthesise evidence (Figure 1; and section 2.10).
The secondary outcomes, calculated as numbers or proportions, are:
5. CPGs that cited a Cochrane review or overview
6. CPGs that report using GRADE methodology
7. CPGs that report using other systems evaluating the strength of the recommendation and type of tool used (e.g. American Heart Association [24])
8. CPGs that report using a level of evidence system and type of system used
9. Currency of the CPG (calculated by the time from last search to full publication)
10. CPGs that report conflicts of interest disclosures by authors
2.7 Data extraction
Data from 50 guidelines were extracted. Each included CPG was first examined to determine whether reviews, or overviews of reviews, were used and cited in support of one or more of the guideline's recommendations (yes or no for each review type). If yes, we evaluated all treatment or management recommendations citing each review type.
A data extraction form was developed in Microsoft Excel (2013). Ten CPGs were independently extracted by six authors and then discussed to come to consensus about definitions and to calibrate the coding (Appendix 2). Full data extraction was completed independently by two authors and compared. Any discrepancies were discussed, and a senior author arbitrated conflicts. A senior author checked that the data was consistently coded across similar or related items.
Data extracted at the guideline level included: our primary and secondary outcomes, name of the guideline, year of publication, country, the organisations or commissioning agency (publisher), type of publisher (government, medical society, university, other [specify]), aim of the guideline, publishing journal (if applicable), open source/paywall, the date of the last search for evidence to be included in the guideline, funding, declaration of conflicts of interest by developers, stakeholder affiliation with/honoraria from pharmaceutical companies, target population (general population, or specific subpopulations such as those identified by age (e.g. children and adolescents, adults of any age, older adults), sex/gender, or co-morbidities), and scope (pharmacological or non-pharmacological treatment (e.g. surgical, medical device).
If a review or overview was cited within a recommendation, we also looked for evidence that critical appraisal of the review or overview was conducted, and recorded which tool was used (e.g. Assessing the Methodological Quality of Systematic Reviews (AMSTAR) 1 or 2 [25], Risk of Bias Assessment Tool for Systematic Reviews (ROBIS) [26]).
2.8 Gaps in evidence supporting a recommendation
If a guideline did not cite a Cochrane systematic review, we searched the Cochrane Database of Systematic Reviews using the keywords used in the main search strategy of the guideline. We noted whether a Cochrane systematic review or overview could have been identified and used to inform the recommendations by checking the search dates of the CPG to ensure it could have been included.
2.10 Risk of bias assessment of the review process for informing the guideline recommendations
To determine if a systematic process was used to gather, assess and synthesise evidence to inform recommendations, we used the following four criteria:
- Explicit statement of the questions or objectives reported in terms of PICOS (Populations, Interventions, Comparisons, Outcomes, and Study design) elements.
- Eligibility criteria reported for all included study designs.
- A systematic search conducted (i.e. two or more databases searched).
- Process reported for selecting/screening studies (e.g. number of authors, independent process).
We considered these criteria to be the minimum that can be used by a CPG to reduce bias and limitations when gathering evidence to inform recommendations. We also assessed whether the CPG working group reported the following methods:
- A full search strategy for at least one database (i.e. keywords reported and a full search strategy reported in an appendix).
- Assessment of the quality/risk of bias of the review or overview supporting/refuting the recommendation.
- Assessment of primary studies for quality/risk of bias.
We adapted these risk of bias items from the ROBIS tool, which comprehensively assesses the risk of bias of a systematic review [26]. The tool includes items relating to internal validity and classifies them in the following domains: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings.
2.10 Open access
All data management and study processes were conducted and recorded in the Open Science Framework.
2.11 Data analysis
We calculated the number and frequencies of citations of reviews and overviews, and their characteristics, found in the 50 included guidelines. We described and tabulated all primary and secondary outcomes. Additional information was put into appendices. To estimate the time taken to conduct each guideline, we calculated the difference between the initial literature search date and publication date using the month and day function in Excel 2013.
As sufficient studies were collected to make meaningful comparisons (≥10), we performed a chi-square test of independence to examine the relation between using the GRADE approach and whether the guideline used a systematic process (Model 3; post hoc analysis). Dependent categorical variables were type of organization (medical association, pharmaceutical, government, no funding, not reported), scope (narrow, broad), and continent (Europe, North America, Intercontinental). We also performed a chi-square test of independence to examine the relationship between GRADE use and type of funder, and CPG having conducted a systematic process and type of funder. We planned to explore whether the characteristics of CPGs differed in terms of pharmacological vs. non-pharmacological scope. However, there were too few CPGs with these characteristics to permit reliable comparisons (≤10 in each group). We formally tested the associations using a chi-square test for one independent variable with 2 levels with categorical dependent variables in R.