2.1. Study design
The design is a bibliometrics study in the knowledge synthesis field. We followed systematic review methods guidance for searching, study selection, and data extraction.
2.2 Eligibility criteria
We included overviews published in any language and reports were translated by one of the authors (French, Spanish, German, Mandarin), where needed. We restricted our search to reports published from January 1, 2000 to December 31, 2018. Given that the Cochrane Handbook chapter on overviews was first published in 2009, we did not expect to identify overviews published prior to 2000.
We reasoned that overviews may exist as stand-alone reports or also packaged as part of a clinical practice guideline and health technology assessment; accordingly, we developed eligibility criteria for both circumstances.
Inclusion criteria for overviews:
- Aims to primarily focus on synthesizing systematic reviews with or without meta-analyses (but the overview may also include primary studies).
- Uses a systematic literature search found in the main body of the paper (i.e. search strategy includes text words and MeSH terms in at least two databases).
- Has a methods section in the main body of the paper (not just in the abstract).
- Examines the effects of health interventions or clinical treatments.
We excluded overviews that based their results solely on primary studies and methodological studies. Reports that were editorials, letters, or comments were also excluded. Overviews of risk, exposure, prevention, measurement instruments, quality indicators, diagnostic, screening or prognostic research were excluded. We excluded protocols of overviews.
Inclusion criteria for clinical practice guidelines (CPGs) and health technology assessments (HTAs):
- Uses an ‘overview of review’ approach to inform their recommendations (i.e. the CPG aims to primarily include, synthesise and present the results of the systematic reviews; but may also include primary studies).
2.3 Search
Overviews were retrieved using a validated search filter for overviews [15] from MEDLINE (Ovid), Epistemonikos and the Cochrane Database of Systematic Reviews. In an empirical methods study of the retrieval sensitivity of 6 databases, the combination of Medline and Epistemonikos retrieved 95.2% of all systematic reviews [16]. We are therefore confident that using both a validated search filter and this proven database combination will retrieve the majority of published overviews of healthcare interventions and clinical treatment in the literature.
We searched from 2000 to December 2020 (Appendix A). The Epistemonikos search was limited to the category called “Broad Syntheses” which includes overviews of systematic reviews, heath technology assessments and clinical practice guidelines. Epistemonikos contains reports from electronic databases as well as the grey literature, including: PubMed, Embase, The Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Literatura Latinoamericana y del Caribe en Ciencias de la Salud, Database of Abstracts of Reviews of Effects, the Campbell Collaboration’s online library, the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports, Cochrane Database of Systematic Reviews, and the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) Evidence Library.
2.4 Screening and study selection
The initial search results were imported into Excel 2010 for screening. A pilot screening of the first 19 papers was conducted in duplicate by all screeners to ensure high levels of agreement and common definitions of coding. We screened the titles and abstracts against the stated eligibility criteria first, then eligible full-text articles were reviewed for inclusion. Two independent reviewers screened reports at the title and abstract, and then again at the full text stage, then compared their results. Discrepancies were resolved by consensus, and arbitration by a third reviewer when necessary.
Categorized as ‘other sources’ in our flowchart (Figure 1), 237 overviews published between 2013 and 2016 were previously identified and screened [15, 17] using identical methods (Appendix A), and added to our list of included studies.
2.5 Data extraction
We extracted data on overviews published between 2000 and 2018 (n=541) to enable a feasible extraction of data. Data extraction was piloted on 20 studies by all authors independently to identify any missing variables, come to agreement on coding definitions, and refine/reword the items. Discrepancies in the piloting phase were discussed and consensus reached by two authors. Full data extraction was performed independently by one investigator, and checked by a second reviewer.
We extracted the medical field of the overview; year of publication; country and continent (Europe, North America, Asia, Australia, South America, Africa) of corresponding author; number of authors; journal of publication; number of included studies; search date; inclusion of meta-analysis; and funding status. We also extracted the number of total patients in the included systematic reviews. If the total number was not reported, we added up total patients across all systematic reviews from a table of characteristics. We extracted the number of times the overview had been cited per year via the Google Scholar database, and the 2-year impact factor for the top journals from Web of Science. Impact factor, as calculated by Clarivate Analytics, is a measure of the average number of times articles from a two-year time frame have been cited in a given year. We checked whether the journal was open access via the Directory of Open Access Journals (https://doaj.org/). We also extracted data on the terminology used to describe overviews in the title (overview of reviews, meta-review, review of reviews, or any related synonyms).
The medical field categorisation used was the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD), a medical classification list by the World Health Organization (WHO) (https://www.who.int/standards/classifications/classification-of-diseases). Time it took to conduct the overview was not reported in the overviews. Therefore, to estimate the time for each overview’s search end date to date of publication (i.e. publication lag), we calculated the difference between the initial literature search date and publication date using the month and day function in Excel 2010.
2.6 Data analysis
Descriptive analysis using frequencies and percentages were performed for categorical data and median and interquartile range (IQR) for continuous data. Citation density was used instead of citation count to avoid bias towards older articles. Citation density was defined as total number of citations divided by the number of years a study has been published. Citation density (the dependent variable) was positively skewed; therefore, the natural log transformation was obtained to approach a normal distribution.
Mean publication frequencies by year were calculated and presented in figures. The distribution of overviews by medical discipline and terms used to describe overviews was described. A free version of StatPlanet Plus (http://www.statsilk.com/software/statplanet) software was used to create the interactive world map to visualize the distribution of all overviews among different countries.
A multivariate regression was conducted to determine the association between predictor variables (number of authors, open versus closed journal access, number of overviews with meta-analyses, funding status, sample size, CAM topic) and two dependent variables (citation density and impact factor). Statistical significance was set at p < .05. Statistical analyses were performed using Stata version 13 (StataCorp LP) using two commands manova and mvreg.