Bibliometric study of ‘overviews of systematic reviews’ of health interventions: evaluation of prevalence, citation and impact factor

Background: Overviews synthesising the results of multiple systematic reviews help inform evidence-based clinical practice. In this rst of two companion papers, we evaluate the bibliometrics of overviews, including their prevalence and factors affecting citation rates and journal impact factor. Methods: We searched MEDLINE, Epistemonikos and Cochrane databases. We included overviews that: (a) synthesised reviews, (b) conducted a systematic search, (c) had a methods section, and (d) examined a healthcare intervention. Multivariate regression was conducted to determine the association between citation density, impact factor and 6 predictor variables. Results: We found 1218 overviews published from 2000 to 2020; the majority (73%) were published in the most recent 5-year period. We extracted a selection of these overviews (n=541; 44%) dated from 2000 to 2018. The 541 overviews were published in 307 journals; Cochrane Database (8%), PLOS ONE (3%) and Sao Paulo Medical Journal (2%) were the most prevalent. The majority (70%) were published in journals with impact factors between 0.05 and 3.97. We found a mean citation count of 10 overviews per year, published in journals with a mean impact factor of 4.4. In multivariate analysis, overviews with a high number of citations and impact factors had more authors, larger sample sizes, were open access and reported the funding source. Conclusions: An 8-fold increase in the number of overviews was found between 2009 and 2020. We identied 332 overviews published in 2020, which is equivalent to 1 overview published per day. Overviews perform above average for the journals in which they publish.


Background
Bibliometric analysis measures the impact of articles using metrics such as the journal impact factor and number of citations [1]. A bibliometric analysis can serve to identify in uential articles that have shaped medical practice, policy decisions, and presented new research ideas [2]. The journal impact factor measures the average number of citations of an article in the journal over the previous year. Citation analysis provides an objective measurement of how often scientists use speci c published work.
Although citations have limitations [3], they may offer vital information on how often scienti c data are used and referenced, and provide one source of information to help prioritise research investment [2].
Several investigations have found that evidence syntheses, such as systematic reviews with metaanalysis, received a higher citation rate compared to other study designs [4][5][6][7][8]; but typically represented only 4-6% of research output [8,9]. It is unclear whether 'overviews of systematic reviews' (henceforth called overviews) are cited equally as highly as systematic reviews, but it stands to reason that they might considering they are a type of evidence synthesis.
The number of systematic reviews has reached epidemic proportions [10]; from none in 1987 [11]; 7 per day in 2003 [5]; 11 a day in 2010 [12]; and 22 per day in 2014 [13]. Overviews have grown in number in tandem with the growth in systematic reviews, and they aim to appraise and synthesise results of systematic reviews. One investigation found a ve-fold increase in the publication of overviews from 2004 to 2011 [14]. The current number of overviews, and whether the trend in annual publication rate is increasing is unknown.
To advance the development of overviews, research is required to understand epidemiological aspects such as annual prevalence and factors that may in uence the citation of overview publications. This paper is the rst of two companion papers. In this rst paper we present our evaluation of the bibliometric characteristics of overviews, and evaluate factors affecting citation rates and journal impact factor. In a second paper, we will present the assessment of whether two or more overviews overlapped in topic.

Study design
The design is a bibliometrics study in the knowledge synthesis eld. We followed systematic review methods guidance for searching, study selection, and data extraction.

Eligibility criteria
We included overviews published in any language and reports were translated by one of the authors (French, Spanish, German, Mandarin), where needed. We used Google translate to translate articles in languages other than the ones listed, and when authors were not native speakers of French, Spanish, German, or Mandarin. We restricted our search to reports published from January 1, 2000 to December 2020. Given that the Cochrane Handbook chapter on overviews was rst published in 2009, we did not expect to identify overviews published prior to 2000.
We reasoned that overviews may exist as stand-alone reports or also packaged as part of a clinical practice guideline and health technology assessment; accordingly, we developed eligibility criteria for both circumstances.
Inclusion criteria for overviews: • Aims to primarily focus on synthesizing systematic reviews with or without meta-analyses (but the overview may also include primary studies).
• Uses a systematic literature search found in the main body of the paper (i.e. search strategy includes text words and MeSH terms in at least two databases).
• Has a methods section in the main body of the paper (not just in the abstract).
• Examines the effects of health interventions or clinical treatments.
We excluded overviews that based their results solely on primary studies and methodological studies.
Reports that were editorials, letters, or comments were also excluded. Overviews of risk, exposure, prevention, measurement instruments, quality indicators, diagnostic, screening or prognostic research were excluded. We excluded protocols of overviews.
Inclusion criteria for clinical practice guidelines (CPGs) and health technology assessments (HTAs): Uses an 'overview of review' approach to inform their recommendations (i.e. the CPG aims to primarily include, synthesise and present the results of the systematic reviews; but may also include primary studies).

Search
Overviews were retrieved using a validated search lter for overviews [15] from MEDLINE (Ovid), Epistemonikos and the Cochrane Database of Systematic Reviews. In an empirical methods study of the retrieval sensitivity of 6 databases, the combination of Medline and Epistemonikos retrieved 95.2% of all systematic reviews [16]. We are therefore con dent that using both a validated search lter and this proven database combination will retrieve the majority of published overviews of healthcare interventions and clinical treatment in the literature. papers was conducted in duplicate by all screeners to ensure high levels of agreement and common de nitions of coding. We screened the titles and abstracts against the stated eligibility criteria rst, then eligible full-text articles were reviewed for inclusion. Two independent reviewers screened reports at the title and abstract, and then again at the full text stage, then compared their results (GS, YWZ, CL, LL, PHY, MD, PN). Discrepancies were resolved by consensus, and arbitration by a third reviewer when necessary.
Categorized as 'other sources' in our owchart (Figure 1), 237 overviews published between 2013 and 2016 were previously identi ed and screened [15,17] (Appendix A), and added to our list of included studies.

Data extraction
We extracted data on overviews published between 2000 and 2018 (n=541) to enable a feasible extraction of data. Data extraction was piloted on 20 studies by all authors independently to identify any missing variables, come to agreement on coding de nitions, and re ne/reword the items. Discrepancies in the piloting phase were discussed and consensus reached by two authors. We extracted the medical eld of the overview; year of publication; country and continent (Europe, North America, Asia, Australia, South America, Africa) of corresponding author; number of authors; journal of publication; number of included studies; search date; inclusion of meta-analysis; and funding status. We also extracted the number of total patients in the included systematic reviews. If the total number was not reported, we added up total patients across all systematic reviews from a table of characteristics. We extracted the number of times the overview had been cited per year via the Google Scholar database, and the 2-year impact factor for the top journals from Web of Science. Impact factor, as calculated by Clarivate Analytics, is a measure of the average number of times articles from a two-year time frame have been cited in a given year. We checked whether the journal was open access via the Directory of Open Access Journals (https://doaj.org/). We also extracted data on the terminology used to describe overviews in the title (overview of reviews, meta-review, review of reviews, or any related synonyms).
The medical eld categorisation used was the 10th revision of the International Statistical Classi cation of Diseases and Related Health Problems (ICD), a medical classi cation list by the World Health Organization (WHO) (https://www.who.int/standards/classi cations/classi cation-of-diseases). Time it took to conduct the overview was not reported in the overviews. Therefore, to estimate the time for each overview's search end date to date of publication (i.e. publication lag), we calculated the difference between the initial literature search date and publication date using the month and day function in Excel 2010.

Data analysis
Descriptive analysis using frequencies and percentages were performed for categorical data and median and interquartile range (IQR) for continuous data. Citation density was used instead of citation count to avoid bias towards older articles. Citation density was de ned as total number of citations divided by the number of years a study has been published. Citation density (the dependent variable) was positively skewed; therefore, the natural log transformation was obtained to approach a normal distribution.
Mean publication frequencies by year were calculated and presented in gures. The distribution of overviews by medical discipline and terms used to describe overviews was described. A free version of StatPlanet Plus (http://www.statsilk.com/software/statplanet) software was used to create the interactive world map to visualize the distribution of all overviews among different countries.
A multivariate regression was conducted to determine the association between predictor variables (number of authors, open versus closed journal access, number of overviews with meta-analyses, funding status, sample size, complementary and alternative therapies (CAM) topic) and two dependent variables (citation density and impact factor). Statistical signi cance was set at p < .05. Statistical analyses were performed using Stata version 13 (StataCorp LP) using two commands manova and mvreg.

Search results
From a total of 16,610 records retrieved from the MEDLINE, Cochrane Database of Systematic Reviews, and Epistemonikos databases, and 237 records retrieved from other sources (Figure 1), 2410 duplicates were removed. Of the remaining 14,437 records, 11,481 were excluded at the title/abstract stage, and 1738 were excluded at the full text stage. A total of 1218 overviews were included that met our eligibility criteria, and of these, 541 overviews published between 2000 and 2018 (44%) were data extracted (data set can be accessed at https://osf.io/38bkg/). Many studies did not have a methods section, did not conduct a systematic search, and did not search for and include systematic reviews (Figure 1).

Prevalence of overviews
We  (Table 1).  SRs: systematic reviews; SD: standard deviation; IQR: interquartile range We found an average citation rate of 90 (SD ±219.7) over 9 years, or 10 citations per overview per year. When examining overviews published in the top 50 journals by impact factors (≥ 7.86), we found that 50 overviews (9%; 50/537) received 18% of citations (range 4-1250). Nine overviews remained un-cited. A total of 108 out of 537 overviews (20%) were cited 100 times or more. An article having 100 or more citations is regarded as a 'classic' [18]. The topics of the six overviews with citations counts of >1000 included two on diseases of the musculoskeletal system and connective tissue, two on factors in uencing health status and contact with health services, and one each on pregnancy, childbirth and the puerperium; and mental and behavioural disorders.

Nomenclature of 541 overviews published between 2000 and 2018
Overall, 14% (73/541) of the overviews did not use any terms or synonyms to identify the study as an overview (Appendix B A word cloud was created to visualize the frequency of words and phrases used, with the sizes of words and phrases representing the frequency (Figure 4).

Multivariate regression
Multiple regression analyses were conducted to examine the relationship between citation density and To our knowledge, this is the rst study to examine the overview annual publication prevalence, predictors of citation rates and journal impact factors. This bibliometrics study provides a baseline of publication trends and impact of overviews, and the list of overviews identi ed can be used in future methods studies. The number of overviews has increased over the last 20 years, with 73% of the overviews published in the most recent 5-year period (2016-2020). In 2020, one overview was published per day. The majority of overviews had 4 to 6 authors on the team, and took on average 1.6 years to produce from search date to publication. Most overviews were published on the medical categories of health systems and services, musculoskeletal system, and mental disorders. The journals that publish the most overviews were the Cochrane Database of Systematic Reviews, PLOS ONE, and the Sao Paulo Medical Journal, but each with <10% of the published overview sample.

Prevalence of overviews
We found an 8-fold increase in the number of overviews between 2009 (n=25) and 2020 (n=332). An annual increasing prevalence of overviews has been found by other authors. One study found that the number of overviews increased from 1 in 2000 to 14 in 2010 [19] and another found a ve-fold increase in the publication of overviews from 2004 to 2011 [14]. The factors driving the increase in overviews are explained by the exponential increase in the number of systematic reviews being published, the publication of guidance on how to conduct an overview of reviews by Cochrane as of 2009, the subsequent publication of the rst Cochrane overview in the same year.

Higher citation rate of overviews
We found that the average citation rate for overviews was 90 (SD ±219.7) over 9 years, or 10 citations per overview per year. Several studies have found that systematic reviews and meta-analyses have higher citation counts than RCTs or observational studies [4,[20][21][22][23]. Mean citation counts for systematic reviews and meta-analyses ranged from 0.74 [4], 5.1 [22] to 6.6 per year [23]. Our study found a signi cantly higher mean citation count of 10 overviews per year than these studies on reviews. The mean impact factor of the journals the overviews publish is was 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing the two-year impact factors of the journals in which they are published.

Terminology
Currently identifying and searching for overviews is problematic because there is no agreed nomenclature, and no standardised de nition. One of the rst overviews was published in 1992 [24]; however, it only became recognized as such after 2009, when Cochrane published guidance for overviews, and its rst overview in the Cochrane library.
We found that only one third of our sample used the term was overview in the title, 12% used umbrella review or meta-review, and 14% used systematic review of reviews, or a close derivative. Of concern is that 7% used the term 'systematic review' to mean an overview of reviews. Inconsistent nomenclature may have affected the performance of our search lter used in this study. Overviews entitled 'systematic reviews' would be miscoded into MEDLINE's Publication Type (PT) for systematic reviews, leading to a compounding problem. Overviews should have their own Publication Type (PT) term in MEDLINE to help differentiate systematic reviews from overviews [15].
A consistent nomenclature for overviews would help in the identi cation of overviews. We recommend that authors of overviews use the term overview of systematic reviews in the title to help users search for and identify relevant reports.

Predictors of citation and impact factor
In our large sample, overviews with a high number of citations and high journal impact factors had group authorship, large sample sizes, were open access, and reported sources of funding. Several studies of citation counts and impact factors in the health eld identi ed predictors of citation rates similar to our own, namely number of authors, sample size, funding status, and open access [20,21,23,25,26].

Strengths and limitations
The main strength of this study is that it is the rst to look at predictors of citations and journal impact factor speci c to overviews published over a wide range of subject areas and journals, and it included a number of characteristics of both the article itself and the journal in which it appeared. Other strengths include comprehensiveness of years searched, the large number of extracted articles (n=541), and our strict eligibility criteria for overviews. We also search a combination of databases (Medline, Epistemonikos and Cochrane) with a high sensitivity (over 95%) in retrieving overviews in all medical elds [16].
A limitation to our study is that we may not have captured the actual number of overviews published as we only searched three databases and did not search the grey literature. Furthermore, we used a veri cation system for data extraction rather than independent extraction, which may not be as robust as what Cochrane standards are for Cochrane reviews. Less rigorously developed overviews, such as those only searching one database, were not included. Also, our selected overviews were limited to those pertaining to therapeutic, clinical and health intervention topics. In addition, the 541 analyzed do not represent the whole sample of 1218 overviews, this limiting generalizability of our ndings.
Our ability to predict the variance of annual citation rates was weak (13%); however, it is consistent with other similar studies. One study predicted 14% of the variance of annual citation rates using emergency medicine studies and 12 possible predictor variables [27], and another study predicted 20% using articles from three high impact journals [25].

Future research
The research provides a baseline of publication trends and impact of overviews for future research. Future research could also explore whether the quality of conduct of overviews predicted citation density.
An examination of the top 100 cited overviews compared to those least cited would inform authors about what factors contribute to citation rate. In addition, future investigations should explore social media impact metrics (e.g. altmetric score) and other article metrics, and composition of overview author networks. Finally, this compilation of 1218 overviews represents a bibliographic database of all overviews published in healthcare interventions or clinical treatments from 2000 to 2020 and can be used by methodologists interested in studying this review type.

Conclusions
This 20-year bibliometric study is the rst of its kind to identify and characterize overviews across 307 journals indexed in three databases. The 8-fold increase in the number of overviews from the publication of the Cochrane guidance for overviews in 2009 to 2020 illustrates the increasing popularity and demand for overviews. This demand is due potentially to the need to synthesise the results of multiple reviews on topics of broad scope, and of relevance to policy and practice.
Many academics, practitioners, and policy makers have failed to see the impact and utility of overviews, and their potential to inform clinical practice, policy and guideline development. Citation analysis and impact factor metrics can help these knowledge users to quantify, compare, and communicate the in uence of overviews of reviews. Our study found a mean citation count of 10 overviews per year, published in journals with a mean impact factor of 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing their two-year impact factors. We also found that highly cited overviews in journals with high impact factors are associated with group authorship, large sample sizes, open access publishing, and reported sources of funding. The study provides a baseline of annual publication trends and bibliometric impact of overviews for future research.

Highlights
Overviews synthesise the results of multiple systematic reviews.
Many academics, practitioners, and policy makers have failed to see the impact and utility of overviews, and their potential to inform clinical practice, policy and guideline development.
Citation analysis and impact factor metrics can quantify, compare, and communicate the in uence of overviews of reviews.
Our study reports the results of an empirical study investigating the prevalence, characteristics, number of citations of 'overviews of reviews' indexed in MEDLINE®, Epistemonikos, and the Cochrane Database of Systematic Reviews.
No bibliometric study has been published on overviews of reviews.
Our study ndings show the growing prevalence of overviews, and an 8-fold increase in the number of overviews published from 2009 to 2020, representing one published overview a day.
We found a high mean citation count of 10 overviews per year, published in journals with a mean impact factor of 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish.

Declarations
Ethics approval and consent to participate N/A

Transparency declaration
The lead author a rms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.