Bibliometric study of ‘overviews of systematic reviews’: evaluation of prevalence, citation and impact factor

Background: Overviews synthesizing the results of multiple systematic reviews help inform evidence-based clinical practice. In this �rst of two companion papers, we evaluate the bibliometrics of ‘overviews of systematic reviews’, including their prevalence, number of citations, and factors affecting citation rates and journal impact factor. Methods: We searched MEDLINE, Epistemonikos and the Cochrane library databases. We applied eligibility criteria to identify overviews that: (a) aimed to focus on synthesizing reviews, (b) conducted a systematic search, (c) had a full methods section, and (d) examined a health intervention or clinical treatment effect. A multivariate regression was conducted to determine the association between citation density and impact factor and 6 predictor variables of interest. Results: We found 1218 overviews published from 2000 to 2020; the majority (73%) of which were published in the most recent 5-year period (2016-2020). We extracted a selection of these overviews (n=541; 44%) dated from 2000 to 2018. The 541 overviews were published in 307 journals; the Cochrane Database of Systematic Reviews (8%), PLOS ONE (3%) and the Sao Paulo Medical Journal (2%) being the most prevalent. The majority of overviews (70%) were published in journals with impact factors between 0.05 and 3.97. The average citation rate was 90 (SD ±219.7) over 9 years, or 10 citations per overview per year. In multivariate analysis, overviews with a high number of citations and high journal impact factors tended to have more authors, larger sample sizes, be open access and report funding source. Conclusions: We found an 8 fold increase in the number of overviews from 2009 to 2020; and a representation of one published a day in 2020. Factors driving the increase in overviews include the exponential increase in the number of systematic reviews, the publication of Cochrane guidance on overview of reviews in 2009 and the subsequent publication of the �rst Cochrane overview in the same year. Our study found a signi�cantly higher mean citation count of 10 overviews per year, published in journals with a mean impact factor of 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish. We


Background
Bibliometric analysis measures the impact of articles using metrics such as the journal impact factor and number of citations [1].A bibliometric analysis can serve to identify in uential articles that have shaped medical practice, policy decisions, and presented new research ideas [2].The journal impact factor measures the average number of citations of an article in the journal over the previous year.Citation analysis provides an objective measurement of how often scientists use speci c published work.Although citations have limitations [3], they may offer vital information on how often scienti c data are used and referenced, and provide one source of information to help prioritise research investment [2].
Several investigations have found that evidence syntheses, such as systematic reviews with metaanalysis, received a higher citation rate compared to other study designs [4][5][6][7][8]; but typically represented only 4-6% of research output [8,9].It is unclear whether 'overviews of systematic reviews' (henceforth called overviews) are cited equally as highly as systematic reviews, but it stands to reason that they might considering they are a type of evidence synthesis.
The number of systematic reviews has reached epidemic proportions [10]; from none in 1987 [11]; 7 per day in 2003 [5]; 11 a day in 2010 [12]; and 22 per day in 2014 [13].Overviews have grown in number in tandem with the growth in systematic reviews, and they aim to appraise and synthesise results of systematic reviews.One investigation found a ve-fold increase in the publication of overviews from 2004 to 2011 [14].The current number of overviews, and whether the trend in annual publication rate is increasing is unknown.
To advance the development of overviews, research is required to understand epidemiological aspects such as annual prevalence and factors that may in uence the citation of overview publications.This paper is the rst of two companion papers.In this rst paper we present our evaluation of the bibliometric characteristics of overviews, and evaluate factors affecting citation rates and journal impact factor.In a second paper, we will present the assessment of whether two or more overviews overlapped in topic.

Study design
The design is a bibliometrics study in the knowledge synthesis eld.We followed systematic review methods guidance for searching, study selection, and data extraction.

Eligibility criteria
We included overviews published in any language and reports were translated by one of the authors (French, Spanish, German, Mandarin), where needed.We restricted our search to reports published from January 1, 2000 to December 31, 2018.Given that the Cochrane Handbook chapter on overviews was rst published in 2009, we did not expect to identify overviews published prior to 2000.
We reasoned that overviews may exist as stand-alone reports or also packaged as part of a clinical practice guideline and health technology assessment; accordingly, we developed eligibility criteria for both circumstances.
Inclusion criteria for overviews: Aims to primarily focus on synthesizing systematic reviews with or without meta-analyses (but the overview may also include primary studies).
Uses a systematic literature search found in the main body of the paper (i.e.search strategy includes text words and MeSH terms in at least two databases).
Has a methods section in the main body of the paper (not just in the abstract).
Examines the effects of health interventions or clinical treatments.
We excluded overviews that based their results solely on primary studies and methodological studies.Reports that were editorials, letters, or comments were also excluded.Overviews of risk, exposure, prevention, measurement instruments, quality indicators, diagnostic, screening or prognostic research were excluded.We excluded protocols of overviews.
Inclusion criteria for clinical practice guidelines (CPGs) and health technology assessments (HTAs): Uses an 'overview of review' approach to inform their recommendations (i.e. the CPG aims to primarily include, synthesise and present the results of the systematic reviews; but may also include primary studies).

Search
Overviews were retrieved using a validated search lter for overviews [15] from MEDLINE (Ovid), Epistemonikos and the Cochrane Database of Systematic Reviews.In an empirical methods study of the retrieval sensitivity of 6 databases, the combination of Medline and Epistemonikos retrieved 95.2% of all systematic reviews [16].We are therefore con dent that using both a validated search lter and this proven database combination will retrieve the majority of published overviews of healthcare interventions and clinical treatment in the literature.
We searched from 2000 to December 2020 (Appendix A).

Screening and study selection
The initial search results were imported into Excel 2010 for screening.A pilot screening of the rst 19 papers was conducted in duplicate by all screeners to ensure high levels of agreement and common de nitions of coding.We screened the titles and abstracts against the stated eligibility criteria rst, then eligible full-text articles were reviewed for inclusion.Two independent reviewers screened reports at the title and abstract, and then again at the full text stage, then compared their results.Discrepancies were resolved by consensus, and arbitration by a third reviewer when necessary.
Categorized as 'other sources' in our owchart (Figure 1), 237 overviews published between 2013 and 2016 were previously identi ed and screened [15,17] using identical methods (Appendix A), and added to our list of included studies.

Data extraction
We extracted data on overviews published between 2000 and 2018 (n=541) to enable a feasible extraction of data.Data extraction was piloted on 20 studies by all authors independently to identify any missing variables, come to agreement on coding de nitions, and re ne/reword the items.Discrepancies in the piloting phase were discussed and consensus reached by two authors.Full data extraction was performed independently by one investigator, and checked by a second reviewer.
We extracted the medical eld of the overview; year of publication; country and continent (Europe, North America, Asia, Australia, South America, Africa) of corresponding author; number of authors; journal of publication; number of included studies; search date; inclusion of meta-analysis; and funding status.We also extracted the number of total patients in the included systematic reviews.If the total number was not reported, we added up total patients across all systematic reviews from a table of characteristics.We extracted the number of times the overview had been cited per year via the Google Scholar database, and the 2-year impact factor for the top journals from Web of Science.Impact factor, as calculated by Clarivate Analytics, is a measure of the average number of times articles from a two-year time frame have been cited in a given year.We checked whether the journal was open access via the Directory of Open Access Journals (https://doaj.org/).We also extracted data on the terminology used to describe overviews in the title (overview of reviews, meta-review, review of reviews, or any related synonyms).
The medical eld categorisation used was the 10th revision of the International Statistical Classi cation of Diseases and Related Health Problems (ICD), a medical classi cation list by the World Health Organization (WHO) (https://www.who.int/standards/classications/classi cation-of-diseases).Time it took to conduct the overview was not reported in the overviews.Therefore, to estimate the time for each overview's search end date to date of publication (i.e.publication lag), we calculated the difference between the initial literature search date and publication date using the month and day function in Excel 2010.

Data analysis
Descriptive analysis using frequencies and percentages were performed for categorical data and median and interquartile range (IQR) for continuous data.Citation density was used instead of citation count to avoid bias towards older articles.Citation density was de ned as total number of citations divided by the number of years a study has been published.Citation density (the dependent variable) was positively skewed; therefore, the natural log transformation was obtained to approach a normal distribution.
Mean publication frequencies by year were calculated and presented in gures.The distribution of overviews by medical discipline and terms used to describe overviews was described.A free version of StatPlanet Plus (http://www.statsilk.com/software/statplanet)software was used to create the interactive world map to visualize the distribution of all overviews among different countries.
A multivariate regression was conducted to determine the association between predictor variables (number of authors, open versus closed journal access, number of overviews with meta-analyses, funding status, sample size, CAM topic) and two dependent variables (citation density and impact factor).Statistical signi cance was set at p < .05.Statistical analyses were performed using Stata version 13 (StataCorp LP) using two commands manova and mvreg.

Search results
From a total of 16,610 records retrieved from the MEDLINE, Cochrane Database of Systematic Reviews, and Epistemonikos databases, and 237 records retrieved from other sources (Figure 1), 2410 duplicates were removed.Of the remaining 14,437 records, 11,481 were excluded at the title/abstract stage, and 1738 were excluded at the full text stage.A total of 1218 overviews were included that met our eligibility criteria, and of these, 541 overviews published between 2000 and 2018 (44%) were data extracted (data set can be accessed at https://osf.io/38bkg/).Many studies did not have a methods section, did not conduct a systematic search, and did not search for and include systematic reviews (Figure 1).

Prevalence of overviews
We identi ed 1218 overviews, CPGs or HTAs published between 2000 and 2020.Between the years of 2000 and 2018, there has been an increase in the publication of overviews (Figure 2).The majority of overviews (415/610 [68%]) were published between a 5 year period, from 2013-2018.Approximately three-quarters of the overviews (537/610 [88%]) were reported during the most recent decade (2010-2020).An 8-fold increase in the number of overviews was found between 2009 (n=25) and 2020 (n=332).We identi ed 332 overviews published in the three databases in 2020, which is approximately equivalent to 1 overview published per day.

Characteristics of included overviews published between 2000 and 2018 (n=541)
Overviews published between 2000 and 2018 were extracted and analysed (n=541).The most frequent subject of the overviews (92/541 [17%]) was ICD-10 classi cation "factors in uencing health status and contact with health services."Another 62/541 (11.5%) focused on diseases of the musculoskeletal system and connective tissue, 56/541 (10.4%) were about mental and behavioural disorders, 42/541 (7.8%) were on diseases of the circulatory system, and 34/541 (6.3%) were focused on neoplasms.A little under half of the overviews focused on 15 other conditions in the ICD-10 medical classi cation (Table 1).We found an average citation rate of 90 (SD ±219.7) over 9 years, or 10 citations per overview per year.
When examining overviews published in the top 50 journals by impact factors (≥ 7.86), we found that 50 overviews (9%; 50/537) received 18% of citations (range 4-1250).Nine overviews remained un-cited.A total of 108 out of 537 overviews (20%) were cited 100 times or more.An article having 100 or more citations is regarded as a 'classic' [18].The topics of the six overviews with citations counts of >1000 included two on diseases of the musculoskeletal system and connective tissue, two on factors in uencing health status and contact with health services, and one each on pregnancy, childbirth and the puerperium; and mental and behavioural disorders.

3.6
Nomenclature of 541 overviews published between 2000 and 2018 Overall, 14% (73/541) of the overviews did not use any terms or synonyms to identify the study as an overview (Appendix B).Of the 541 overviews, the majority used the term overview (196/541 [36%]) and 'systematic review of reviews' (74/541 [14%]) or a close derivative in the title (Appendix B).Synonyms and other terms were used less frequently; 42 overviews (8%) used the term umbrella review, and 19 (4%) used the term meta-review (3%) in the title.The term 'systematic review' was used in 33/541 (7%) to mean an overview.
A word cloud was created to visualize the frequency of words and phrases used, with the sizes of words and phrases representing the frequency (Figure 4).

Multivariate regression
Multiple regression analyses were conducted to examine the relationship between citation density and impact factor with various potential predictors.Controlling for the other variables in the model, citation density and impact factor is positively and signi cantly correlated with the number of authors, sample size, funding reported, open access article (Appendix C).This indicates that overviews with a high number of citations and high journal impact factors tend to have higher number of authors, larger sample sizes, be open access and report sources of funding.The multiple regression model with impact factor, and four predictors produced R² = .119,F(5, 514) = 17.18, p < .001,and citation density and four predictors produced R² = .126,F(5, 514) = 18.31, p < .001.Overviews on topics related to complementary and alternative therapies (CAM) and overviews using a meta-analysis were not associated with citation density and impact factor so were not included in the model.

Summary of results
To our knowledge, this is the rst study to examine the overview annual publication prevalence, predictors of citation rates and journal impact factors.This bibliometrics study provides a baseline of publication trends and impact of overviews, and the list of overviews identi ed can be used in future methods studies.The number of overviews has increased over the last 20 years, with 73% of the overviews published in the most recent 5-year period (2016-2020).In 2020, one overview was published per day.The majority of overviews had 4 to 6 authors on the team, and took on average 1.6 years to produce from search date to publication.Most overviews were published on the medical categories of health systems and services, musculoskeletal system, and mental disorders.The journals that publish the most overviews were the Cochrane Database of Systematic Reviews, PLOS ONE, and the Sao Paulo Medical Journal, but each with <10% of the published overview sample.

Prevalence of overviews
We found an 8-fold increase in the number of overviews between 2009 (n=25) and 2020 (n=332).An annual increasing prevalence of overviews has been found by other authors.One study found that the number of overviews increased from 1 in 2000 to 14 in 2010 [19] and another found a ve-fold increase in the publication of overviews from 2004 to 2011 [14].The factors driving the increase in overviews are explained by the exponential increase in the number of systematic reviews being published, the publication of guidance on how to conduct an overview of reviews by Cochrane as of 2009, the subsequent publication of the rst Cochrane overview in the same year.

citation rate of overviews
found that the average citation rate for overviews was 90 (SD ±219.7) over 9 years, or 10 citations per overview per year.Several studies have found that systematic reviews and meta-analyses have higher citation counts than RCTs or observational studies [4,[20][21][22][23].Mean citation counts for systematic reviews and meta-analyses ranged from 0.74 [4], 5.1 [22] to 6.6 per year [23].Our study found a signi cantly higher mean citation count of 10 overviews per year than these studies on reviews.The mean impact factor of the journals the overviews publish is was 4.4.These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing the two-year impact factors of the journals in which they are published.

Terminology
Currently identifying overviews is problematic because there is no agreed nomenclature, and no standardised de nition.We found that only one third of our sample used the term was overview in the title, 12% used umbrella review or meta-review, and 14% used systematic review of reviews, or a close derivative.A consistent nomenclature for overviews would help in the identi cation of overviews.We recommend that authors of overviews use the term overview of systematic reviews in the title to help users identify relevant reports.
One of the rst overviews was published in 1992 [24]; however, it only became recognized as such after 2009, when Cochrane published guidance for overviews, and its rst overview in the Cochrane library.Of concern is that 7% used the term 'systematic review' to mean an overview of reviews.These misclassi ed systematic reviews would in turn be miscoded into MEDLINE's Publication Type (PT) for systematic reviews, leading to a compounding problem.Overviews should have their own Publication Type (PT) term in MEDLINE to help differentiate systematic reviews from overviews [15].

Predictors of citation and impact factor
In our large sample, overviews with a high number of citations and high journal impact factors had group authorship, large sample sizes, were open access, and reported sources of funding.Several studies of citation counts and impact factors in the health eld identi ed predictors of citation rates similar to our own, namely number of authors, sample size, funding status, and open access [20,21,23,25,26].

and limitations
The main strength of this study is that it is the rst to look at predictors of citations and journal impact factor speci c to overviews published over a wide range of subject areas and journals, and it included a number of characteristics of both the article itself and the journal in which it appeared.Other strengths include comprehensiveness of years searched, the large number of extracted articles (n=541), and our strict eligibility criteria for overviews.We also search a combination of databases (Medline, Epistemonikos and Cochrane) with a high sensitivity (over 95%) in retrieving overviews in all medical elds [16].
A limitation to our study is that we may not have captured the actual number of overviews published as we only searched three databases and did not search the grey literature.Furthermore, we used a veri cation system for data extraction rather than independent extraction, which may not be as robust as what Cochrane standards are for Cochrane reviews.Less rigorously developed overviews, such as those only searching one database, were not included.Also, our selected overviews were limited to those pertaining to therapeutic, clinical and health intervention topics.
Our ability to predict the variance of annual citation rates was weak (13%); however, it is consistent with other similar studies.One study predicted 14% of the variance of annual citation rates using emergency medicine studies and 12 possible predictor variables [27], and another study predicted 20% using articles from three high impact journals [25].

Future research
The research provides a baseline of publication trends and impact of overviews for future research.4.9 This 20-year bibliometric study is the rst of its kind to identify and characterize overviews across 307 journals indexed in three databases.The 8 fold increase in the number of overviews from the publication of the Cochrane guidance for overviews in 2009 and 2020 illustrates the increasing popularity and demand for overviews.This demand is due potentially to the need to synthesise the results of multiple reviews on topics of broad scope, and of relevance to policy and practice.Our study found a mean citation count of 10 overviews per year, published in journals with a mean impact factor of 4.4.These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing their two-year impact factors.We also found that highly cited overviews in journals with high impact factors are associated with group authorship, large sample sizes, open access publishing, and reported sources of funding.The study provides a baseline of annual publication trends and bibliometric impact of overviews for future research.The designations employed and the presentation of the material on this map do not imply the expression of any opinion whatsoever on the part of Research Square concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries.This map has been provided by the authors.

Future research could also
explore whether the quality of conduct of overviews predicted citation density.An examination of the top 100 cited overviews compared to those least cited would inform authors about what factors contribute to citation rate.In addition, future investigations should explore social media impact metrics (e.g.altmetric score) and other article metrics, and composition of overview author networks.Finally, this compilation of 1218 overviews represents a bibliographic database of all overview published in healthcare interventions or clinical treatments from 2000 to 2020 and can be used by methodologists interested in studying this review type.

Table 2 :
Journals, impact factor, and citation density of 541 overviews published between 2000 and 2018