Bibliometric study of ‘overviews of systematic reviews’ of health interventions: Evaluation of prevalence, citation and journal impact factor

Overviews synthesising the results of multiple systematic reviews help inform evidence‐based clinical practice. In this first of two companion papers, we evaluate the bibliometrics of overviews, including their prevalence and factors affecting citation rates and journal impact factor (JIF). We searched MEDLINE, Epistemonikos and Cochrane Database of Systematic Reviews (CDSR). We included overviews that: (a) synthesised reviews, (b) conducted a systematic search, (c) had a methods section and (d) examined a healthcare intervention. Multivariable regression was conducted to determine the association between citation density, JIF and six predictor variables. We found 1218 overviews published from 2000 to 2020; the majority (73%) were published in the most recent 5‐year period. We extracted a selection of these overviews (n = 541; 44%) dated from 2000 to 2018. The 541 overviews were published in 307 journals; CDSR (8%), PLOS ONE (3%) and Sao Paulo Medical Journal (2%) were the most prevalent. The majority (70%) were published in journals with impact factors between 0.05 and 3.97. We found a mean citation count of 10 overviews per year, published in journals with a mean JIF of 4.4. In multivariable analysis, overviews with a high number of citations and JIFs had more authors, larger sample sizes, were open access and reported the funding source. An eightfold increase in the number of overviews was found between 2009 and 2020. We identified 332 overviews published in 2020, which is equivalent to one overview published per day. Overviews perform above average for the journals in which they publish.


Highlights
• Overviews synthesise the results of multiple systematic reviews. • Many academics, practitioners and policy makers have failed to see the impact and utility of overviews, and their potential to inform clinical practice, policy and guideline development. • Citation analysis and journal impact factor (JIF) metrics can quantify, compare and communicate the influence of overviews of reviews. • Our study reports the results of an empirical study investigating the prevalence, characteristics, number of citations of 'overviews of reviews' indexed in MEDLINE, Epistemonikos and the Cochrane Database of Systematic Reviews. • Our study findings show the growing prevalence of overviews, and an eightfold increase in the number of overviews published from 2009 to 2020, representing one published overview a day. • We found a high mean citation count of 10 overviews per year, published in journals with a mean JIF of 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish.

| BACKGROUND
Bibliometric analysis measures the impact of articles using metrics such as the journal impact factor (JIF) and number of citations. 1 A bibliometric analysis can serve to identify influential articles that have shaped medical practice, policy decisions and presented new research ideas. 2 Citation analysis provides an objective measurement of how often scientists use specific published work. Although citations have limitations, 3 they may offer vital information on how often scientific data are used and referenced, and provide one source of information to help prioritise research investment. 2 Several investigations have found that evidence syntheses, such as systematic reviews with meta-analysis, received a higher citation rate compared to other study designs [4][5][6][7][8] ; but typically represented only 4-6% of research output. 8,9 It is unclear whether 'overviews of systematic reviews' (henceforth called overviews) are cited equally as highly as systematic reviews, but it stands to reason that they might considering they are a type of evidence synthesis.
The number of systematic reviews has reached epidemic proportions 10 ; from none in 1987 11 ; 7 per day in 2003 5 ; 11 a day in 2010 12 ; and 22 per day in 2014. 13 Overviews have grown in number in tandem with the growth in systematic reviews, and they aim to appraise and synthesise results of systematic reviews. One investigation found a five-fold increase in the publication of overviews from 2004 to 2011. 14 The current number of overviews, and whether the trend in annual publication rate is increasing is unknown.
To advance the development of overviews, research is required to understand epidemiological aspects such as annual prevalence and factors that may influence the citation of overview publications. This paper is the first of two companion papers. In this first paper, we present our evaluation of the bibliometric characteristics of overviews, and evaluate factors affecting citation rates and JIF. In a second paper, we will present the assessment of whether two or more overviews overlapped in topic.

| Study design
The design is a bibliometrics study in the knowledge synthesis field. We followed systematic review methods guidance for searching, study selection and data extraction.

| Eligibility criteria
We included overviews published in any language and reports were translated by one of the authors (French, Spanish, German, Mandarin), where needed. We used Google Translate to translate articles in languages other than the ones listed, and when authors were not native speakers of French, Spanish, German or Mandarin. We restricted our search to reports published from January 1, 2000 to December 2020. Given that the Cochrane Handbook for Systematic Reviews of Interventions 15 chapter on overviews was first published in 2009, we did not expect to identify overviews published prior to 2000.
We reasoned that overviews may exist as stand-alone reports or also packaged as part of a clinical practice guideline and health technology assessment; accordingly, we developed eligibility criteria for both circumstances.
Inclusion criteria for overviews: • Aims to primarily focus on synthesising systematic reviews with or without meta-analyses (but the overview may also include primary studies). • Uses a systematic literature search found in the main body of the paper (i.e., search strategy includes text words and MeSH terms in at least two databases). • Has a methods section in the main body of the paper (not just in the abstract). • Examines the effects of health interventions or clinical treatments.
We excluded overviews that based their results solely on primary studies and methodological studies. Reports that were editorials, letters or comments were also excluded. Overviews of risk, exposure, prevention, measurement instruments, quality indicators, diagnostic, screening or prognostic research were excluded. We excluded protocols of overviews.
Inclusion criteria for clinical practice guidelines (CPGs) and health technology assessments (HTAs): • Uses an 'overview of review' approach to inform their recommendations (i.e., the CPG aims to primarily include, synthesise and present the results of the systematic reviews; but may also include primary studies).

| Search
Overviews were retrieved using a validated search filter for overviews 16 from MEDLINE (Ovid), Epistemonikos and the Cochrane Database of Systematic Reviews. In an empirical methods study of the retrieval sensitivity of six databases, the combination of MEDLINE and Epistemonikos retrieved 95.2% of all systematic reviews. 17 We are therefore confident that using both a validated search filter and this proven database combination will retrieve the majority of published overviews of healthcare interventions and clinical treatment in the literature. We searched from 2000 to December 2020 (Appendix A). The Epistemonikos search was limited to the category called 'Broad Syntheses', which includes overviews of systematic reviews, health technology assessments and clinical practice guidelines. Epistemonikos contains reports from electronic databases as well as the grey literature, including: PubMed, Embase, The Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Literatura Latinoamericana y del Caribe en Ciencias de la Salud, Database of Abstracts of Reviews of Effects, the Campbell Collaboration's online library, the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports, Cochrane Database of Systematic Reviews and the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) Evidence Library.

| Screening and study selection
The initial search results were imported into Excel 2010 for screening. A pilot screening of the first 19 papers was conducted in duplicate by all screeners to ensure high levels of agreement and common definitions of coding. We screened the titles and abstracts against the stated eligibility criteria first, then eligible full-text articles were reviewed for inclusion. Two independent reviewers screened reports at the title and abstract, and then again at the full text stage, then compared their results (G.S., Y.W.Z., C.L., L.L., P.H.Y., M.D., P.N.). Discrepancies were resolved by consensus, and arbitration by a third reviewer when necessary.
Categorised as 'other sources' in our flowchart (Figure 1), 237 overviews were published between 2013 and 2016 were previously identified and screened 16,18 (Appendix A), and added to our list of included studies.

| Data extraction
We extracted data on overviews published between 2000 and 2018 (n = 541) to enable a feasible extraction of data. Data extraction was piloted on 20 studies by all authors independently to identify any missing variables, come to agreement on coding definitions, and refine/reword the items. Discrepancies in the piloting phase were discussed and consensus reached by two authors. Full data extraction was performed independently by one investigator (T.N., A.S., G.S., S.T., S.A., U.E., E.K.R., C.L.), and checked by a second reviewer (T.N., A.S., G.S., S.T., S.A., U.E., E.K.R., C.L.).
We extracted the medical field of the overview; year of publication; country and continent (Europe, North America, Asia, Australia, South America, Africa) of corresponding author; number of authors; journal of publication; number of included studies; search date; inclusion of meta-analysis; and funding status. We also extracted the number of total patients in the included systematic reviews. If the total number was not reported, we added up total patients across all systematic reviews from a table of characteristics. We extracted the number of times the overview had been cited per year via the Google Scholar database, and the 2-year JIF for the top journals from Web of Science. The JIF, as calculated by Clarivate Analytics, is the ratio of citations to a journal in a given year to the citable items in the prior 2 years. We provide the Journal Citation Reports 2020 (JIF & Ranking of 2019) that we used in Appendix B. When we did not find the JIF in the Clarivate Analytics JOF list, we searched the journal's website for a 2020 JIF. We checked whether the journal was open access via the Directory of Open Access Journals (https://doaj.org/). We also extracted data on the terminology used to describe overviews in the title (overview of reviews, meta-review, review of reviews or any related synonyms).
The medical field categorisation used was the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD), a medical classification list by the World Health Organisation (WHO; https://www.who.int/standards/classifications/ classification-of-diseases). Time it took to conduct the overview was not reported in the overviews. Therefore, to estimate the time for each overview's search end date to date of publication (i.e., publication lag), we calculated the difference between the most recent search date reported and publication date using the month and day function in Excel 2010. We used the most recent search date in the overview if there were multiple search dates reported.
F I G U R E 1 PRISMA flowchart of study selection. CPG: clinical practice guideline; HTA, health technology assessment; MA, meta-analysis; SRs, systematic reviews 2.6 | Data analysis Descriptive analysis using frequencies and percentages were performed for categorical data and median and interquartile range (IQR) for continuous data. Citation density was used instead of citation count to avoid bias towards older articles. Citation density was defined as total number of citations divided by the number of years a study has been published. Citation density (the dependent variable) was positively skewed; therefore, the natural log transformation was obtained to approach a normal distribution.
Number of overviews published per year were presented in figures. The distribution of overviews by medical discipline and terms used to describe overviews was described. A free version of StatPlanet Plus (http://www. statsilk.com/software/statplanet) software was used to create the interactive world map to visualise the distribution of all overviews among different countries.
A multivariable regression was conducted to determine the association between six predictor variables (number of authors, open (free) versus closed (paywalled) journal access, number of overviews with meta-analyses, funding status, sample size, complementary and alternative therapies (CAM) topic) and two dependent variables (citation density and JIF). Statistical significance was set at p < 0.05. Statistical analyses were performed using Stata version 13 (StataCorp LP) using two commands manova and mvreg.

| Search results
From a total of 16,610 records retrieved from the MEDLINE, Cochrane Database of Systematic Reviews, and Epistemonikos databases, and 237 records retrieved from other sources (Figure 1), 2410 duplicates were removed. Of the remaining 14,437 records, 11,481 were excluded at the title/abstract stage, and 1738 were excluded at the full text stage. A total of 1218 overviews were included that met our eligibility criteria, and of these, 541 overviews were published between 2000 and 2018 (44%) were data extracted (data set can be accessed at https://osf.io/38bkg/). Many studies did not have a methods section, did not conduct a systematic search, and did not search for and include systematic reviews (Figure 1).  (Table 1).

| Prevalence of overviews
Several authors working on one overview was common; the average authorship per overview was 5.  We found an average citation rate of 90 (SD ±219.7) over 9 years, or 10 citations per overview per year. Four reports did not have a JIF and did not have citation counts and were therefore omitted from this analysis. When examining overviews published in the top 50 journals by JIFs (≥7.86), we found that 50 overviews (9%; 50/537) received 18% of citations (range 4-1250). Nine overviews remained un-cited. A total of 108 out of 537 overviews (20%) were cited 100 times or more. An article having 100 or more citations is regarded as a 'classic'. 19 The topics of the six overviews with citations counts of >1000 included two on diseases of the musculoskeletal system and connective tissue, two on factors influencing health status and contact with health services, and one each on pregnancy, childbirth and the puerperium; and mental and behavioural disorders.

| Nomenclature of 541 overviews published between 2000 and 2018
Overall, 14% (73/541) of the overviews did not use any terms or synonyms to identify the study as an overview (Appendix C). Of the 541 overviews, the majority used the term overview (196/541 [36%]) and 'systematic review of reviews' (74/541 [14%]) or a close derivative in the title (Appendix C). Synonyms and other terms were used less frequently; 42 overviews (8%) used the term umbrella review, and 19 (4%) used the term meta-review (3%) in the title. The term 'systematic review' was used in 33/541 (7%) to mean an overview.
A word cloud was created to visualise the frequency of words and phrases used, with the sizes of words and phrases representing the frequency (Figure 4).   The multiple regression model with JIF, and four predictors, produced R 2 = 0.119, F(5, 514) = 17.18, p < 0.001 and citation density and four predictors produced R 2 = 0.126, F(5, 514) = 18.31, p < 0.001. Overviews on topics related to CAM and overviews using a meta-analysis were not associated with citation density and JIF, and were therefore not included in the model.

| Summary of results
To our knowledge, this is the first study to examine the annual publication prevalence, predictors of citation rates and JIFs of overviews. This bibliometrics study provides a baseline of publication trends and impact of overviews, and the list of overviews identified can be used in future methods studies. The number of overviews has increased over the last 20 years, with 73% of the overviews published in the most recent 5-year period (2016-2020). In 2020, one overview was published per day. The majority of overviews had 4-6 authors on the team, and took on average 1.6 years to produce from search date to publication. Most overviews were published on the medical categories of health systems and services, musculoskeletal system and mental disorders. The journals that publish the most overviews were the Cochrane Database of Systematic Reviews, PLOS ONE and the Sao Paulo Medical Journal, but each with <10% of the published overview sample.

| Prevalence of overviews
We found an eightfold increase in the number of overviews between 2009 (n = 25) and 2020 (n = 332). An annual increasing prevalence of overviews has been found by other authors. One study found that the number of overviews increased from 1 in 2000 to 14 in 2010 20 and another found a five-fold increase in the publication of overviews from 2004 to 2011. 14 The factors driving the increase in overviews are explained by the exponential increase in the number of systematic reviews being published, the publication of guidance on how to conduct an overview of reviews by Cochrane as of 2009, and the subsequent publication of the first Cochrane overview in the same year.

| Higher citation rate of overviews
We found that the average citation rate for overviews was 90 (SD ±219.7) over 9 years, or 10 citations per overview per year. Several studies have found that systematic reviews and meta-analyses have higher citation counts than RCTs or observational studies. 4,[21][22][23][24] Mean citation counts for systematic reviews and meta-analyses ranged from 0.74, 4 5.1 23 to 6.6 per year. 24 Our study found a significantly higher mean citation count of 10 overviews per year than these studies on reviews. The mean JIF of the journals the overviews publish is was 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing the 2-year JIFs of the journals in which they are published.

| Terminology
Currently identifying and searching for overviews is problematic because there is no agreed nomenclature, and no standardised definition. One of the first overviews was published in 1992 25 ; however, it only became recognised as such after 2009, when Cochrane published guidance for overviews and its first overview in the Cochrane library.
We found that only one third of our sample used the term overview in the title, 12% used umbrella review or meta-review and 14% used systematic review of reviews, or a close derivative. Of concern is that 7% used the term 'systematic review' to mean an overview of reviews. Inconsistent nomenclature may have affected the performance of our search filter used in this study. Overviews entitled 'systematic reviews' would be miscoded into MEDLINE's Publication Type (PT) for systematic reviews, leading to a compounding problem. Overviews should have their own Publication Type (PT) term in MEDLINE to help differentiate systematic reviews from overviews. 16 A consistent nomenclature for overviews would help in the identification of overviews. We recommend that authors of overviews use the term overview of systematic reviews in the title to help users search for and identify relevant reports.

| Strengths and limitations
The main strength of this study is that it is the first to look at predictors of citations and JIF specific to overviews published over a wide range of subject areas and journals, and it included a number of characteristics of both the article itself and the journal in which it appeared. Other strengths include comprehensiveness of years searched, the large number of extracted articles (n = 541), and our strict eligibility criteria for overviews. We also search a combination of databases (MEDLINE, Epistemonikos and Cochrane) with a high sensitivity (over 95%) in retrieving overviews in all medical fields. 17 A limitation to our study is that we may not have captured the actual number of overviews published as we only searched three databases and did not search the grey literature. Furthermore, we used a verification system for data extraction rather than independent extraction, which may not be as robust as what Cochrane standards are for Cochrane reviews. Less rigorously developed overviews, such as those only searching one database, were not included. Also, our selected overviews were limited to those pertaining to therapeutic, clinical and health intervention topics. In addition, the 541 analysed do not represent the whole sample of 1218 overviews, this limiting generalizability of our findings.
Our ability to predict the variance of annual citation rates was weak (13%); however, it is consistent with other similar studies. One study predicted 14% of the variance of annual citation rates using emergency medicine studies and 12 possible predictor variables, 28 and another study predicted 20% using articles from three high impact journals. 26

| Future research
The research provides a baseline of publication trends and impact of overviews for future research. Future research could also explore whether the quality of conduct of overviews predicted citation density. An examination of the top 100 cited overviews compared to those least cited would inform authors about what factors contribute to citation rate. In addition, future investigations should explore social media impact metrics (e.g., Altmetric score) and other article metrics, and composition of overview author networks. Finally, this compilation of 1218 overviews represents a bibliographic database of all overviews published in

| CONCLUSIONS
This 20-year bibliometric study is the first of its kind to identify and characterise overviews across 307 journals indexed in three databases. The eightfold increase in the number of overviews from the publication of the Cochrane guidance for overviews in 2009-2020 illustrates the increasing popularity and demand for overviews. This demand is due potentially to the need to synthesise the results of multiple reviews on topics of broad scope, and of relevance to policy and practice.
Many academics, practitioners and policy makers have failed to see the impact and utility of overviews, and their potential to inform clinical practice, policy and guideline development. Citation analysis and JIF metrics can help these knowledge users to quantify, compare and communicate the influence of overviews of reviews. Our study found a mean citation count of 10 overviews per year, published in journals with a mean JIF of 4.4. These data indicate that, overall, overviews perform above average for the journals in which they publish, and therefore may contribute to increasing their 2-year JIFs. We also found that highly cited overviews in journals with high JIFs are associated with group authorship, large sample sizes, open access publishing and reported sources of funding. The study provides a baseline of annual publication trends and bibliometric impact of overviews for future research.