Characteristics of retracted systematic reviews and meta-analyses in the biomedical literature

Introduction

Systematic reviews and meta-analyses—collectively referred to as “reviews” in this article for simplicity—provide important support for evidence-based medicine because they summarize data quantitatively to produce composite findings that can be used to guide future research and the clinical implementation of research findings (Evans 2003). The number of reviews being published has shown a consistently increasing trend over the past decade or so, and it had been argued that reviews are becoming increasingly redundant (Ioannidis 2016), with more than one meta-analysis of randomized trials for most topics and quite large numbers of meta-analyses in some fields. For example, 185 meta-analyses about treating clinical depression with antidepressants were published between 2007 and 2014 (Ioannidis 2016). Redundancies among such articles could give false confidence in the strength of their conclusions. The output of meta-analyses produced in China in particular has increased dramatically in recently years, especially for studies about genetic associations (Ioannidis et al. 2013).

Accuracy within reviews, which depends on the quality of the data included, is critical for their interpretation. For example, a research group led by Boldt in Germany had a large number of retracted papers due to fraud; however, those retracted works may have impacted critical care medicine practices (Bellomo 2011). Specifically, in Boldt’s retracted studies, hydroxyethyl starch had been reported to benefit patients significantly (Boldt et al. 2007 and Boldt et al. 2010). Following exclusion of those retracted studies from a meta-analysis examining the use of hydroxyethyl starch in critically ill patients, results opposite to the findings of Boldt’s studies were obtained in the meta-analysis (Zarychanski et al. 2013).

Generally, retraction occurs when an article’s integrity has been compromised and the validity and/or reliability of a study have come into question (Wager et al. 2009). The number of retractions in the biomedical literature has increased dramatically in recent years (Wager and Williams 2010; Steen 2013). Major causes for retraction include fraud/suspected fraud, error, duplicate publication, and plagiarism (Fang 2012). A recent surge in retractions of papers, including many meta-analyses, from China due to a faked review process attracted substantial attention (Fang 2012; Chen 2018; Wang 2018). Cochrane reviews, which is a highly regarded database for biomedical and healthcare related studies and treated as a key resource in evidence-based medicine, has implemented its own indexing withdrawal policy to deal with these challenges (Policy on withdrawing published Cochrane Reviews (including protocols)).

The characteristics of retracted reviews, such as country of origin, reason for retraction, and differing treatment among different indexing resources, have yet to be defined. In particular, it would be of interest to determine whether retraction trends differ between Cochrane indexed and non-Cochrane indexed articles. The aim of the present work was to obtain a systematic outline of the characteristics of retracted systematic reviews and meta-analyses.

Methods

Collation of published reviews

On April 1, 2018, we searched PubMed for article type and publication date. To find appropriate articles, we used the following search terms: meta-analysis, systematic review, retraction of publication, and retracted publication. To detect trends over time, we collected articles at 5-year intervals from 1990 to 2015 (1990, 1995, 2000, 2005, 2010, and 2015) as well as from 2017. We also searched for reviews in The Cochrane Database of Systematic Reviews with the aforementioned publication dates and the requirement of being in a journal that is indexed in PubMed.

Identification of retracted articles

On February 11, 2017, we conducted a PubMed literature search to identify all PubMed-indexed systematic reviews and meta-analyses without a limitation on the publication date. We used the publication-type search terms “retracted publication” or “retraction of publication”. We also conducted a title word search with the search terms “retracted”, “retraction”, “withdrawal”, and “withdrawn”. We applied a filter to limit the results to systematic reviews and meta-analyses.

On April 11, 2017, we conducted a Cochrane Library search for articles with the terms “retracted”, “retraction”, “withdrawal”, or “withdrawn” in the title, abstract or keywords. We refined the results to articles classified as reviews.

Determined characteristics of retracted meta-analysis

On September 01, 2019, we searched the Medline database on the Web of Science for titles containing the phrase ‘‘meta-analysis’’ to determine research areas of published (or retracted) articles. For determining types of these meta-analyses according to methodology (genetics-related, treatment-related, diagnostic, and etc.), we also searched the Medline database on the PubMed using the strategy as previous study (Ioannidis et al. 2013) reported and separated the publications (retractions) in to five groups as shown in table4.

Data processing

We examined the titles and abstracts of all identified papers manually, and excluded those that were redundant, had not been retracted, or were not systematic reviews or meta-analyses. We extracted and recording the following information for each of the remaining articles: journal name, journal impact factor according to the 2015 Web of Science, first author’s country of origin, publication year, and retraction year.

Because the Cochrane Library has its own withdrawal policy, we categorized the articles as Cochrane (published in The Cochrane Database of Systematic Review) and non-Cochrane (published in other journals) reviews for further analysis. Causes of retraction were defined according to each retraction notice. For non-Cochrane reviews, the retraction reason classes were: error, duplicate publication, plagiarism, faked review process, authorship dispute, unclear, and others. For Cochrane reviews, the retraction reason classes were: update issues (out of date, replaced, superseded, updated, update pending, unable to update, split, or merged), error, sponsorship policy issue, and others.

The data were recorded and analyzed in Excel® (Microsoft Corporation, Redmond, WA). The data are presented as means ± standard deviations (SDs), numbers, and percentages.

Results

General trends in the publication of reviews

From 1990 to 2017, the numbers of systematic reviews and meta-analyses published each year and indexed in PubMed increased by some 50-fold and some 30-fold, respectively (Table 1). Note that these increases occurred in the context of only a 3.1-fold increase in the overall number of all PubMed-indexed items during this period (Table 1). Meanwhile, the Cochrane library, which is regarded as an important systematic review source, had relatively small numbers of review-type articles and did not exhibit the increasing trends seen in other resources (Table 1). With respect to percent of systematic reviews, The Cochrane Database of Systematic Reviews indexed 16.9% of the quantity of systematic reviews in Pubmed in 2000, but only 1.9% in 2017 (Table 1). We documented a dramatic increase in retractions in recent years, with less than 100 retractions per year occurring before 2005, 3-fold that number in 2010, and nearly 7-fold that number in 2015 (Table 1).

Of these meta-analyses, there were more papers about research areas of biochemistry molecular biology, oncology, genetics heredity and etc. from China. And, more meta-analyses from USA were about pharmacology pharmacy, cardiovascular system cardiology, behavioral sciences and etc. (Table 2). As shown in Table 3, there were also more gene-related and less treatment-related papers from China compared with from other countries.

Retracted non-Cochrane reviews

A total of 171 retracted non-Cochrane reviews were identified in the study years (Fig. 1), of which 120 were meta-analyses. There were 176 causes listed for these retractions, which were, in order from most to least common: faked review process (n = 70), error (n = 36), plagiarism (n = 20), other (n = 17), and duplicate (n = 12). The earliest retractions occurred in 1984, and the number of retracted publications increased dramatically after 2010. The first retraction for which faked peer review process was the cited reason was in 2014. The mean (SD) time to retraction for these 171 retracted reviews was 32.4 (36.1) months.

Of these 171 articles, 155 (90.6%) were published in Science Citation Index listed journals, including 29 that were published in journals with impact factors > 5.0. The retracted articles were published in a cumulative total of 93 journals; 5 of these journals had ≥ 5 retractions (Table 1). With respect to meta-analyses, most (n = 68) were retracted for faked review process. Manipulation of the peer review process was associated with large numbers of retractions in a small number of journals, such as Tumor Biology (n = 30), Molecular Biology Reports (n = 14), European Journal of Medical Research (n = 6), and Molecular Neurobiology (n = 4). Meanwhile, the journals with the most retractions due to error (n = 5) and “other” reasons (n = 13) were Obstetrics and Gynecology and Contraception, respectively.

Of the 171 retracted non-Cochrane reviews identified, the first author was from China in 99 cases, from the USA in 29 cases, from the UK in 7 cases, and from India in 6 cases. The remaining 30 retracted articles’ first authors were from 22 other countries. The distribution of retraction causes varied according to the first author’s country of origin (Fig. 2). Notably, articles retracted for a faked review process were concentrated among retracted articles with Chinese first authors. Conversely, of a total of nine meta-analysis retractions with first authors from the USA, five were retracted for the reason of error. Most of the retracted articles from Chinese first authors (n = 96) were meta-analyses.

There were more gene-related meta-analyses retracted compared to its total publication numbers (Table 3). Most of these retracted papers were from areas of oncology, biochemistry molecular biology, or genetics heredity (Table 5).

Retracted Cochrane reviews

The impact factor of the Cochrane database of systematic reviews in 2015 was 6.10. There were 439 retracted reviews from the Cochrane library in the study years (Fig. 2). The reasons for retraction, from most to least common, were: updating issues (n = 382), errors (n = 26), other (n = 18), and sponsorship policy issues (n = 13). The major first author countries of origin, from most to least common, were the UK (n = 184), Canada (n = 50), Australia (n = 41), the USA (n = 26), the Netherlands (n = 20), China (n = 17), and Brazil (n = 13). Most of retracted systematic reviews were classified as “withdrawn”.

Discussion

The present study showed that among retracted non-Cochrane reviews, the most prevalent first-author country of origin was China and the most common reason for retraction was a faked peer review process. These retracted publications were generally published in journals with low impact factors and retractions were concentrated, in majority, to a relatively limited number of journals. There were more retracted meta-analyses of gene-related types and of research areas about oncology, biochemistry molecular biology, genetics heredity and etc. Meanwhile, for the 439 retracted Cochrane reviews examined, the most prevalent first-author country of origin was the UK and the most prevalent reason for retraction was updating-related issues, which are unrelated to error or scientific misconduct generally.

Papers retracted for updating-related issues are considered to be “retired” due to providing guidelines or practical recommendations that are outdated and not amenable to a simple update (Alison 2018). Notably, we found that most retractions from The Cochrane Database of Systematic Reviews for updating-related issues were not tagged as “Retracted Publication” or “Retraction of Publication” type articles in Medline. The high representation of British first authors among retracted Cochrane reviews may be a simple reflection of the output of systematic reviews by country. Cochrane Reviews appear to be relatively underused in the USA especially, as well as in Canada and Australia, among other countries (Grimes et al. 2008).

According to first-author country of origin, China was the source of the largest number of non-Cochrane systematic reviews identified in this study. There has been a surge in the number of reviews from Chinese authors in particular in recent years (Ioannidis 2013). The annual number of meta-analyses from China increased 40-fold between 2003 and 2011 versus 2.4-fold for the USA. China has become the most prolific producer of English-language, PubMed indexed meta-analyses. For example, of 9,135 articles published in 2014 and classified as meta-analyses in PubMed, over a third (n = 3,150, 34%) had Chinese author affiliations; the USA was a remote second, producing only 822 meta-analyses in 2014 (9%) (Ioannidis et al. 2013).

The present study had shown some difference of published meta-analyses about disciplines between from China and USA. Chinese authors had published much papers about oncology, biochemistry molecular biology, and genetics heredity. While most of articles from USA were about pharmacology pharmacy, cardiovascular system cardiology, behavioral sciences and etc. Accordingly, most of retracted meta-analyses were from research areas of oncology, biochemistry molecular biology, and genetics heredity. These results were consistent with one recent study (Campos-Varela I et al. 2019), which show more retracted all types papers in PubMed published in journals of oncology and biochemistry molecular biology. There were also higher number of retracted meta-analyses related to genetics in present study. It had been reported that genetics was a discipline with high retraction rate (Dal-Ré et al. 2019). Chinese authors had the largest numbers of retracted medical genetics articles and USA had produced more retractions about non-medical genetics (Dal-Ré et al. 2019).

It is interesting to note that most of the retracted meta-analyses related to genetics were from Chinese authors. In 2014, Chinese authors wrote 63% of the global total (1,210/1,910), whereas authors from the USA produced only 7% (136/1,910) (Ioannidis 2013). These meta-analyses may be redundant. Typically, meta-analyses from China neglected genome-wide data, and often included candidate gene studies published in Chinese-language journals (Ioannidis et al. 2016; Ioannidis et al. 2013). Meanwhile, reviews based on high-quality original reports were relatively sparse from Chinese authors (Ioannidis et al. 2013). Indeed, of the 100 most cited meta-analyses, none were produced by Chinese authors (Uthman et al. 2013).

The present findings confirm a dramatically increasing trend of retracted publications from mainland China (Chen et al. 2018; Wang et al. 2018). There were very few retractions from Chinese authors before 2005, less than a 100 per year between 2005 and 2014, and more than 200 articles retracted per year since 2015 (Chen et al. 2018; Wang et al. 2018). Since 2012, articles with Chinese authors represent a quarter of the retractions in Medline. The common reasons for these retractions from Chinese authors overall in the last years were plagiarism (21.7%), error (18.3%), duplicate publication (16.2%), and faked review process (15.5%) (Chen et al. 2018).

Considerable manipulation of the peer review process has caused cascades of retraction of articles from China (Chen et al. 2018; Wang et al. 2018). The preeminence of peer review manipulation as a reason for retraction of reviews produced by Chinese authors was documented in this study. These retracted meta-analyses were concentrated in a small number of journals, especially Tumor Biology and Molecular Biology Reports. It has been suggested that an irrational scientific evaluation system may be partly blame for this misconduct (Chen et al. 2018; Wang et al. 2018; Liao et al. 2018). Chinese researches have been pressured heavily to publish articles in Web of Science-indexed journals to achieve promotion and funding (Liao et al. 2018). Simultaneously, meta-analyses can be produced and published in journals with high impact factors with relatively limited financial resources. Tragically, some editing companies made and sold meta-analyses from China for proﬁt (Mara 2014). Numerous Obstetrics and Gynecology retractions were due to withdrawal of articles providing practical recommendations or guidelines and subsequent replacement with new reviews, rather than errors or misconduct.

This study had several limitations. Firstly, it is likely some cases of misconduct remain undetected in the literature. Th reluctance of some journals to investigate and potentially retract suspect papers may have led to an underestimation of the true frequency of retraction due to scientific misconduct. Secondly, some systematic reviews that encompass data from flawed articles have not been retracted. For example, 3 guidelines and 15 reviews in the radiation oncology literature have been reported to cite retracted articles as valid sources of data (Zietman et al. 2019). One recent study (Garmendia et al. 2019) had found that nearly half of 24 meta-analyses including falsified data of clinical trials had conclusions altered if excluded the flawed results.

References

Alison McCook. (2018). The retraction process needs work. Is there a better way? https://retractionwatch.com/2018/03/13/the-retraction-process-needs-work-is-there-a-better-way/.

Bellomo, R. (2011). Critical Care and Resuscitation's response to the Boldt scandal. Critical Care and Resuscitation, 13(2), 71.

Boldt, J., Mayer, J., Brosch, C., Lehmann, A., & Mengistu, A. (2010). Volume Replacement with a Balanced Hydroxyethyl Starch (HES) Preparation in Cardiac Surgery Patients[J]. Journal of cardiothoracic and vascular anesthesia, 24(3):399-407.

Boldt, J., Schöllhorn, T., Münchbach, J., & Pabsdorf, M. (2007). A total balanced volume replacement strategy using a new balanced hydoxyethyl starch preparation (6% HES 130/0.42) in patients undergoing major abdominal surgery. European journal of anaesthesiology, 24(3), 267-275.

Campos-Varela I, et al. (2019). Retraction of publications: a study of biomedical journals retracting publications based on impact factor and journal category. Gac Sanit. 2019. https://doi.org/10.1016/j.gaceta.2019.05.008

Chen, W., Xing, Q. R., Wang, H., & Wang, T. (2018). Retracted publications in the biomedical literature with authors from mainland China. Scientometrics, 114(1), 217-227.

Dal-Ré, Rafael & Ayuso, Carmen. (2019). Reasons for and time to retraction of genetics articles published between 1970 and 2018. Journal of Medical Genetics. jmedgenet-2019. 10.1136/jmedgenet-2019-106137.

Evans, D. (2003). Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. Journal of clinical nursing, 12(1), 77-84.

Fang, F. C., Steen, R. G., & Casadevall, A. (2012). Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences, 109(42), 17028-17033.

Garmendia, C. A. , Nassar Gorra, L. , Rodriguez, A. L. , Trepka, M. J. , Veledar, E. , & Madhivanan, P. . (2019). Evaluation of the inclusion of studies identified by the fda as having falsified data in the results of meta-analyses. JAMA Internal Medicine.

Grimes, D. A., Hou, M. Y., Lopez, L. M., & Nanda, K. (2008). Do clinical experts rely on the Cochrane library?. Obstetrics & Gynecology, 111(2), 420-422.

Ioannidis, J. P. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta‐analyses. The Milbank Quarterly, 94(3), 485-514.

Ioannidis, J. P., Chang, C. Q., Lam, T. K., Schully, S. D., & Khoury, M. J. (2013). The geometric increase in meta-analyses from China in the genomic era. PloS one, 8(6), e65602.

Liao, Q. J., Zhang, Y. Y., Fan, Y. C., Zheng, M. H., Bai, Y., Eslick, G. D., ... & He, H. (2018). Perceptions of Chinese biomedical researchers towards academic misconduct: a comparison between 2015 and 2010. Science and engineering ethics, 1-17.

Mara, H. (2014). Copycat papers flag continuing headache in China. Resource document. http://www. sciencemag.org/news/2014/10/copycat-papers-flag-continuing-headache-China

Policy on withdrawing published Cochrane Reviews (including protocols). https://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane-review-management/policy-withdrawing-published-cochrane-reviews-including-protocols.

Qi, X., Deng, H., & Guo, X. (2017). Characteristics of retractions related to faked peer reviews: an overview. Postgraduate medical journal, 93(1102), 499-503.

Steen, R. G., Casadevall, A., & Fang, F. C. (2013). Why has the number of scientific retractions increased?. PloS one, 8(7), e68397.

Uthman, O. A., Okwundu, C. I., Wiysonge, C. S., Young, T., & Clarke, A. (2013). Citation classics in systematic reviews and meta-analyses: who wrote the top 100 most cited articles?. PloS one, 8(10), e78517.

Wager, E., & Williams, P. (2011). Why and how do journals retract articles? An analysis of Medline retractions 1988–2008. Journal of medical ethics, 37(9), 567-570.

Wager, E., Barbour, V., Yentis, S., & Kleinert, S. (2009). Retractions: guidance from the Committee on Publication Ethics (COPE). Maturitas, 64(4), 201-203.

Wang, T., Xing, Q. R., Wang, H., & Chen, W. (2018). Retracted publications in the biomedical literature from open access journals. Science and engineering ethics, 1-14.

Zarychanski, R., Abou-Setta, A. M., Turgeon, A. F., Houston, B. L., McIntyre, L., Marshall, J. C., & Fergusson, D. A. (2013). Association of hydroxyethyl starch administration with mortality and acute kidney injury in critically ill patients requiring volume resuscitation: a systematic review and meta-analysis. Jama, 309(7), 678-688.

Zietman, A. L., Yom, S. S., & Braverman, L. C. (2019). Making Sure Retractions Matter. International Journal of Radiation Oncology• Biology• Physics, 103(5), 1043-1044.

Tables

Table 1. Number of articles tagged as a “meta-analysis”, “systematic review”, “retraction of publication”, “retracted publication” in PubMed and number of reviews published in The Cochrane Database of Systematic Review by publication year.

Year	All article types	Meta-analysis	Systematic review	Cochrane review	Retraction of publication	Retracted publication
1990	410,275	272	823	–	19	39
1995	448,552	429	2,023	–	20	45
2000	531,304	849	5,969	1,009	24	102
2005	698,923	2,169	10,459	549	74	255
2010	940,239	4,691	19,950	774	313	438
2015	1,253,884	12,788	37,909	1,007	693	491
2017	1,275,076	7,544	40,919	769	513	101
All	18,992,706	85,862	346,640	13,312	5,542	5,564

PubMed search conduced on March 22, 2018. Cochrane review data became accessible in PubMed in 2000.

Table 2. Distribution of meta-analyses in the Medline according to research areas from China, USA and others up to year of 2017 searched thorough Web of Science.

Country	Research Area	Number	Percentage
China			% of 20129
	Biochemistry Molecular Biology	8352	41.492
	Oncology	7762	38.561
	Genetics Heredity	6242	31.01
	Pharmacology Pharmacy	5888	29.251
	Pathology	4781	23.752
USA			% of 18120
	Pharmacology Pharmacy	6305	34.796
	Cardiovascular System Cardiology	4373	24.134
	Behavioral Sciences	4321	23.847
	Neurosciences Neurology	4091	22.577
	Mathematics	4089	22.566
Others			% of 32453
	PHARMACOLOGY PHARMACY	28417	28.614
	BIOCHEMISTRY MOLECULAR BIOLOGY	22240	22.395
	ONCOLOGY	19843	19.981
	CARDIOVASCULAR SYSTEM CARDIOLOGY	19666	19.803
	SURGERY	19042	19.174

Table 3. Types of the published (or retracted) meta-analyses in the Medline according to countries up to year of 2017.

	meta-analysis			retracted meta-analysis
Search strategy	China	others	all	China	others	all
Gene OR genetic OR polymorphism OR genome OR mutation OR haplotype	6520	6773	13293	40	4	44
Trial OR randomi* OR treatment	10559	48865	59424	25	24	49
Sensitivity	1376	2683	4059	2	0	2
Cohort OR case control	1130	3020	4150	6	1	7
Miscellaneous meta-analyses	1552	10290	11842	1	3	4
All	21137	71631	92768	74	32	106

Table 4. Journals with most retracted reviews in the biomedical literature.

Journal	Impact factor	Number
The Cochrane Database of Systematic Reviews	6.1	439
Tumor Biology	2.926	30
Molecular Biology Report	1.6979	14
Obstetrics & Gynecology	5.656	13
European Journal of Medical Research	1.684	6
Contraception	2.788	5

Table 5. Research areas of the retracted meta-analysis in the Medline up to year of 2017 searched thorough Web of Science.

Research Area	Number	% of 118
Oncology	60	50.847
Biochemistry Molecular Biology	57	48.305
Genetics Heredity	46	38.983
Mathematics	31	26.271
Gastroenterology Hepatology	28	23.729
Pharmacology Pharmacy	28	23.729
Research Experimental Medicine	23	19.492
Pathology	21	17.797
Neurosciences Neurology	20	16.949