A survey of retracted articles in the three high-impact journals

doi:10.21203/rs.3.rs-1558719/v1

PDF

Research Article

A survey of retracted articles in the three high-impact journals

https://doi.org/10.21203/rs.3.rs-1558719/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Objectives: The effect of a retracted publication can be significant, especially in high-impact journals. We aimed to identify the characteristics of retracted articles in three high-impact clinical journals, including The Journal of the American Medical Association (JAMA), Lancet, and The New England Journal of Medicine (NEJM).

Methods: The Retraction Watch database was searched from inception to March 21, 2021. Data collected for each article included title, article type, reasons for retraction, journal, publisher, publication year, retraction year, first author, country of origin, and times a retracted article was cited. Data analysis was performed by GraphPad Prism version 8.00 (La Jolla, CA, United States), and SPSS Statistics 26.0.

Results: A total of 77 articles were retracted, of which 20 were from JAMA, 31 from Lancet, and 26 from NEJM. Over one-half of papers were retracted within the last decade (n=44, 57.1%). Since 2005, the number of retracted articles had increased dramatically. The United States (n=29, 37.7%) was the country with the highest number of retracted publications in the three journals, followed by the United Kingdom (n=9, 11.7%) and China (n=5, 6.5%). The most common reasons for retractions were “Error” (n=20, 26.0%), “Fraud” (n=19, 24.7%) and “Reliability” (n=18, 23.4%) in these three journals. The media number of citations was 79 (range 0-2677). The media time to retraction was 651 (range 2-8474) days. The time was longer when an article was retracted because of fraud than that because of error.

Conclusions: By analyzing the features of retracted articles in the three high-impact journals, we could identify the reason for retraction more comprehensively and benefit in improving the quality of biomedical literature.

Retraction of Publication

Scientific Misconduct

high-impact journals

Retraction, an important deterrent to scientific misconduct, means removing seriously flawed publications to correct the literature and alerts readers to articles that contain seriously flawed or erroneous content that their findings and conclusions cannot be relied upon [1]. In recent years, there has been a dramatic increase in the number of retracted publications in the biomedical community [2–4]. Meanwhile, more and more retracted articles have been linked to scientific misconduct. A previous study suggested that 67.4% of retractions were due to research misconduct [5]. Unfortunately, plenty of authors used an unreliable paper that had already been retracted as their reference without mentioning the retraction related to the citation [6]. Therefore, unreliable medical publications have much negative influence on account of the ongoing propagation of erroneous data, causing spurious clinical decision-making and consequently affecting patient care [7].

At the same time, awareness of retraction in the academic community has grown. Since October 2018, the Retraction Watch database, as the largest and most comprehensive database worldwide containing retracted papers in all disciplines, has been publicly available [8]. Publishers or authors often issue a retraction notice when they decide to retract an article. However, there were significant variations in the notice from different journals because the retraction practices of journals are not unified [9]. Under the circumstance, the Retraction Watch Database used a unified taxonomy when they coped with retraction notices from different editors or authors.

The 2020 impact factor was 56.274 for JAMA, 79.323 for Lancet, 91.253 for NEJM (From Journal Citation Reports) [10]. Therefore, those retracted papers with such high citations in these high-impact journals would cause serious damage to academic research. Up to now, there has been no publication regarding retracted articles in such high-impact journals. Hence, the purpose of this paper was to analyze the features of retracted articles in the three high-impact clinical journals, including The Journal of the American Medical Association, Lancet, and The New England Journal of Medicine, and provide some suggestions for authorities, publishers, and authors.

Search methods

The Retraction Watch database was searched from inception to March 21, 2021 [8]. The following search strategy was applied: Journal: “JAMA: Journal of the American Medical Association”; “Lancet”; “NEJM: The New England Journal of Medicine”; The nature of notice: “retraction”.

Data extraction

For each article, the following data were collected on the Retraction Watch Database: title, article type, reasons for retraction, journal, publisher, publication year, retraction year, the time between publication and retraction, first author, country of origin, and times a retracted article was cited.

The reasons for retraction of each article were extracted from the Retraction Watch Database. Some retractions listed multiple reasons, but we assigned each retraction to a single, mutually exclusive category which Brainard J reported in Science already.[11] The classification of reasons was conducted by one author (Kang Zou) and checked by another author (Jiao Jiang). If there was a discrepancy in opinion, the authors would discuss to reach an agreement. The number of citations to each article was acquired from the Web of Science database. In addition, the time between publication and retraction was calculated in days.

Data analysis

Continuous variables were presented as median (range). Frequency data were reported as number (percentage). Comparisons of the time to retraction and the citations of each retracted article among the three journals were performed by Kruskal-Wallis test. Multiple comparisons were performed by Bonferroni correction. The level of significance was set at p < 0.05. GraphPad Prism version 8.00 (La Jolla, CA, United States), and SPSS Statistics 26.0 were used for data analysis.

In total, 77 retracted articles were identified from these three journals within the Retraction Watch Database, of which 20 were from JAMA, 31 from Lancet, and 26 from NEJM.

Article type

When classified by article type in the three top journals, both clinical study and research article accounted for 35.1.% of the retracted articles (n = 27), followed by Letter (n = 10, 13.0%), case report (n = 5, 6.5%), commentary/Editorial (n = 4, 5.2%) meta-analysis (n = 2, 2.6%) and review article (n = 2, 2.6%) (Table 1).

Table 1

Characteristics of the retracted articles
Characteristic	N (%)
	JAMA (n = 20)	Lancet (n = 31)	NEJM (n = 26)	Total (n = 77)
Article type
Clinical Study	6 (30.0)	12 (38.7)	9 (34.6)	27 (35.1)
Research Article	11 (55.0)	9 (29.0)	7 (26.9)	27 (35.1)
Letter	2 (10.0)	2 (6.5)	6 (23.1)	10 (13.0)
Case Report	-	2 (6.5)	3 (11.5)	5 (6.5)
Commentary/Editorial	1 (5.0)	2 (6.5)	1 (3.8)	4 (5.2)
Meta-Analysis	-	2 (6.5)	-	2 (2.6)
Review Article	-	2 (6.5)	-	2 (2.6)
Authors with at least two retraction	1 (5.0)	1 (3.2)	2 (7.7)	4 (5.2)
Retraction in the first year after publication	4 (20%)	16 (51.6%)	10 (38.5%)	30 (39.0%)
Time to retraction, days, (range)	813 (154–4907)	266 (2−8474)	735 (18−2716)	651 (2−8474)
Times cited	77 (2−214)	91 (0−1499)	48 (1−2677)	79 (0−2677)
Retraction after 2000	19 (95.0)	23 (74.2)	18 (69.2)	60 (77.9)
Retraction after 2010	17 (85.0)	18 (58.1)	9 (34.6)	44 (57.1)

Trends over time

Of the 77 articles retracted, most have been published during the past 20 years (n = 56, 72.7%), with nearly one-half published after 2010 (n = 35, 45.5%) (Fig. 1). Figure 2 presented the years of retractions for each included article. The first retraction notice issued in 1975 was from NEJM. There was no or modest increase until 2005 when a marked growth occurred and this growth continued, reaching a peak in 2019 with 8 articles retracted. However, there was a slight decrease in the number of retracted articles after 2019. The first retraction of Lancet and JAMA occurred in 1989. Similarly, the majority of articles were retracted after 2000 (n = 60, 77.9%), with over one-half retracted within the last decade (n = 44, 57.1%). Especially in JAMA, the proportion was as high as 95% and 85%, respectively (Table 1).

Time to retraction and Citations

The media time to retraction was 813 (range 2-214) days in JAMA, 266 (range 2-8474) days in Lancet, and 735 (range 18-2716) days in NEJM, 651 (range 2-8474) days in total, with no statistical significance in the three journals (p = 0.20). About 39% (n = 30) of the papers were retracted in the first year after publication (Table 1). Table 2 displayed the time between publication and retraction by reasons. The time to retraction significantly differed among reasons (p = 0.04). And the time was longer when an article was retracted because of fraud than that because of error (p = 0.08).

The median number of citations was also not significantly different in the three top journals (p = 0.50), of which 77 (range 2-214) in JAMA, 91 (range 0-1499) in Lancet, 48 (range 1-2677) in NEJM, 79 (range 0-2677) in total (Table 1).

Table 2

Time to retraction by reasons
Reason	Time to retraction, median (range)	P value
Fraud	1703 (18−4361)	P = 0.004
Other misconduct	406 (56−4907)
Possible misconduct	416 (133–8474)
Reliability	805 (13−2677)
Error^a	290 (19−1896)
Miscellaneous^b	59 (2−2163)
^a: p = 0.008 VS Fraud;
^b: P = 0.008 VS Fraud;

Countries

Articles originated from 19 countries (Fig. 3). For JAMA, countries with the highest count of retracted articles were the United States (n = 12, 60%), followed by Australia (n = 2, 10%), each of the remaining 6 countries was accounted for 5%. For Lancet, articles were most commonly from the United States (n = 7, 22.6%), followed by the United Kingdom (n = 6, 19.4%), France (n = 3, 9.7%). For NEJM, the top two countries were the United States (n = 10, 38.5%) and the United Kingdom (n = 3, 11.5%), each of the rest 10 countries were accounted for less than 8%. Overall, The country with the highest count of retracted articles in the three journals was the United States (n = 29, 37.7%), followed by the United Kingdom (n = 9, 11.7%) and China (n = 5, 6.5%) (Table 3).

Table 3

Number of retractions based on country of origin
	JAMA, n (%)	Lancet, n (%)	NEJM, n (%)	Total, n (%)
Country of origin
United States	12 (60.0)	7 (22.6)	10 (38.5)	29 (37.7)
United Kingdom	-	6 (19.4)	3 (11.5)	9 (11.7)
China	1 (5.0)	2 (6.5)	2 (7.7)	5 (6.5)
Canada	1 (5.0)	2 (6.5)	1 (3.8)	4 (5.2)
Japan	1 (5.0)	2 (6.5)	1 (3.8)	4 (5.2)
Australia	2 (10.0)	-	1 (3.8)	3 (3.9)
France	-	3 (9.7)	-	3 (3.9)
Germany	-	2 (6.5)	1 (3.8)	3 (3.9)
India	1 (5.0)	-	2 (7.7)	3 (3.9)
Norway	-	1 (3.2)	2 (7.7)	3 (3.9)
Spain	-	2 (6.5)	1 (3.8)	3 (3.9)
Austria	-	1 (3.2)	-	1 (1.3)
Brazil	1 (5.0)	-	-	1 (1.3)
Italy	-	-	1 (3.8)	1 (1.3)
Netherlands	1 (5.0)	-	-	1 (1.3)
Pakistan	-	-	1 (3.8)	1 (1.3)
Sweden	-	1 (3.2)	-	1 (1.3)
Switzerland	-	1 (3.2)	-	1 (1.3)
Turkey	-	1 (3.2)	-	1 (1.3)

Reasons for retraction

Most of the retractions were due to multiple reasons. These reasons were grouped into six categories by the single, mutually exclusive method. Table 4 shows the reasons for article retraction. The most common reasons for retractions were “Error” (n = 9, 45.0%) for JAMA, “Reliability” (n = 9, 29.0%) for Lancet, “Fraud” (n = 7, 26.9%) and “Reliability” (n = 7, 26.9%) for NEJM. In total, “Error” (n = 20, 26.0%) was the most common reason for the three high-impact journals, followed by “Fraud” (n = 19, 24.7%) and “Reliability” (n = 18, 23.4%). Besides, both “Other misconduct” and “Miscellaneous” accounted for 10.4% (n = 8), “Possible misconduct” only 5.2% (n = 4).

Table 4

Reasons for retracted articles in the three high-impact journals
	JAMA, n (%)	Lancet, n (%)	NEJM, n (%)	Total, n (%)
Reason
Fraud	5 (25.0)	7 (22.6)	7 (26.9)	19 (24.7)
Other misconduct	1 (5.0)	3 (9.7)	4 (15.4)	8 (10.4)
Possible misconduct	1 (5.0)	2 (6.5)	1 (3.8)	4 (5.2)
Reliability	2 (10.0)	9 (29.0)	7 (26.9)	18 (23.4)
Error	9 (45.0)	7 (22.6)	4 (15.4)	20 (26.0)
Miscellaneous	2 (10.0)	3 (9.7)	3 (11.5)	8 (10.4)

In our study, we identified a total of 77 retracted articles in these three high-impact journals. Among the 77 retracted articles, 94.8% of first authors were related to one retracted article, only four (5.2%) authors were responsible for two retracted publications (Supplemental Appendix 1).

Hua F et al pointed out that high-impact journals, especially biomedical journals, policed articles they published by adopting the Committee on Publication Ethics (COPE) or similar policies [12]. Fang et al also concluded that high-impact journals may retract fraudulent publications more rapidly with their greater visibility and enhanced scrutiny [5]. Our study showed that the median time to retraction was 651 (range 2-8474) days in the three top journals: 1703 days for retraction because of fraud, and 290 days for error. Gaudino M et al observed a median time of 1.8 years in biomedical literature [13]. It seems that the time for retraction in high-impact journals was not significantly shorter in the present study. The time from publication to retraction varied among reasons. Our results corresponded to earlier studies that concluded those articles retracted for error took a shorter time than fraud to retract [5, 14, 15]. The phenomenon may be partly attributed to the time interval required to identify fraud [5]. Besides, articles are most likely to be retracted in the first year after publication and most retractions occur in the first several years [16]. Likewise, about two-fifths of retractions occurred in the first year after publication in the present study.

The number of citations of retracted articles was correlated with the time-to-retraction and thus with how widely the invalid findings were propagated [17]. Gaudino M et al also identified the median number of citations was 9 [13]. Besides, Chambers LM et al observed a media number of 8 in 2019 [4]. However, our study implied that the median number of citations was 79 (range 0-2677), which is significantly higher than these studies. It means it is easy for compromised scientific conclusions of these top journals to be disseminated and will bring devastating disaster. Hence, these high-impact journals need to take more rigorous methods to reduce the lag-time to retraction.

Seventy-seven articles originated from 19 countries. We found the greatest number of retracted articles within these three journals were in the United States (37.7%), followed by the United Kingdom (11.7%) and China (6.5%). Similarly, a previous study suggested that about 41% of retractions searched from Retraction Watch Database were from the United States and China [4]. Notably, China is second only to the United States in the volume of papers published in journals listed in the Science Citation Index. However, China only accounted for 6.5% of retractions in the three journals. There was a possible relationship between the number of retractions for each country and its research output [18]. Although the United States and China are the two countries with the largest research output, the United States leads in most subject areas in the Scimago Journal Rankings (1996–2020) [19]. It indicates that the bulk of its articles was published in high-impact journals. Hence, the number of retractions in the United States was much higher than that in China in these high-impact journals.

Two studies on Medline pointed error and fraud were the most common reason for retractions in high-impact journals [5, 18]. Likewise, error and fraud were also the lead reason in our study. Previous studies reported that the proportion of retraction attributed to scientific misconduct was about 50%, 62.2%, 67.4% respectively in all journals [11, 13, 5]. However, fraud was only involved in 24.7% (n = 19) of retractions in our study. Firstly, we deduce these top journals may be more proficient in identifying misconduct after authors submitting their manuscripts. Secondly, the definition of scientific misconduct varied among studies. We included fabrication, falsification, and plagiarism, which was defined as scientific misconduct by the United States government. Others may included fake peer reviews, forged authorship, duplication, lack or withdrawal of informed consent, etc additionally which were thought to be dishonest and unethical behaviors by scientists widely. Hence, future studies may attach importance to developing a practical and standardized classification of reasons for retractions that could improve the relevance and uniformity of the research regarding retracted papers. Because better understanding the root causes of retractions can not only help to change scientific culture but also prevent the public from losing trust in science [20, 21]. A fake peer review is also a frequent reason for retractions. A previous retrospective analysis confirmed that 32.8% of articles were retracted because of a fake review in 134 retracted articles [22]. Besides, Q.-H. Vuong et al concluded a fake peer review was the most common reason in a study on the basis of retraction data through February 2019 [17]. Unexpectedly, no retraction was because of a fake peer review in our datasets. It may suggest that these high-impact journals own an excellent system of peer review which is worthy for other publishers to follow.

There are some strengths in our study. It is the first report on the features of retractions in high-impact journals. We used the Retraction Watch Database which is the largest and most comprehensive database worldwide containing a large number of retracted papers in all disciplines. It adopts a unified taxonomy when dealing with retraction notices issued by various editors/authors with different forms, clarity, and level of detail [23]. We believe our analysis of the three top journals could serve as a reference for future investigation. We acknowledge that there are several limitations in the present study. Firstly, The quantity of retracted articles in our study is too small. Our retraction data are only based on the Retraction Watch Database, but not all the retracted articles were included in this database. Secondly, some articles were retracted for multiple reasons, but the classification of reasons we adopted is mutually exclusive. It means the conclusion may change if we assign the several reasons for one retraction into all corresponding categories. Thirdly, we did not identify which citations were post-retraction citations that were defined by Kim et al as “those occurring at least one year after the retraction” [24]. The result may be different if we adopt such a method to extract data. Although there are some defects, we hope the present study can attract enough attention among the biomedical community and bring benefits in improving the quality of biomedical literature.

We would like to give some suggestions to reduce the number of retractions in these high-impact journals. For the policymakers: First, reinforce integrity education, especially in countries with high research output. Second, improve the incentives in the academic community. Abritis A et al confirmed that some institutions worldwide give money rewards when researchers publish a certain number of articles in these high-impact journals [25]. For the publishers, develop new tools to help editors to discover certain types of errors. For the authors: First, it is wise to keep the raw data carefully and conduct a comprehensive check to the error of data, methods, analyses, text, results, and conclusions when they submit their manuscripts. Second, it will be advised to check the status of the referenced articles before publication.

Although the total number of retracted publications was small, multiple measures should be taken to ensure there is no retracted article in these high-impact journals. The peer review of such high-impact journals was encouraging because no retraction was due to a fake peer review in our study. Further research could also investigate the discrepancy of features of retracted articles between high-impact journals and general medical journals through a large-sample study.

JAMA: The Journal of the American Medical Association; NEJM: The New England Journal of Medicine; COPE: the Committee on Publication Ethics

Ethics approval and consent to participate

Not applicable.

Consent for publication

All authors have provided their consent for the publication of the final manuscript.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

We declared that no conflicts of interest or financial ties to disclose.

Funding

This study is independent research funded by the following grants: Cooperation Project of Southwest Medical University and Luzhou Government (No. 2019LZXNYDJ24), Youth Foundation of Southwest Medical University (No. 0903-00031099), Doctoral research start-up funding project of Affiliated Hospital of Southwest Medical University (No. 16229).

Author contributions

Study conception and design: Xiaowei Tang, Muhan Lü; acquisition of data: Kang Zou, Jiao Jiang, Yinqin Lü; critical revision: Shu Huang,; interpretation of data: Huan Xu, Xinxin Pu; drafting of manuscript: Kang Zou; revision of manuscript: Xiaowei Tang, Yan Peng; and final approval of manuscript: Xiaowei Tang.

Acknowledgements

Not applicable.

COPE Council. COPE Retraction guidelines — English. https://doi.org/10.24318/cope.2019.1.4 Version 2: November 2019. ©2019 Committee on Publication Ethics (CCBY -NC-ND 4.0)
Mousavi T, Abdollahi M. A review of the current concerns about misconduct in medical sciences publications and the consequences. Daru. 2020;28(1):359-369.
Cokol M, Ozbay F, Rodriguez-Esteban R. Retraction rates are on the rise. EMBO Rep. 2008;9(1):2.
Chambers LM, Michener CM, Falcone T. Plagiarism and data falsification are the most common reasons for retracted publications in obstetrics and gynaecology. BJOG. 2019;126(9):1134-1140.
Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc Natl Acad Sci U S A. 2012;109(42):17028-17033.
Inoue Y, Muto K. Noncompliance with Human Subjects' Protection Requirements as a Reason for Retracting Papers: Survey of Retraction Notices on Medical Papers Published from 1981 to 2011. Account Res. 2016;23(2):123-135.
George SL. Research misconduct and data fraud in clinical trials: prevalence and causal factors. Int J Clin Oncol. 2016;21(1):15-21.
Retraction Watch Database. http://retractiondatabase.org/RetractionSearch.aspx. Accessed March 21, 2021.
Bilbrey E, O'Dell N, Creamer J. A novel rubric for rating the quality of retraction notices. Publications, 2014, 2(1): 14-26.
Journal Citation Reports: https://jcr.clarivate.com. Accessed 23 Mar 2022.
Brainard J. Rethinking retractions. Science. 2018;362(6413):390-393.
Hua F, Walsh T, Glenny AM, Worthington H. Surveys on Reporting Guideline Usage in Dental Journals. J Dent Res. 2016;95(11):1207-1213.
Gaudino M, Robinson NB, Audisio K, et al. Trends and Characteristics of Retracted Articles in the Biomedical Literature, 1971 to 2020. JAMA Intern Med. 2021;181(8):1118-1121.
Steen RG. Retractions in the scientific literature: do authors deliberately commit research fraud?. J Med Ethics. 2011;37(2):113-117.
Trikalinos NA, Evangelou E, Ioannidis JP. Falsified papers in high-impact journals were slow to retract and indistinguishable from nonfraudulent papers. J Clin Epidemiol. 2008;61(5):464-470.
Vuong Q H, La V P, Hồ M T, Vuong T T, Ho M T. Characteristics of retracted articles based on retraction data from online sources through February 2019. Science Editing, 2020, 7(1): 34-44.
Madhugiri VS, Nagella AB, Uppar AM. An analysis of retractions in neurosurgery and allied clinical and basic science specialties. Acta Neurochir (Wien). 2021;163(1):19-30.
Ribeiro M D, Vasconcelos S M R. Retractions covered by Retraction Watch in the 2013–2015 period: prevalence for the most productive countries. Scientometrics, 2018, 114(2): 719-734.
Scimago Journal & Country Rank: https://www.scimagojr.com. Accessed 23 Mar 2022.
Gauchat G. Politicization of science in the public sphere: A study of public trust in the United States, 1974 to 2010. American sociological review, 2012, 77(2): 167-187.
Sovacool BK. Using criminalization and due process to reduce scientific misconduct. Am J Bioeth. 2005;5(5):W1-W7.
Moylan EC, Kowalczuk MK. Why articles are retracted: a retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open. 2016;6(11):e012047.
Dal-Ré R, Ayuso C. Reasons for and time to retraction of genetics articles published between 1970 and 2018 [published correction appears in J Med Genet. 2020 Jun;57(6):435-436]. J Med Genet. 2019;56(11):734-740.
Faggion CM Jr, Ware RS, Bakas N, Wasiak J. An analysis of retractions of dental publications. J Dent. 2018;79:19-23.
Abritis A, McCook A; Retraction Watch. Cash incentives for papers go global. Science. 2017;357(6351):541.

No competing interests reported.

Additionalfile1.pdf
Additional file 1: Characteristics of the retracted articles. A description of some general features of retracted articles from the three high-impact journals.
Additionalfile2.pdf
Additional file 2: Number of retractions based on country of origin. A description of the original countries of retracted articles and the numbers for each country.

PDF

Editor assigned by journal
25 Apr, 2022
Editor invited by journal
18 Apr, 2022
Submission checks completed at journal
18 Apr, 2022
First submitted to journal
14 Apr, 2022

You are reading this latest preprint version

A survey of retracted articles in the three high-impact journals

Status:

Version 1

Abstract

Figures

Background

Methods

Search methods

Data extraction

Data analysis

Results

Article type

Trends over time

Time to retraction and Citations

Countries

Reasons for retraction

Discussion

Conclusions

Abbreviations

Declarations

References

Competing Interests

Supplementary Files

Status:

Version 1