Electronic and additional search resulted in a total of 1,945 records. Of these records, 945 were excluded as duplicates. After screening the titles, abstract and full texts, a total of 150 non-Cochrane SRs in medicine retracted between 2004 and 2020 were included. The process of study selection including the reasons for exclusion is shown in Figure 1.
The characteristics of the 150 included studies are displayed in Table 1. The total number of published SRs is large and has been growing over the past decade (from 2011 to 2020), and the number of retracted SRs is small but increased as well (Figure 2). The oldest included SR, published in 2003, was retracted in 2004, and the highest retraction rate was in 2015 (n=43, 28.7%). The first authors of retracted SRs were from 18 countries, most from Asia (n=126, 84.0%). China was the leading country of origin for the total number of retracted publications (n=113, 75.3%), followed by the United States (n=5, 3.3%), Italy (n=5, 3.3%), the United Kingdom (n=4, 2.7%) and Iran (n=4, 2.7%). The most frequent type of organizations of the first author were hospitals (n=106, 70.7%), followed by universities (n=41, 27.3%).
A total of 79 journals were involved. Tumor Biology retracted the most (n=23, 15.3%), followed by Molecular Biology Reports (n=13, 8.7%), Medicine (n=9, 6.0%) and Molecular Neurobiology (n=9, 6.0%). All included SRs were written in English (except one that was published in Chinese. Most articles (n=120, 80.0%) were published in journals listed in the Science Citation Index (SCI). Most retracted SRs (n=85, 56.7%) were published in low impact (≤3) journals, and the distribution across the JCR ranking quartiles was relatively even.
Not all retraction notices were clearly labelled and linked to the retracted articles. For a total of 148 (98.7%) studies a retraction notice was published, the first of which in 2004. For two (1.3%) studies no retraction notice was issued, but the study was presented with a watermark to identify the article was retracted. For these studies, we determined the issuer of the retraction notice as unclear.
Characteristic of the retractions
Issuer of the retraction notice
We categorized the entities that retracted the 150 SRs into nine groups (Table 2). The largest proportion of retraction notices were issued by the publisher and editor(s) jointly (n=60, 40.0%), followed by the editor alone (n=27, 18.0%) and author alone (n=16, 10.7%). For nine (6.0%) included studies, we were unable to determine who issued the retraction notice. Of these nine notices, two were from the author(s) and/or editor, one was from publisher or journal, three used the words as “we”, one used the words as “they”, and for two no notice was issued. Although most retraction notices declared who retracted the article, 7 (4.7%) did not explicitly state this information.
Reasons of retraction
A total of 161 reasons for retraction were mentioned for the included 150 SRs (Table 3). We divided the retraction reasons into three categories: honest error (unreliable data), misconduct (data fabrication, plagiarism, duplicate publication, authorship issues, compromised peer review), and unclear (not possible to distinguish “honest error” from “misconduct”). The vast majority of SRs (n=100, 66.7%) were retracted due to some form of research misconduct.
The most common reason for retraction was fraudulent peer-review (n=61, 37.9%), followed by unreliable data (n=41, 25.5%) and plagiarism (n=30, 18.6%). Most reasons classified as unreliable data were due to error in either the selection of included studies, or in data analysis (Table 4). Other reasons included duplicate publication (n=9, 5.6%), authorship issues (n=8, 5.0%) and only one (0.6%) was retracted due to data fabrication. No articles were retracted due to repetitive research. We assigned seven (4.4%) retraction statements, which appeared to use deliberately ambiguous wording that made it difficult to distinguish honest errors from suspected (or proven) misconduct, into others (n=7, 4.4%). No reason was provided for four (2.5%) retractions. Figure 3 shows the growth and variation in reasons for retractions over time.
In 11 (7.3%) studies two reasons for retraction were given. Among the 11 retracted SRs, four were retracted due to fraudulent peer-review and plagiarism, two were retracted due to plagiarism and authorship issues, two were retracted due to fraudulent peer-review and authorship issues, one was retracted due to plagiarism and authorship issues, one due to plagiarism and unreliable data, and one due to unreliable data and concerns about third party involvement in editing submitted files.
In addition to this, causes for retraction also varied according to journal IF (P＜0.001) and journal rank in JCR (P＜0.001): retractions due to unreliable data were most frequent from journals with a high IF (equal or more than 5) or high journal rank (Q1 and Q2), while a faked review process was the main cause of retraction from journals with a low IF (less than 5), low journal rank (Q3 and Q4) and journals with no citation index. In-depth analysis showed that the most common reason for retraction for articles with led authors from hospitals and universities were fraudulent peer-review and unreliable data, respectively.
The median (range) time between publication of the 150 SRs and their retraction was 14.0 (0.0~108.0) months. Most of the included SRs (n=105, 70.0%) were retracted within 20 months after publication. Seven SRs were retracted within one month, of which, five on the same day the article was published, we therefore determined the month to retraction was zero.
The duration from publication to retraction also varied by the reasons (Table 5). When excluding the reason of data fabrication (only one study with an interval of 75 months), authorship issues resulted in the longest interval with a minimum of six month and a maximum of 50 months, followed by fraudulent peer-review, plagiarism, duplicate publication and unreliable data. In general, SRs with research misconduct (including data fabrication, fraudulent peer-review, authorship issues, plagiarism, and duplicate publication) took longer to be retracted (16.0 [10.00~31.3] months) than honest error (7.0 [4.0~22.0] months) in total, P＜0.001.
Reliability of retracted SRs
Retracted SRs have the potential to mislead current practice and future research through citations that occur before the publication of the retraction note, and erroneous citations that are made after the retraction. It is therefore important for researchers and readers to understand how much influence retracted SR can have on other studies. Our research group divided the reliability of retracted SRs into three categories:
- High risk: The findings or conclusions of retracted SRs will be changed due to retraction so that citations will be severely affected (including unreliable data, data falsification).
- Low risk: The findings or conclusions of retracted SRs will not be changed due to retractions so that citations will not be affected (including repetitive research, duplicate publication and authorship issues).
- Unclear: we are uncertain about whether the findings or conclusions of retracted SRs will be changed or whether citations will be affected (including compromised peer review and plagiarism)
The results of our study showed that 42 (28.0%) retracted SRs were considered as “high risk”, and only 12 (8.0%) as “low risk”.